Hands-on Review of the NVIDIA Jetson AGX Orin Developer's Kit: The Cube of AI Power. The Jetson AGX Orin Developer's Kit is a costly box at $1,999, but its performance is hard to ignore.
The NVIDIA AGX Orin, the company's successor to the Jetson AGX Xavier family of systems-on-modules (SOMs), is here — at least in developer kit form — with the promise of significant performance improvements over its predecessor.
Whereas the AGX Xavier line promised "cloud native" capabilities, the AGX Orin aims for something bigger: ushering in the "new age of robotics," as its developers put it. It boasts a significant performance gain for artificial intelligence and deep learning workloads, and serves as the flagship platform for NVIDIA's newest artificial intelligence and deep learning software development kit, JetPack 5.0.
Tech Specs
- CPU: 12-core 2.2GHz NVIDIA Cortex-A78AE ARMv8.2 with 3MB L2 and 6MB L3 cache
- GPU: 1.3GHz NVIDIA Ampere with 2,048 CUDA cores and 64 Tensor cores
- Accelerators: 2× 1.6GHz NVIDIA Deep Learning Accelerator v2 (NVDLA v2), 1× Programmable Vision Accelerator v2.0 (PVA v2.0)
- RAM: 32GB LPDDR5 at 3,200MHz
- Storage: 64GB eMMC 5.1, microSD, M.2 Key M NVMe
- USB: 2× USB 3.2 Gen 2 Type-A, 2× USB 3.2 Gen 1 Type-A, 2× USB 3.2 Gen 2 Type-C with USB PD (one used for power input); 1× micro-USB 2.0 (debug)
- PCIe: Mechanical 16× with 8× PCIe Gen 4 lanes, 1× M.2 Key M with 4× PCIe Gen 4 lanes, 1× M.2 Key E with 1× PCIe Gen 4 lane (populated by wireless module)
- Networking: 10-gig-Ethernet, M.2 Key E 802.11ac 2x2 2.4/5GHz, Bluetooth 5.0
- Display Outputs: 1× multi-mode DisplayPort 1.4a, 8k60
- Camera Inputs: 1× 120-pin expansion connector: 16-lane MIPI CSI-2 with 16 virtual channels, D-PHY 2.1 (up to 40Gbps), C-PHY 2.0 (up to 164Gbps)
- GPIO: 40-pin header (populated) with UART, SPI, I2C, I2S, PWM, 12-pin automation header, 10-pin audio header, 10-pin JTAG header
- Video Encode (H.265): 2× 4k60, 4× 4k30, 8× 1080p60, 16× 1080p30; H.264, AV1 also supported
- Video Decode (H.265): 1× 8k30, 3× 4k60, 7× 4k30, 11× 1080p60, 22× 1080p30; H.265, VP9, AV1 also supported
- Dimensions: 110×110×71.65mm (around 4.3×4.3×2.82")
- Kit Contents: 1× Jetson AGX Orin Developer’s Kit in case with heatsink and fan, 1× AzureWave AW-CB375NF 802.11a/b/g/n/ac Wi-Fi and Bluetooth 5.0 M.2 2230 module (installed), 1× 90W USB PD Type-C power supply, 1x USB Type-A to Type-C cable, Quick Start Guide
On paper, the Jetson AGX Orin appears to be a monster, yet it only has a 4.34.3" footprint if you don't count the rather large 90W USB PD external power supply that comes with it. There's plenty of capacity for expansion, including a full-size PCI Express slot, though depending on what you connect into it, you'll probably need to replace out the provided power supply for something with a little more headroom.
The Jetson AGX Orin comes pre-loaded with the current Jetpack 5.0 software development kit, which is based on Canonical's Ubuntu 20.04 LTS, which is still in mainstream support until April 2023, under its Developer's Kit version, as reviewed. NVIDIA gives access to a variety of pre-trained models — over 100 architectures and over 25 task-based models — as in prior editions, with training and modification available through the TAO Toolkit on an external workstation or pay-to-play cloud computation platform.
The goal, according to NVIDIA, is to create apps on the tiny box alongside an existing workstation before deploying them on AGX Orin systems-on-module that have yet to be delivered. There's no official support for direct development and training on-device, as there was with prior Jetson developer kits — though you could easily try, given that the machine boots into a functioning Ubuntu desktop environment.
Performance
Previous Jetson modules have been lightly chastised for lower-than-expected CPU performance, which, in NVIDIA's defence, was never the emphasis of the boards. The AGX Orin, on the other hand, is a pleasant surprise: its 12 Arm Cortex-A78AE cores, which can run at up to 2.2GHz in the highest power envelope, are a significant upgrade over its predecessor's eight cores, approaching but not quite matching the performance of an eight-core 16-thread AMD Ryzen 2700X desktop processor at a much lower power draw.
When it comes to power consumption, the new Ampere GPU in the AGX Orin stands true to its moniker. As you walked down to the bare minimum, the Jetson AGX Xavier could be configured for a 10W to 30W target power envelope, disabling various cores to reduce power requirements. The AGX Orin, on the other hand, features a 15W absolute minimum power envelope and a 60W maximum power envelope, which is double that of its predecessor.
The reported power envelopes are measured exclusively at the system-on-chip, as with NVIDIA's prior Jetson kits, and are clearly conservative as a result. When measured at the wall, NVIDIA's claimed 60W draw in "MAXN" mode, which activates all CPU and GPU cores and removes any clock restrictions, is the highest at 98W — a measurement that, in NVIDIA's defence, includes the inefficiency of the supplied USB power brick and overhead for the carrier board.
The Jetson AGX Orin is a hungry beast even at idle, sucking 22W in MAXN mode and over 19W even when set to the lowest "15W" level — but NVIDIA promises this will be fixed in a future software update, allowing presently disabled power saving measures to lower idle draw to under 5W.
All of that power needs to go someplace, and that somewhere is a heatsink that comes pre-installed with the casing. The AGX Orin case, while clearly a development of the one provided with the AGX Xavier kit, is a significant advance. There's a clearer route for the on-board fan to suck air in and push it over the heatsink now that the full-size PCI Express slot is shielded by a detachable cover, although one tied to the rest of the chassis by wires for a pair of adhesive antennas glued on the inside. The fan itself is temperature-controlled and, at roughly 50dBA measured at arm's length, is rather quiet for the quantity of air it moves.
Module with a hybrid design
Another issue to bear in mind for developers is that the system-on-module that drives the Developer's Kit doesn't exist — at least not as a stock-keeping device that could be plugged into a final design. So far, two AGX Orin modules have been confirmed, both of which will be available later this year: a 32GB model with eight CPU cores, 1,792 CUDA cores, 56 Tensor cores, and lower maximum GPU and NVDLA frequencies; and a 64GB model with 12 CPU cores, 2,048 CUDA cores, and 64 Tensor cores, as well as higher maximum clock speeds.
The Developer's Kit's core module, on the other hand, is a combination of the two. Except for the 32GB of RAM, its characteristics are nearly identical to the higher-end 64GB model, with a stated 275 TOPS of sparse INT8 computation compared to the lower-end model's 200 TOPS, all three four-core CPU clusters present and proper, and full clock rates for CPU and NVDLA. It provides a useful speed increase for those constructing projects that will fit in 32GB of memory, but it may give a misleading sense of the performance you'll get from the genuine 32GB module after your project is launched.
Support for sparse networks is a key element of the new Ampere GPU architecture, which provides a significant speed boost in certain situations. It's also crucial to NVIDIA's 275 TOPS compute performance claims: While the AGX Orin can achieve 275 TOPS of INT8 compute across all of its cores and accelerators, it can only do so on sparse networks; on a standard dense network, as compatible with the previous-generation AGX Xavier, its throughput drops to 138 TOPS — still a significant improvement over the 32 TOPS of its predecessor.
Gains and losses across generations
There are a few downgrades elsewhere. There's only one video output, and H.265 video encode capabilities are roughly half of the AGX Xavier's, though there's now support for hardware AV1 encoding in place of VP9; hardware decode is similarly halved, with only one 8k30 stream compared to the AGX Xavier's two, though AV1 has been added alongside VP9, H.265, and H.264. The MIPI CSI-2's 16 lanes now allow just 16 virtual channels, compared to 36 on the AGX Xavier, however the C-PHY has been improved to 2.0, with a throughput of 164Gbps, up from 62Gbps on the predecessor.
However, these are accompanied by some significant enhancements. The new CPU cores are significantly improved and there are more of them, the GPU is easily more than three times faster for FP32 workloads, there are four 10-gigabit Ethernet connections on the module with one brought out to RJ45 on the carrier board, and the on-board eMMC has been doubled to 64GB — though that's still a little small for comfort, and an M.2 NVMe drive should be considered a must-have accessory, and an M.2 NVMe
Which leads us to the kit's most serious flaw: its cost. The AGX Orin Developer's Kit costs $1,999, which is $700 higher than the AGX Xavier Developer's Kit when acquired through the NVIDIA Developer Program at launch. It's also not an all-in-one package: in addition to NVMe storage, buyers will need to budget for a traditional desktop or laptop with an NVIDIA GPU on which to run training — or sign up for a cloud computing platform for the same reason — because, despite its impressive specs, there's no official way to run training on the AGX Orin directly.
The Jetson AGX Orin Developer's Kit is now available for $1,999 on the NVIDIA website; the Jetson AGX Xavier Developer's Kit was previously available for $649, but it's been out of stock for some time due to industry-wide component shortages.
Posts
0 Comments