NVIDIA’s Jetson Nano is an embedded computing platform built specifically for AI inference at the edge. With its 4GB RAM, 128-core Maxwell GPU, and power consumption under 10 watts, it sits at an interesting intersection: capable enough for meaningful LLM inference but constrained enough that model choice and optimisation matter significantly. Running an LLM on Jetson Nano in 2026 is practical for the right models and use cases — this guide covers the setup, model selection, performance expectations, and the use cases where Jetson Nano genuinely makes sense.
Jetson Nano Specifications and What They Mean for LLMs
The Jetson Nano 4GB has 4GB of shared memory used by both the CPU and GPU — this is the most important constraint for LLM inference. The GPU has 128 CUDA cores, which is modest compared to consumer GPUs but still substantially faster than CPU-only inference for supported operations. The ARM Cortex-A57 CPU (quad-core at 1.43GHz) handles tasks the GPU cannot. Power consumption in 5W mode limits performance further but extends battery life for battery-powered deployments; 10W mode provides better inference speed for mains-powered installations.
The 4GB shared memory limit means the maximum usable model size is roughly a 3B parameter model at Q4 quantisation (approximately 2GB), leaving room for the operating system, inference runtime, and KV cache. In practice, 1–2B models at Q4–Q8 run most reliably. Attempting larger models risks out-of-memory errors or extreme inference slowness due to memory pressure.
Software Setup: JetPack and CUDA
NVIDIA provides JetPack — a software stack for Jetson devices that includes CUDA, cuDNN, TensorRT, and a Ubuntu-based operating system. Install the latest compatible JetPack version (JetPack 4.6 for original Jetson Nano; check compatibility with your hardware revision). After flashing JetPack, verify CUDA is working with a quick test before installing inference frameworks — catching CUDA setup issues early saves debugging time later.
The key software consideration for LLM inference on Jetson Nano is that many modern inference frameworks target newer CUDA versions and ARM64 GPU architectures than the Jetson Nano supports. llama.cpp with CUDA support is the most reliably compatible option — it supports the Maxwell GPU in the Jetson Nano through the older CUDA versions JetPack provides. Ollama also works on Jetson Nano with GPU acceleration through its llama.cpp backend, which is the most convenient setup for developers already familiar with Ollama’s workflow.
Figure 1 — Jetson Nano LLM performance by model size
Installing llama.cpp with CUDA on Jetson Nano
Clone the llama.cpp repository and build with CUDA support enabled. The build requires cmake and the CUDA toolkit from JetPack. Set the CUDA architecture flag to 53 for the Maxwell GPU in the Jetson Nano — this is different from consumer GPUs and using the wrong architecture flag produces a binary that either fails to compile or falls back to CPU inference. After building, download a GGUF model in Q4 quantisation — SmolLM2-1.7B-Instruct-Q4_K_M.gguf from Hugging Face Hub is a good starting point. Run inference with the GPU layer count set to the maximum to offload all layers to the Maxwell GPU.
Verify GPU utilisation with the tegrastats tool (NVIDIA’s monitoring utility for Jetson devices) — it shows GPU utilisation, memory use, and temperature in real time. If tegrastats shows 0% GPU utilisation during inference, the CUDA build did not succeed and inference is running on CPU. A successful GPU-accelerated run shows 50–90% GPU utilisation and significantly higher tokens-per-second than CPU-only inference.
Jetson Orin: The Better Option for 2026 Deployments
While this guide focuses on the Jetson Nano, it is worth noting that the Jetson Orin Nano (released 2023) is the natural upgrade for new deployments in 2026. The Orin Nano 8GB offers 8GB of LPDDR5 memory, 1024 CUDA cores, a dedicated DLA (Deep Learning Accelerator), and supports CUDA 11.4+ — substantially better for LLM inference than the original Nano. At the same 10W power budget, the Orin Nano runs SmolLM2-1.7B at 20–35 tokens per second versus the original Nano’s 5–10, and handles 3B models interactively rather than only for batch processing. For new edge AI projects requiring Jetson hardware, the Orin Nano is the right platform; the original Nano guide above applies to existing deployments and budget-constrained projects where the original Nano is already in hand.
Use Cases Where Jetson Nano Makes Sense
The Jetson Nano LLM deployment makes most sense in specific scenarios where its combination of embedded form factor, low power consumption, CUDA acceleration, and NVIDIA ecosystem integration provides genuine value. Industrial quality control: a Jetson Nano embedded in production equipment can run visual inspection combined with a small language model for result classification and reporting, entirely locally with no network dependency. Robotics: ROS (Robot Operating System) runs well on Jetson, and integrating a small language model for natural language command interpretation adds a conversational interface to a robot without cloud connectivity. Edge analytics: processing sensor data, log files, or short text inputs locally to extract structured information before selectively sending summarised results to a central server. Home automation server: running permanently at low power consumption to provide local voice and text AI features to a home network without cloud dependency. The common thread: applications where NVIDIA’s CUDA ecosystem, the Jetson’s long-term support, and low-power embedded form factor outweigh the modest inference speed compared to a laptop or cloud GPU.
Thermal Management and Cooling
The Jetson Nano’s Maxwell GPU generates meaningful heat during sustained inference. The developer kit board includes a fan header; attach a 5V fan for any deployment involving continuous or frequent inference. Without active cooling, the Jetson Nano throttles aggressively when the SoC temperature exceeds 50°C, which happens quickly under sustained load. With a fan, sustained inference at 10W runs comfortably at 40–45°C without throttling. For embedded deployments in enclosures, ensure adequate ventilation or active cooling — thermal throttling on the Jetson Nano reduces inference speed by 30–50% and is a common source of inconsistent performance in field deployments.
Running LLMs on Jetson Nano in 2026 is practical for the right model sizes and use cases. The 1.7B range delivers genuinely interactive performance with GPU acceleration; 3B models are better suited to batch processing. The Orin Nano offers substantially better performance at the same power budget for new deployments. The value of Jetson over alternatives like Raspberry Pi 5 is the CUDA ecosystem and NVIDIA’s long-term support commitment for industrial use — for projects that need those properties, Jetson Nano remains a reasonable platform even as faster alternatives emerge.
Comparing Jetson Nano to Alternative Edge Hardware
The Jetson Nano is not the only option for embedded LLM inference, and comparing it to alternatives clarifies when it is the right choice. Raspberry Pi 5 (8GB) costs less and runs the same sub-1B models on CPU at comparable speeds, but lacks a GPU and CUDA. Orange Pi 5 with its Mali GPU offers a middle ground at a competitive price point. Mini PCs with Intel Arc or AMD integrated graphics offer substantially more GPU compute for LLM inference at 15–25W, trading the embedded form factor for meaningfully better inference performance. The Jetson Nano’s unique value is the combination of NVIDIA CUDA ecosystem, JetPack SDK with TensorRT and DeepStream, long-term NVIDIA support commitments for industrial use, and the embedded PCB-level integration options that matter for custom hardware products.
Networking and API Access
A Jetson Nano running Ollama exposes an API that any device on the same network can query, turning it into a shared inference server. This is a compelling setup for home or small office environments: a single Jetson Nano running at 5–10W provides LLM inference to all network devices through a simple OpenAI-compatible endpoint. Configure Ollama to listen on the local network interface rather than localhost and any device with an OpenAI-compatible client can query it. This network server model makes the Jetson Nano’s modest individual inference speed less limiting — it serves one device at a time, which is sufficient for household or small team use where simultaneous queries are rare.
Model Options Specifically Optimised for Jetson
NVIDIA maintains a collection of models optimised for Jetson hardware through their NGC (NVIDIA GPU Cloud) catalog and Jetson AI Lab initiative. These include TensorRT-optimised versions of popular models that take advantage of the INT8 and FP16 acceleration capabilities of Jetson hardware more efficiently than generic GGUF models. For teams doing serious production deployments on Jetson hardware, checking the Jetson AI Lab model catalog at jetson-ai-lab.nvidia.com is worthwhile — it lists current recommended models, performance benchmarks on Jetson Orin hardware specifically, and deployment guides that account for Jetson-specific considerations like DLA (Deep Learning Accelerator) offloading and Power Mode configuration. The community benchmark results on Jetson-specific forums (NVIDIA Developer Forums, the Jetson subreddit) are also more relevant than general LLM benchmarks for understanding real-world performance on these devices.