From Zero to Running LLMs in One Hour

The complete beginner's guide to unboxing your Jetson Orin, flashing JetPack, and running your first local LLM — step by step, with zero prior experience required.

Time Estimate: 45–60 minutes from unboxing to chatting with Llama 3.1. This guide assumes you have an Jetson Orin AI hardware developer kit, a USB-C cable, an Ubuntu host machine (for flashing), keyboard, mouse, monitor, and internet connection.

Pre-Flight Checklist

Phase 1: Hardware Setup (5 minutes)

Step 1

Unbox your Orin developer kit. You'll find the main carrier board with the Orin module pre-installed, a power supply, and (usually) WiFi antennas. Do not power it on yet — we need to put it in recovery mode for flashing.

Step 2

Locate the Force Recovery button (typically labeled "REC" or "FC REC") and the Power button on the carrier board. Connect the USB-C cable between your Ubuntu host and the Orin's USB-C port (the one labeled for flashing, not the USB-C power port). Connect HDMI, keyboard, mouse, and Ethernet.

Step 3

Hold the REC button, press Power, release Power, then release REC after 2 seconds. The Orin enters recovery mode. On your host machine, verify by running:

$ lsusb | grep NVIDIA
# Should show: NVIDIA Corp. APX

Common Pitfall: If lsusb doesn't show NVIDIA, your USB-C cable might be charge-only (not data). Try a different cable — this is the #1 cause of flashing failures for first-time JetPack users.

Phase 2: Flash JetPack (20 minutes)

Step 4

On your Ubuntu host, install NVIDIA SDK Manager. This is the official tool for flashing JetPack — it downloads the OS image, CUDA, cuDNN, TensorRT, and all AI libraries in one operation.

# Download SDK Manager
$ wget -O sdkmanager.deb https://developer.nvidia.com/downloads/sdk-manager
$ sudo apt install -y ./sdkmanager.deb

# Launch SDK Manager
$ sdkmanager
Step 5

In SDK Manager, log in with your NVIDIA Developer account. Select your target hardware (Jetson Orin AGX or NX), choose JetPack 6.x (the latest release), and select all components. Click "Continue" and then "Flash." SDK Manager will download approximately 8–12 GB of software and flash it to the Orin.

During the flash, the Orin's screen will show progress. When complete, the device reboots into Ubuntu 22.04 setup. Follow the on-screen prompts to configure your language, keyboard layout, timezone, and user account. For Orin desktop AI setup users, you're now looking at a full Ubuntu desktop running on ARM64.

Pro Tip: Create a dedicated user account named "jetson" with a simple password for development. You'll be SSHing into this machine frequently, and a short username saves keystrokes. Security hardening comes later — right now we're optimizing for speed.

Phase 3: First Boot and System Update (10 minutes)

Step 6

After the Ubuntu setup wizard completes, open a terminal on the Orin (Ctrl+Alt+T) and update the system:

$ sudo apt update && sudo apt upgrade -y
$ sudo apt install -y curl git htop neofetch build-essential
# Verify GPU is accessible:
$ nvidia-smi
# Should show Orin GPU with driver version and CUDA version
Step 7

Enable SSH for remote access and check the power mode:

$ sudo systemctl enable ssh --now
$ sudo nvpmodel -q
# Should show "NV Power Mode: MAXN" (mode 0) for best AI performance

Phase 4: Install Docker and Ollama (10 minutes)

Step 8

Install Docker with NVIDIA GPU support. This is the foundation for all future AI workloads:

$ curl -fsSL https://get.docker.com | sudo sh
$ sudo usermod -aG docker $USER
$ newgrp docker
$ sudo apt install -y nvidia-container-toolkit nvidia-docker2
$ sudo systemctl restart docker

# Verify Docker GPU access:
$ docker run --rm --runtime=nvidia nvcr.io/nvidia/l4t-ml:r36.4.0-py3 nvidia-smi
Step 9

Install Ollama — the easiest way to run LLMs on Jetson:

$ curl -fsSL https://ollama.com/install.sh | sh
$ ollama --version
# Verify GPU detection:
$ ollama run llama3.2:1b "Say hello and identify yourself briefly."
# Should respond within a few seconds if GPU is working

GPU Verification: If Ollama's first response is slow (30+ seconds for a 1B model), it may be running on CPU. Run sudo tegrastats in another terminal — if GR3D (GPU) utilization stays at 0% during inference, Ollama isn't using the GPU. Reinstall nvidia-container-toolkit and restart Docker.

Phase 5: Run Your First Large Model (5 minutes)

Step 10

Now the moment you've been working toward — running a real, capable LLM entirely on your own hardware:

# Pull a capable 8B model (downloads ~5 GB, takes 3-5 minutes)
$ ollama pull llama3.1:8b-q4_K_M

# Chat with your private AI:
$ ollama run llama3.1:8b-q4_K_M
>>> Explain quantum computing in simple terms.

You're now running a state-of-the-art LLM entirely on your own Jetson private AI hardware. No cloud, no API keys, no data leaving your device. This is the essence of Orin self-hosted AI — powerful, private, and running on less power than a traditional desktop PC.

What's Next?

From here, you can explore the broader ecosystem. The Jetson Orin guide series covers advanced topics like TensorRT optimization, multi-model serving, and vision pipeline construction. If you're interested in running your Orin as a permanent network appliance, our Orin AI server documentation covers headless operation, API gateway configuration, and monitoring.

For those comparing options, the ClawBox vs Jetson Orin comparison illustrates what a turnkey solution adds, and the ClawBox Orin comparison helps you decide whether DIY or pre-built better fits your needs. For Orin persistent server configurations that run 24/7, our deployment guide covers the production considerations.

Finally, bookmark the NVIDIA Orin developer portal — it's your source for JetPack updates, container images, and official SDK documentation. Happy building!