Multi-Model Serving on Jetson Orin

How to run multiple AI models simultaneously on a single Jetson Orin device — LLMs, vision models, speech-to-text, and embeddings — all working together under Docker Compose.

The Multi-Model Vision

Single-model inference is where most Jetson Orin projects start. But the real power of Jetson Orin AI hardware emerges when you run multiple models concurrently — a chat LLM for conversation, a vision model for camera analysis, an embedding model for document search, and Whisper for speech-to-text — all on one device, all running 24/7. With 64 GB of unified memory on the AGX, this is not just possible; it's practical.

An Orin desktop AI setup running multiple models transforms a single Orin into what feels like a personal AI laboratory. You can ask your LLM about a document, have an embedding model surface relevant passages, use a vision model to analyze a photo you just took, and have Whisper transcribe your voice notes — all without leaving the device. This is the promise of Jetson private AI at its fullest: a complete AI stack that's entirely yours.

What We're Building: A Docker Compose stack with 5 AI services — Ollama (LLM chat), Faster Whisper (speech-to-text), CLIP (image embeddings), a sentence transformer (text embeddings), and YOLOv8 (object detection) — all running concurrently on a single Jetson Orin AGX 64GB, with a unified API gateway for external access.

Memory Planning: The Critical First Step

Running multiple models requires deliberate memory planning. The Orin's unified memory means GPU and CPU share the same pool — every model you load consumes from that single 32 or 64 GB reserve. Here's a realistic memory budget for an Orin AGX 64GB running 5 concurrent models:

Even the 16 GB NX can comfortably run 3–4 lightweight models simultaneously. The key is choosing the right model sizes and quantization levels for concurrent operation. For Orin self-hosted AI setups, document this memory budget — it becomes your capacity planning reference as you add services.

Docker Compose: The Multi-Model Stack

Docker Compose is the ideal orchestration layer for multi-model Orin deployments. It provides declarative service definitions, automatic restarts, network isolation, resource limits, and health checks — everything you need to keep multiple AI services running reliably.

# docker-compose.yml — Multi-model AI stack for Jetson Orin
version: '3.8'

services:

  # 1. LLM Chat Service (Ollama)
  ollama:
    image: ollama/ollama:latest
    runtime: nvidia
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_HOST=0.0.0.0
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 16G

  # 2. Speech-to-Text (Faster Whisper)
  whisper:
    image: onerahmet/openai-whisper-asr:latest
    runtime: nvidia
    environment:
      - ASR_MODEL=large-v3
      - ASR_ENGINE=faster_whisper
    ports:
      - "9000:9000"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 6G

  # 3. Text Embeddings (Sentence Transformers)
  embeddings:
    image: ghcr.io/huggingface/text-embeddings-inference:latest
    runtime: nvidia
    command: --model-id sentence-transformers/all-MiniLM-L6-v2
    ports:
      - "8081:80"
    restart: unless-stopped
    deploy:
      resources:
        limits:
          memory: 2G

Start Small, Scale Up: Deploy services one at a time. Start with Ollama, verify it works, then add Whisper, then embeddings. This incremental approach makes debugging much easier than launching a full stack and wondering which service is misbehaving.

GPU Scheduling and Resource Contention

When multiple models share the same GPU, the CUDA driver handles scheduling automatically — but not always optimally. A large LLM inference that saturates the GPU can starve a concurrent vision model request, causing latency spikes. For Orin persistent server deployments where consistent latency matters, you need to manage this contention deliberately.

NVIDIA's Multi-Process Service (MPS) allows multiple CUDA processes to share GPU execution resources more efficiently. For Jetson, the equivalent is configuring per-process GPU memory limits and using CUDA streams to interleave operations from different models. In practice, smaller models (embeddings, vision) rarely cause contention issues — it's the large LLM inference that dominates GPU time.

The pragmatic approach: configure your LLM service (Ollama) to use only a portion of GPU memory, leaving headroom for other models. Ollama's OLLAMA_NUM_PARALLEL and context-length settings indirectly control GPU memory usage. For concurrent workloads, reducing max context length from 8192 to 4096 tokens frees significant GPU memory without meaningfully impacting chat quality.

Unified API Gateway

With 4–5 AI services running on different ports, you need a unified entry point. Nginx or Caddy as a reverse proxy provides a single hostname and port, routing requests to the appropriate backend service based on the URL path:

# Caddyfile — unified AI API gateway
ai.yourdomain.com {
  # Route /chat to Ollama
  handle /chat* {
    reverse_proxy ollama:11434
  }
  # Route /transcribe to Whisper
  handle /transcribe* {
    reverse_proxy whisper:9000
  }
  # Route /embeddings to sentence transformer
  handle /embeddings* {
    reverse_proxy embeddings:80
  }
  # Route /vision to YOLO
  handle /vision* {
    reverse_proxy yolo:8000
  }
}

This unified gateway pattern is essential for applications that need to orchestrate across multiple models — for example, a voice assistant that transcribes with Whisper, finds relevant context with embeddings, generates a response with the LLM, and optionally analyzes camera input with the vision model. The application calls a single endpoint and the gateway handles routing.

Real-World Multi-Model Scenarios

Here are three production-grade multi-model configurations we've tested and documented:

Scenario A: Personal AI Assistant

LLM (Llama 3.1 8B) + Whisper (large-v3) + Embeddings (all-MiniLM-L6). The LLM handles conversation, Whisper transcribes voice input, and embeddings enable semantic search across personal documents. Memory usage: ~11 GB. Runs comfortably on NX 16GB.

Scenario B: Smart Home Hub

LLM (Phi-4-mini) + YOLOv8-nano + Whisper (medium) + Embeddings. Phi-4-mini is fast enough for natural voice interaction, YOLO monitors camera feeds for people/animals, Whisper handles voice commands, embeddings index documents. Memory usage: ~6 GB. Runs on NX 16GB with room to spare.

Scenario C: Development Lab

LLM (Llama 3.1 8B) + CodeLLM (CodeLlama 7B) + Embeddings + CLIP. Two LLMs for different tasks (general chat, code assistance), embeddings for RAG, CLIP for image understanding. Memory usage: ~15 GB. Requires AGX 64GB for comfortable operation.

These scenarios demonstrate that Jetson Orin guide configurations can scale from simple single-model setups to sophisticated multi-model AI labs. The Orin AI server path extends these patterns to serve multiple users simultaneously.

ClawBox Multi-Model Experience

The ClawBox vs Jetson Orin multi-model comparison is particularly relevant here. ClawBox ships with a pre-configured multi-model stack — LLM, Whisper, embeddings — with optimized memory allocations and tested compatibility. Raw Orin users need to configure this stack manually, which the ClawBox Orin comparison reveals as the primary differentiator: turnkey convenience versus full customization control.

For official guidance on GPU resource management and multi-process CUDA, the NVIDIA Orin developer portal provides detailed documentation on MPS, CUDA streams, and memory management on Jetson platforms.