NVIDIA NIM Guide 2025: Deploy AI Models with Inference Microservices

What Is NVIDIA NIM?

NIM stands for NVIDIA Inference Microservices. Each NIM is a Docker container built by NVIDIA that bundles everything needed to serve an AI model at peak GPU efficiency: TensorRT-LLM for LLMs, ONNX Runtime or Triton Inference Server for vision and speech, GPU-specific TensorRT engine profiles, and an OpenAI-compatible REST API on port 8000.

NIMs are available for four model families:

• LLMs — Llama 3.1/3.3, Mistral, Nemotron, Phi-3, DeepSeek
• Embedding models — NV-EmbedQA-E5-v5, NV-EmbedQA-Mistral7B
• Vision models — PaliGemma, Llama-3.2-Vision, Neva-22B
• Speech models — Riva ASR (speech-to-text), Riva TTS (text-to-speech)

NIMs are distributed via the NVIDIA NGC container registry (nvcr.io/nim/) and require an NGC API key from build.nvidia.com.

NIM vs vLLM vs TGI — Key Differences

All three serve LLMs with OpenAI-compatible APIs. The differences are in who builds and owns the stack:

NIM's key advantage: NVIDIA ships pre-compiled TensorRT-LLM engines for each GPU model (H100 SXM, H100 PCIe, A100, L40S, etc.), so you never rebuild TensorRT engines yourself. vLLM is more flexible but requires manual tuning to reach equivalent throughput.

Self-Hosted Quickstart: Docker Pull & Run

You need an NVIDIA GPU (Ampere or newer — A100, H100, RTX 4090, L40S, A10G), Docker with NVIDIA Container Toolkit, and an NGC API key from build.nvidia.com.

export NGC_API_KEY=nvapi-xxxxxxxxxxxxxxxxxxxx

docker login nvcr.io \
  --username '$oauthtoken' \
  --password $NGC_API_KEY

docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

docker run -it --rm \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -v /path/to/nim-cache:/opt/nim/.cache \
  -p 8000:8000 \
  nvcr.io/nim/meta/llama-3.1-8b-instruct:latest

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-8b-instruct",
    "messages": [{"role": "user", "content": "Hello!"}],
    "max_tokens": 128
  }'

The first run downloads the TensorRT engine profile for your GPU and caches it in /opt/nim/.cache. Subsequent starts are fast. Mount the cache directory to avoid re-downloading on container restarts.

NVIDIA Cloud Endpoints — Free API Access at build.nvidia.com

Don't have a GPU? NVIDIA hosts NIM-backed endpoints at build.nvidia.com with a free tier (1000 credits/month). The API surface is identical to a self-hosted NIM — same OpenAI SDK, same model names, different base_url.

curl https://integrate.api.nvidia.com/v1/chat/completions \
  -H "Authorization: Bearer $NGC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama-3.1-70b-instruct",
    "messages": [{"role": "user", "content": "Explain TensorRT in one paragraph."}],
    "max_tokens": 256,
    "stream": false
  }'

The cloud endpoint supports 100+ models including embedding and vision NIMs, accessible without any GPU provisioning.

Python Quickstart with the OpenAI SDK

NIM's OpenAI-compatible API means you only change base_url to switch between self-hosted and NVIDIA cloud:

from openai import OpenAI
import os

# Self-hosted NIM
client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",
)

# NVIDIA cloud NIM (swap base_url + api_key)
# client = OpenAI(
#     base_url="https://integrate.api.nvidia.com/v1",
#     api_key=os.environ["NGC_API_KEY"],
# )

response = client.chat.completions.create(
    model="meta/llama-3.1-8b-instruct",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is TensorRT-LLM?"},
    ],
    temperature=0.2,
    max_tokens=512,
    stream=True,
)

for chunk in response:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

from openai import OpenAI

client = OpenAI(
    base_url="https://integrate.api.nvidia.com/v1",
    api_key=os.environ["NGC_API_KEY"],
)

response = client.embeddings.create(
    input=["NVIDIA NIM simplifies GPU inference deployment."],
    model="nvidia/nv-embedqa-e5-v5",
    encoding_format="float",
    extra_body={"input_type": "query", "truncate": "END"},
)

print(response.data[0].embedding[:8])  # First 8 dims

Available Models in the NIM Catalog

The full catalog lives at build.nvidia.com/explore/discover. Key NIMs by category:

Performance: TensorRT-LLM Optimizations Inside NIM

The throughput advantage of NIM comes from TensorRT-LLM — NVIDIA's compiler that fuses operations, selects optimal kernels per GPU, and applies these optimizations automatically:

• FP8 quantization — native on H100/H200, near-lossless quality vs BF16, 2x throughput uplift
• INT4 AWQ quantization — for A100/L40S, fits 70B models in 40 GB VRAM
• Continuous batching — same mechanism as vLLM, saturates GPU compute across concurrent requests
• Speculative decoding — draft model proposes tokens, target model verifies in parallel, 2-3x latency reduction
• In-flight batching — new requests join an active batch at token boundaries, not batch boundaries
• Paged KV cache — identical to vLLM's PagedAttention, eliminates KV cache fragmentation

# NIM with TensorRT-LLM FP8 on H100 SXM
Throughput: ~3,800 tokens/sec (output)  @ batch=32

# vLLM BF16 on H100 SXM (same model)
Throughput: ~2,200 tokens/sec (output)  @ batch=32

# NIM advantage: ~1.7x on H100, up to 2x on A100 FP8

Exact numbers vary by model size, sequence length, and batch size. NIM auto-selects the best engine profile for your GPU SKU at container startup.

Enterprise Use Cases: Air-Gapped, HIPAA & SOC2

NIM's container packaging makes it well-suited for enterprise environments with strict data governance requirements:

• Air-gapped on-prem deployment — pull the container image and model weights to a private registry, then run with no external network access. NGC credentials are only needed at pull time.
• HIPAA / SOC2 certified infrastructure — when deployed on certified on-prem hardware (DGX systems, DGX Cloud) no PHI or PII leaves your network. NVIDIA provides compliance documentation for enterprise agreements.
• Latency-critical applications — TensorRT-LLM's kernel fusion reduces decode latency to 15-30 ms TTFT (time to first token) on H100 for 8B models, enabling real-time streaming UIs.
• Private model fine-tunes — NIM supports custom model weights (via NeMo Customizer or direct weight injection) so you can serve fine-tuned checkpoints behind the same NIM API contract.

# On an internet-connected machine:
docker pull nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
docker tag nvcr.io/nim/meta/llama-3.1-8b-instruct:latest \
  registry.internal.corp/nim/llama-3.1-8b-instruct:latest
docker push registry.internal.corp/nim/llama-3.1-8b-instruct:latest

# On the air-gapped server (no internet required):
docker pull registry.internal.corp/nim/llama-3.1-8b-instruct:latest
docker run --gpus all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  registry.internal.corp/nim/llama-3.1-8b-instruct:latest