vLLM Guide 2025: Fast & Efficient LLM Serving Framework

What Is vLLM?

vLLM (from UC Berkeley Sky Computing Lab) is an open-source LLM serving library optimized for high throughput and memory efficiency. Unlike naive inference that allocates a fixed KV-cache block per request, vLLM's PagedAttention manages KV cache like an OS manages virtual memory — allocating pages dynamically and sharing them across requests. The result: near-zero memory waste and dramatically higher concurrency.

pip install vllm

Key features: PagedAttention memory management, continuous batching, CUDA graphs for fast decoding, speculative decoding, tensor parallelism across multiple GPUs, and an OpenAI-compatible REST API server out of the box.

Why vLLM Is Fast: PagedAttention & Continuous Batching

Three techniques stack to produce vLLM's throughput advantage:

• PagedAttention — KV cache stored in non-contiguous memory pages (like OS virtual memory). Requests share physical pages for the same prompt prefix (prefix caching). Fragmentation drops to near zero.
• Continuous batching — Rather than waiting to fill a static batch before decoding, vLLM inserts new requests into an ongoing batch at token boundaries. GPU utilization stays high even under variable load.
• CUDA graphs — The decode step is compiled into a CUDA graph on first run, eliminating Python overhead on subsequent tokens. Decoding speed increases 2-3x for small batch sizes.

Quickstart: Offline Batch Inference

The LLM class is the simplest entry point — load a model and run batch inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)

prompts = [
    "Explain transformer attention in one paragraph.",
    "Write a Python function to reverse a linked list.",
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

Requires a HuggingFace token for gated models: huggingface-cli login

OpenAI-Compatible API Server

The vllm serve command launches an HTTP server compatible with the OpenAI Chat Completions API. Any client using the OpenAI SDK can point at it with base_url:

# Serve Llama 3.1 8B
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 8192

# Serve Mistral 7B
vllm serve mistralai/Mistral-7B-Instruct-v0.3

# Serve DeepSeek V3 (requires 8x A100 or 4x H100)
vllm serve deepseek-ai/DeepSeek-V3 --tensor-parallel-size 8

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed",  # vLLM ignores the key by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)

for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Quantization: AWQ, GPTQ & FP8

Quantization halves or quarters VRAM usage with minimal quality loss, enabling larger models on the same hardware:

# AWQ 4-bit (recommended — fastest + best quality on consumer GPUs)
vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ --quantization awq

# GPTQ 4-bit
vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ --quantization gptq

# FP8 (H100/H200 only — near-lossless quality)
vllm serve meta-llama/Llama-3.1-70B-Instruct --quantization fp8

AWQ cuts Llama 3.1 70B VRAM from 140 GB (BF16) to ~40 GB, making it runnable on 2x A100 40 GB or 1x H100 80 GB.

Tensor Parallelism & Multi-GPU Serving

For models that don't fit on a single GPU, vLLM distributes attention heads and MLP layers across GPUs with a single flag:

# Serve Qwen2-72B across 4 GPUs
vllm serve Qwen/Qwen2-72B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768

# Serve DeepSeek V3 685B across 8 H100s
vllm serve deepseek-ai/DeepSeek-V3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 1

Speculative Decoding

Speculative decoding uses a small draft model to propose multiple tokens at once, which the large target model verifies in parallel. On most workloads this cuts latency 2-3x with no output quality change:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5 \
  --tensor-parallel-size 4

Hardware Requirements

vLLM vs TGI vs Ollama vs llama.cpp

See also: Ollama guide · Llama guide · Modal guide