LiteLLM is an open-source Python library and proxy server that provides a unified OpenAI-compatible API for 100+ LLM providers including Anthropic, Google Gemini, Mistral, Cohere, Together AI, Groq, Replicate, Bedrock, and more. You write code once using the completion() function and switch providers by changing a single model string.

How does LiteLLM Proxy Server work?

The LiteLLM Proxy Server launches a local HTTP server that exposes an OpenAI-compatible /v1/chat/completions endpoint. You configure provider API keys in a config.yaml file, then point any OpenAI SDK or curl request at http://localhost:4000 instead. The proxy handles routing, load balancing, retries, fallbacks, and cost tracking transparently.

Does LiteLLM support async requests?

Yes. LiteLLM provides acompletion() for async Python code using asyncio. You can run dozens of concurrent LLM calls in parallel without blocking, which is essential for batch processing, embedding pipelines, and high-throughput API servers.

How does LiteLLM compare to OpenRouter?

LiteLLM is a self-hosted Python library and proxy — you manage the infrastructure and API keys, but you pay provider rates with no markup. OpenRouter is a managed cloud service with a simpler setup but adds a small fee per token and routes through their servers. LiteLLM gives more control, better privacy, and is cheaper at scale; OpenRouter is faster to get started with.

LiteLLM LLM Proxy Python 9 min read

LiteLLM Guide 2025: Unified API for 100+ LLMs

LiteLLM lets you call OpenAI, Anthropic, Google Gemini, Mistral, Groq, Cohere, Together AI, Replicate, and 90+ more providers through one consistent Python interface. Switch providers by changing a single string — no SDK rewrites, no credential sprawl, no vendor lock-in.

What Is LiteLLM?

LiteLLM is an open-source Python library (and optional proxy server) that normalises the request/response format across every major LLM provider. Instead of learning Anthropic's messages.create(), Google's generate_content(), and Cohere's chat() separately, you call one function — litellm.completion() — with an OpenAI-style payload. LiteLLM translates it under the hood.

Key capabilities: unified completion() and acompletion() across 100+ providers, a self-hosted OpenAI-compatible proxy server, a Router class for load balancing and fallbacks, per-request cost tracking, and budget limits.

Install LiteLLM

pip install litellm

Python SDK Quickstart: completion()

The completion() function mirrors the OpenAI Chat Completions API. The only thing you change when switching providers is the model string:

quickstart.py

import litellm

# OpenAI
response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Anthropic Claude — same call, different model string
response = litellm.completion(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Google Gemini
response = litellm.completion(
    model="gemini/gemini-2.0-flash",
    messages=[{"role": "user", "content": "Hello!"}],
)

# Groq (ultra-fast inference)
response = litellm.completion(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(response.choices[0].message.content)

Set provider API keys as environment variables: OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, GROQ_API_KEY, etc. LiteLLM picks them up automatically.

LiteLLM Proxy Server

The LiteLLM Proxy Server launches an OpenAI-compatible HTTP server at http://localhost:4000. Any tool or SDK that works with OpenAI — LangChain, LlamaIndex, AutoGen, raw curl — works with the proxy unchanged. You centralize all API keys and routing logic in one place.

Start the proxy (single model, no config file needed)

pip install 'litellm[proxy]'

# Route to Claude via OpenAI-compatible API
litellm --model anthropic/claude-3-5-sonnet-20241022

# Route to Gemini
litellm --model gemini/gemini-2.0-flash

# Route to Groq
litellm --model groq/llama-3.3-70b-versatile --port 4000

Query the proxy with curl

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet-20241022",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Query the proxy with the OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:4000",
    api_key="any-string",  # proxy handles real keys
)

response = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Summarize LiteLLM in one sentence."}],
)
print(response.choices[0].message.content)

Router: Load Balancing, Fallbacks & Retries

The Router class distributes traffic across multiple model deployments and automatically falls back to alternatives when a provider is unavailable. This is the core of production reliability with LiteLLM:

router_example.py

from litellm import Router

router = Router(
    model_list=[
        {
            "model_name": "gpt-4o",
            "litellm_params": {
                "model": "gpt-4o",
                "api_key": "OPENAI_API_KEY",
            },
        },
        {
            "model_name": "gpt-4o",          # same logical name
            "litellm_params": {
                "model": "anthropic/claude-3-5-sonnet-20241022",
                "api_key": "ANTHROPIC_API_KEY",
            },
        },
    ],
    # Retry up to 3 times on rate-limit errors
    num_retries=3,
    # Fall back to next provider after 2 failures
    fallbacks=[{"gpt-4o": ["groq/llama-3.3-70b-versatile"]}],
    # Routing strategy: least-busy deployment first
    routing_strategy="least-busy",
)

response = router.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

Routing strategies include least-busy, simple-shuffle, latency-based-routing, and usage-based-routing (RPM/TPM aware).

Cost Tracking with completion_cost()

LiteLLM ships with a built-in pricing database for all supported providers. You can calculate the cost of any completion response immediately after the call:

cost_tracking.py

import litellm

response = litellm.completion(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about APIs."}],
)

# Cost in USD
cost = litellm.completion_cost(completion_response=response)
print(f"Cost: ${cost:.6f}")
# e.g. Cost: $0.000375

# Also available: token breakdown
print(response.usage.prompt_tokens)      # input tokens
print(response.usage.completion_tokens)  # output tokens

Budget limits in the proxy config (config.yaml)

general_settings:
  max_budget: 10.00        # hard stop at $10 spend
  budget_duration: "1d"   # reset daily

model_list:
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: os.environ/ANTHROPIC_API_KEY
      max_budget: 5.00     # per-model budget limit

Async Support with acompletion()

For high-throughput use cases — batch processing, concurrent API calls, async web servers — use acompletion(). It is a drop-in async version of completion():

async_example.py

import asyncio
import litellm

async def call_llm(prompt: str, model: str) -> str:
    response = await litellm.acompletion(
        model=model,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

async def main():
    prompts = [
        "Summarize REST APIs in one sentence.",
        "What is a vector database?",
        "Explain token budgets for LLMs.",
    ]

    # Run 3 calls concurrently across different providers
    tasks = [
        call_llm(prompts[0], "gpt-4o"),
        call_llm(prompts[1], "anthropic/claude-3-5-haiku-20241022"),
        call_llm(prompts[2], "groq/llama-3.3-70b-versatile"),
    ]

    results = await asyncio.gather(*tasks)
    for result in results:
        print(result)

asyncio.run(main())

Using asyncio.gather() with acompletion() fires all requests in parallel — wall-clock time equals the slowest single call, not the sum of all calls.

LiteLLM vs Direct SDKs vs OpenRouter

Option	Setup	Cost	Multi-provider	Fallbacks
LiteLLM	pip install, self-hosted	Provider rates, no markup	100+ providers	Built-in Router
Direct SDKs	One SDK per provider	Provider rates, no markup	1 per SDK	Manual code
OpenRouter	API key only, managed	Provider rates + small fee	100+ models	Automatic
AWS Bedrock	AWS account + IAM	Provider rates via AWS	Bedrock models only	Manual

LiteLLM is the best choice when you want full control, zero markup, and a single codebase that works across every provider. OpenRouter wins if you want managed infrastructure with minimal setup.

Common Use Cases

Four patterns that LiteLLM is purpose-built for:

Model switching for experiments — benchmark GPT-4o vs Claude vs Gemini on the same eval suite by changing the model string in a loop. No SDK rewrites, no format conversions.
Fallback chains for reliability — primary provider goes down at 2 AM? Router automatically retries with Groq or Anthropic. Your users see no downtime.
Cost optimization — route simple tasks to cheaper models (Haiku, Gemini Flash, Llama 3.3 70B via Groq) and complex tasks to premium ones. Track spend per request with completion_cost().
Drop-in OpenAI replacement — teams already using OpenAI SDK can point base_url at the LiteLLM proxy and instantly gain access to every other provider without changing application code.

Monitor the 100+ AI APIs LiteLLM Routes To

Monitor the 100+ AI APIs LiteLLM routes to — Prismix tracks real-time status for all of them. When a provider goes down, your LiteLLM fallback chains activate automatically — know before your users do.

Check AI Provider Status →