HuggingFace 503: Model is Currently Loading

What the error looks like

When you hit the HuggingFace Inference API and the model is not yet loaded, you get a HTTP 503 with a JSON body:

HTTP/1.1 503 Service Unavailable

{
  "error": "Model meta-llama/Llama-3-8b-hf is currently loading",
  "estimated_time": 47.3
}

The key field is estimated_time — it is the number of seconds until the model is ready. This is not an outage. It is a normal cold-start on the shared free tier. The fix is to wait and retry.

Step 1 — Poll until ready with estimated_time

Do not retry immediately — it will still be loading. Read estimated_time, sleep that many seconds, then retry. Cap at 10 attempts for large models that take several minutes.

import requests
import time

API_URL = "https://api-inference.huggingface.co/models/meta-llama/Llama-3-8b-hf"
HEADERS = {"Authorization": "Bearer hf_YOUR_TOKEN"}

def query_with_retry(payload, max_retries=10):
    for attempt in range(max_retries):
        response = requests.post(API_URL, headers=HEADERS, json=payload)

        if response.status_code == 200:
            return response.json()

        if response.status_code == 503:
            data = response.json()
            wait_seconds = data.get("estimated_time", 20)
            print(f"Model loading, waiting {wait_seconds:.0f}s (attempt {attempt + 1}/{max_retries})...")
            time.sleep(wait_seconds + 2)  # add 2s buffer
            continue

        # Other errors (401, 429, etc.) — raise immediately
        response.raise_for_status()

    raise TimeoutError("Model did not load after max retries")

result = query_with_retry({"inputs": "Hello, how are you?"})
print(result)

The + 2 buffer accounts for the estimate being slightly optimistic. For models over 30B, set max_retries=20 since loading can take 5–10 minutes.

Step 2 — Use wait_for_model to skip polling code

HuggingFace's huggingface_hub Python library and the raw HTTP API both support a parameter that tells the server to block until the model is ready instead of returning 503 immediately:

from huggingface_hub import InferenceClient

client = InferenceClient(
    model="meta-llama/Llama-3-8b-hf",
    token="hf_YOUR_TOKEN"
)

# wait_for_model=True blocks until the model is warm (up to ~10 min)
result = client.text_generation(
    "Explain transformers in one sentence.",
    wait_for_model=True,
    timeout=600
)
print(result)

For raw HTTP, pass the parameter in the request body:

payload = {
    "inputs": "Explain transformers in one sentence.",
    "options": {"wait_for_model": True}
}
response = requests.post(API_URL, headers=HEADERS, json=payload, timeout=600)

Step 3 — Know loading times by model size

Loading time is dominated by how long it takes to move weights from storage into GPU VRAM. Rough estimates on the free shared tier:

For 70B+ models on the free shared API, cold starts are impractical for production use. See Step 5 for dedicated alternatives.

Step 4 — Pick always-warm models to avoid 503 entirely

HuggingFace keeps a set of high-traffic models warm at all times on the free tier. These never return a loading 503:

To verify a model is hosted on the free API: open its model card on huggingface.co and look for the "Hosted inference API" panel on the right sidebar. If it shows an interactive widget, the model is available. If it says "This model is not currently available via the Inference API", it must be self-hosted or used via Endpoints.

Step 5 — Inference API vs Inference Endpoints: which to use

Create a Dedicated Endpoint at ui.endpoints.huggingface.co. The endpoint URL is https://[id].us-east-1.aws.endpoints.huggingface.cloud/ — use it identically to the shared API, but with no 503 cold starts.

Step 6 — Use Groq or Together.ai for zero-latency open model inference

For Llama 3, Mistral, Mixtral, and Gemma, third-party hosted inference providers eliminate cold starts entirely and are often faster than HuggingFace Endpoints:

Both use the OpenAI SDK — just swap the base URL and API key:

# Groq — drop-in replacement for HuggingFace Inference API
from groq import Groq

client = Groq(api_key="gsk_YOUR_GROQ_KEY")
response = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Explain transformers"}]
)
print(response.choices[0].message.content)

# Together.ai — same OpenAI-compatible interface
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_TOGETHER_KEY",
    base_url="https://api.together.xyz/v1"
)
response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Explain transformers"}]
)
print(response.choices[0].message.content)

Frequently asked questions

Related AI API guides

HuggingFace not working Groq API not working OpenAI API not working Ollama not working