OpenAI API 500 Internal Server Error — How to Fix It

Before writing any retry code, find out whether OpenAI is having a platform-wide problem or whether your 500s are isolated to one request. The fastest signal is the Prismix status dashboard, which polls the OpenAI API and aggregates incidents in real time.

The official openai Python and Node SDKs automatically retry connection errors and 5xx responses (including 500, 502, and 503) with exponential backoff. The default is 2 retries — raise it for production workloads:

# Python — increase SDK retries to 5 (covers most transient 500/502/503 spikes)
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    max_retries=5,   # default is 2; the SDK retries 5xx and connection errors
    timeout=60.0,    # seconds per attempt
)

resp = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

// TypeScript — same pattern
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  maxRetries: 5,
  timeout: 60_000,  // ms
});

With max_retries=5, the SDK waits roughly 1 s, 2 s, 4 s, 8 s, and 16 s (with jitter) before surfacing the error. This alone resolves the vast majority of short 500 waves with zero extra code.

For batch processing or long-running server jobs, write an explicit retry wrapper. It should retry only the server-side 5xx codes, cap the wait time, add jitter, and re-raise client errors (400/401) immediately so a broken request does not loop forever:

import os
import time
import random
from openai import OpenAI, APIStatusError

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"], max_retries=0)

RETRIABLE = {500, 502, 503}  # server-side only

def create_with_backoff(max_attempts: int = 6, **kwargs):
    """Retry OpenAI 5xx server errors with exponential backoff + jitter."""
    for attempt in range(max_attempts):
        try:
            return client.chat.completions.create(**kwargs)
        except APIStatusError as e:
            if e.status_code not in RETRIABLE:
                raise  # 400/401/429 are not server faults — handle separately
            if attempt == max_attempts - 1:
                raise
            wait = min(60.0, (2 ** attempt) + random.uniform(0, 1))
            print(f"OpenAI {e.status_code} server error. Retry {attempt + 1}/{max_attempts - 1} in {wait:.1f}s")
            time.sleep(wait)

# Usage
response = create_with_backoff(
    max_attempts=6,
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
)

// TypeScript async version — catch OpenAI.APIError and check err.status
import OpenAI from 'openai';

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY, maxRetries: 0 });

const RETRIABLE = new Set([500, 502, 503]);

async function createWithBackoff(
  params: OpenAI.ChatCompletionCreateParamsNonStreaming,
  maxAttempts = 6,
) {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      return await client.chat.completions.create(params);
    } catch (e) {
      if (!(e instanceof OpenAI.APIError) || !RETRIABLE.has(e.status ?? 0)) throw e;
      if (attempt === maxAttempts - 1) throw e;
      const wait = Math.min(60_000, (2 ** attempt) * 1000 + Math.random() * 1000);
      console.warn(`OpenAI ${e.status} — retry ${attempt + 1} in ${(wait / 1000).toFixed(1)}s`);
      await new Promise(r => setTimeout(r, wait));
    }
  }
}

If a 500 reproduces on one specific request while the rest of your traffic succeeds, it is almost certainly the request, not an outage. An extremely large prompt, a huge max_tokens, or a malformed function / tool schema can push the backend into an internal error instead of a clean 400. Shrink the request and stream long outputs to isolate the trigger:

# A single giant, buffered request is the most common self-inflicted 500.
# Prefer smaller, streamed calls over one enormous completion.
stream = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    max_tokens=1024,   # avoid pathologically large output caps
    stream=True,       # stream long output instead of buffering it all
)
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="")

• Validate tool/function schemas: a malformed tools definition can surface as a 500 rather than a 400 — double-check the JSON Schema.
• Trim the prompt: if you are near the model's context window, a 500 can appear before you hit a clean context-length error. Cut input tokens and retry.
• Test the request in isolation: replay the exact failing payload with a tiny max_tokens. If it still 500s alone, the payload is the cause.

During a prolonged 500/503 incident, retrying every request just queues up a backlog and hammers a backend that is already failing. A circuit breaker stops sending requests after a failure threshold, waits a fixed recovery window, then probes once before resuming:

import time
from openai import APIStatusError

class CircuitBreaker:
    def __init__(self, failure_threshold=3, recovery_timeout=60):
        self.failures = 0
        self.threshold = failure_threshold
        self.timeout = recovery_timeout
        self.opened_at = None
        self.state = "closed"  # closed | open | half-open

    def record_failure(self):
        self.failures += 1
        if self.failures >= self.threshold:
            self.state = "open"
            self.opened_at = time.monotonic()
            print(f"Circuit OPEN — pausing OpenAI calls for {self.timeout}s")

    def record_success(self):
        self.failures = 0
        self.state = "closed"

    def allow_request(self) -> bool:
        if self.state == "closed":
            return True
        elapsed = time.monotonic() - self.opened_at
        if elapsed >= self.timeout:
            self.state = "half-open"
            return True  # probe request
        return False

breaker = CircuitBreaker()

def safe_create(**kwargs):
    if not breaker.allow_request():
        raise RuntimeError("OpenAI circuit open — skipping request")
    try:
        result = client.chat.completions.create(**kwargs)
        breaker.record_success()
        return result
    except APIStatusError as e:
        if e.status_code in (500, 502, 503):
            breaker.record_failure()
        raise