OpenAI Context Length Exceeded — How to Fix It

1. Count tokens before you send

Install tiktoken — OpenAI's official tokenizer — and measure your prompt before it hits the API. This lets you catch the error locally with no API call wasted.

pip install tiktoken

import tiktoken

def count_tokens(messages: list, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    total = 3  # every reply is primed with 3 tokens
    for msg in messages:
        total += 4  # role/name/content envelope overhead
        for value in msg.values():
            total += len(enc.encode(value))
    return total

MODEL = "gpt-4o"
CONTEXT_LIMIT = 128_000
MAX_COMPLETION = 4_096

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in detail..."},
]

prompt_tokens = count_tokens(messages, MODEL)
available_for_completion = CONTEXT_LIMIT - prompt_tokens
print(f"Prompt tokens: {prompt_tokens}")
print(f"Available for completion: {available_for_completion}")

Always set max_tokens to at most available_for_completion to avoid the error even when your prompt is large.

2. Truncate conversation history

Long multi-turn chats accumulate tokens fast. Always keep the system prompt. Trim from the oldest user/assistant turns first until you fit within budget:

def trim_messages(messages: list, model: str, token_budget: int) -> list:
    """Keep system prompt + most-recent turns that fit in token_budget."""
    enc = tiktoken.encoding_for_model(model)
    system_msgs = [m for m in messages if m["role"] == "system"]
    history = [m for m in messages if m["role"] != "system"]

    def token_count(msgs):
        total = 3
        for m in msgs:
            total += 4 + sum(len(enc.encode(v)) for v in m.values())
        return total

    # Drop from the front of history until we fit
    while history and token_count(system_msgs + history) > token_budget:
        history.pop(0)

    return system_msgs + history

BUDGET = 128_000 - 4_096  # leave 4K for the reply
trimmed = trim_messages(full_conversation, "gpt-4o", BUDGET)

3. Summarize old history instead of dropping it

For chatbots where you cannot drop context, compress older turns into a summary before the live window:

def summarize_old_turns(client, old_turns: list) -> str:
    text = "\n".join(
        f"{m['role']}: {m['content']}" for m in old_turns
    )
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize the conversation below concisely."},
            {"role": "user", "content": text},
        ],
        max_tokens=512,
    )
    return resp.choices[0].message.content

# Usage: when history grows too large, summarize the oldest half
if count_tokens(messages, "gpt-4o") > 100_000:
    mid = len(messages) // 2
    summary = summarize_old_turns(client, messages[1:mid])
    messages = [
        messages[0],  # system prompt
        {"role": "assistant", "content": f"[Summary of earlier conversation: {summary}]"},
        *messages[mid:],  # recent turns
    ]

4. Chunk long documents

When processing large files — PDFs, codebases, transcripts — split them into overlapping chunks and call the API per chunk. A 10–15% overlap preserves context at chunk boundaries:

import tiktoken

def chunk_text(text: str, model: str = "gpt-4o",
               chunk_tokens: int = 2000, overlap_tokens: int = 200) -> list[str]:
    enc = tiktoken.encoding_for_model(model)
    token_ids = enc.encode(text)
    chunks = []
    start = 0
    while start < len(token_ids):
        end = min(start + chunk_tokens, len(token_ids))
        chunks.append(enc.decode(token_ids[start:end]))
        start += chunk_tokens - overlap_tokens
    return chunks

def process_document(client, document: str, question: str) -> list[str]:
    chunks = chunk_text(document)
    answers = []
    for i, chunk in enumerate(chunks):
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": "Answer the question using only the provided text."},
                {"role": "user", "content": f"Text:\n{chunk}\n\nQuestion: {question}"},
            ],
            max_tokens=512,
        )
        answers.append(resp.choices[0].message.content)
    return answers

For large-scale retrieval, use a vector store (Pinecone, Qdrant, pgvector) to embed chunks and retrieve only the most relevant ones — this scales beyond what fits in any context window.

5. Understand context window vs max_tokens

These two parameters are often confused:

# Always compute safe max_tokens dynamically:
prompt_tokens = count_tokens(messages, "gpt-4o")
safe_max_tokens = min(4096, 128_000 - prompt_tokens - 10)  # 10 token buffer

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    max_tokens=safe_max_tokens,
)