OpenAI Embeddings API Error? Fix Rate Limits, Token Limits & pgvector Issues

429 Rate limit exceeded — free vs paid tier

The error looks like: "Rate limit reached for text-embedding-3-small in organization org-xxx on tokens per min. Limit: 200,000, Used: 200,000, Requested: 5,000." Rate limits for the Embeddings API differ by tier:

The fastest fix on any tier is to use batching — pass an array of up to 2,048 strings per request instead of calling the API once per document. One batched request uses 1 RPM regardless of how many strings it contains:

from openai import OpenAI
import time

client = OpenAI()

def embed_batch(texts: list[str], model: str = "text-embedding-3-small"):
    """Embed up to 2048 strings in a single API call."""
    response = client.embeddings.create(
        input=texts,   # list of strings, max 2048
        model=model,
    )
    # response.data is ordered the same as input
    return [item.embedding for item in response.data]

def embed_documents(docs: list[str], batch_size: int = 512):
    """Embed a large document set with rate-limit backoff."""
    import random
    from openai import RateLimitError

    embeddings = []
    for i in range(0, len(docs), batch_size):
        batch = docs[i : i + batch_size]
        for attempt in range(6):
            try:
                embeddings.extend(embed_batch(batch))
                break
            except RateLimitError:
                if attempt == 5:
                    raise
                wait = (2 ** attempt) + random.random()
                time.sleep(wait)
    return embeddings

• Batch size: max 2,048 strings per call; stay at 512 to leave headroom for token limits.
• Billing limit 429: different from RPM — go to platform.openai.com/settings/billing/limits and raise monthly spend cap.
• Check tier: platform.openai.com/settings/organization/limits shows your current RPM/TPM ceiling.

Input exceeds 8,191 token limit

The error: "This model's maximum context length is 8191 tokens, however you requested 9342 tokens." Both text-embedding-3-small and text-embedding-3-large cap each individual input at 8,191 tokens. Chunk long documents before embedding:

import tiktoken

def chunk_text(text: str, max_tokens: int = 800, overlap: int = 100) -> list[str]:
    """Split text into overlapping chunks under the token limit."""
    enc = tiktoken.encoding_for_model("text-embedding-3-small")
    tokens = enc.encode(text)

    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))
        if end == len(tokens):
            break
        start += max_tokens - overlap  # overlap for context continuity
    return chunks

# Usage
chunks = chunk_text(long_document)
embeddings = embed_documents(chunks)

• Chunk size: 500–1,000 tokens is the typical range; overlap of 10–20% preserves sentence context across boundaries.
• Install tiktoken: pip install tiktoken — same tokenizer the API uses, so counts are exact.

text-embedding-3-small vs text-embedding-ada-002

text-embedding-3-small is the current recommended model. It replaces text-embedding-ada-002 and adds a dimensions parameter to produce smaller vectors for lower storage and faster search:

# Full 1536-dimension embedding (default)
response = client.embeddings.create(
    input="Your document text here",
    model="text-embedding-3-small",
)
vector = response.data[0].embedding  # length 1536

# Reduced 256-dimension embedding (faster similarity search, less storage)
response = client.embeddings.create(
    input="Your document text here",
    model="text-embedding-3-small",
    dimensions=256,
)
vector = response.data[0].embedding  # length 256

Storing and querying embeddings with pgvector

Set up pgvector in PostgreSQL, create a table with the correct dimension, and query by cosine similarity (<=>) or L2 distance (<->):

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table matching the embedding dimensions you use
-- text-embedding-3-small default: 1536
-- text-embedding-3-small with dimensions=256: 256
CREATE TABLE documents (
    id BIGSERIAL PRIMARY KEY,
    content TEXT NOT NULL,
    embedding vector(1536)
);

-- Index for fast approximate nearest-neighbor (cosine)
CREATE INDEX ON documents
    USING ivfflat (embedding vector_cosine_ops)
    WITH (lists = 100);

-- Cosine similarity search (lower = more similar for <=>)
SELECT id, content,
       1 - (embedding <=> '[0.12, 0.34, ...]'::vector) AS similarity
FROM documents
ORDER BY embedding <=> '[0.12, 0.34, ...]'::vector
LIMIT 5;

• Cosine vs L2: use cosine (<=>) for text similarity — it is robust to vector magnitude differences. L2 (<->) works well only when vectors are normalised.
• Dimension mismatch error: "expected 1536 dimensions, not 256" — the vector(N) column size must exactly match your embedding dimensions. ALTER COLUMN embedding TYPE vector(256) + full re-embed to fix.
• lists parameter: set lists to rows / 1000 for IVFFlat index — run SET ivfflat.probes = 10 at query time for higher recall.