OpenAI API Streaming Not Working? Fix SSE & Stream Errors

Add retry logic that detects stream failures (ERR_STREAM_PREMATURE_CLOSE, fetch abort, timeout) and retries with exponential backoff. Generate a unique idempotency key per user request so a retry never duplicates tokens already displayed. Only retry on APIConnectionError or network-level errors — not on content errors like content_filter.

// Node.js / TypeScript
import OpenAI from 'openai';
import { v4 as uuidv4 } from 'uuid';

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function streamWithRetry(messages, maxRetries = 3) {
  const idempotencyKey = uuidv4();
  let lastError;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const stream = await client.chat.completions.create({
        model: 'gpt-4o',
        messages,
        stream: true,
      });

      for await (const chunk of stream) {
        const content = chunk.choices[0]?.delta?.content;
        if (content) process.stdout.write(content);
      }
      return; // success
    } catch (error) {
      lastError = error;
      const retryable =
        error instanceof OpenAI.APIConnectionError ||
        error.code === 'ERR_STREAM_PREMATURE_CLOSE';
      if (retryable && attempt < maxRetries - 1) {
        const delayMs = Math.min(500 * Math.pow(2, attempt) + Math.random() * 500, 30000);
        console.error(`Stream failed (attempt ${attempt + 1}), retrying in ${delayMs}ms`);
        await new Promise(r => setTimeout(r, delayMs));
      } else {
        throw error;
      }
    }
  }
  throw lastError;
}

Configure all intermediary layers — load balancers, reverse proxies, CDNs, firewalls — with timeouts of at least 90 seconds (120+ recommended). If failures occur at exactly 5 minutes in Python or 10 minutes in Node.js, an intermediary timeout is almost certainly the cause. Also set the OpenAI client timeout explicitly on both platforms.

// Node.js: explicit client timeout
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  timeout: 90_000, // 90 seconds in milliseconds
});

# Python: explicit timeout via httpx
import httpx
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),
    timeout=httpx.Timeout(90.0),
)

# Nginx — add to the location block that proxies to your app
proxy_read_timeout    120s;
proxy_connect_timeout  30s;
proxy_send_timeout     30s;

Each new OpenAI() call creates a new internal connection pool. Creating a client per request or per route handler exhausts system file descriptors and connection limits under load. Create one client at module level and reuse it across your entire application.

// lib/openai-client.ts — create once, import everywhere
import OpenAI from 'openai';

export const openaiClient = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  timeout: 90_000,
  maxRetries: 2,
});

// In route handlers:
// import { openaiClient } from './lib/openai-client';

# openai_client.py — module-level singleton
import os
from openai import OpenAI

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# In handlers:
# from openai_client import client

finish_reason may be null until the final chunk, and may be absent entirely when content filters are active or max_tokens is exceeded. Never treat a missing finish_reason as a fatal error. Detect stream completion via the [DONE] sentinel or a clean connection close, and log the null case with context for diagnostics.

// Node.js: safe finish_reason handling
async function streamChatCompletion(messages) {
  const client = new OpenAI();
  let fullResponse = '';
  let finishReason = null;

  const stream = await client.chat.completions.create({
    model: 'gpt-4o',
    messages,
    stream: true,
  });

  for await (const event of stream) {
    const choice = event.choices[0];
    if (choice.delta?.content) {
      fullResponse += choice.delta.content;
      process.stdout.write(choice.delta.content);
    }
    // finish_reason is null on all but the last chunk
    if (choice.finish_reason) finishReason = choice.finish_reason;
  }

  if (!finishReason) {
    // Not fatal — log for diagnostics and continue
    console.warn('Stream completed without finish_reason', {
      responseLength: fullResponse.length,
      timestamp: new Date().toISOString(),
    });
    finishReason = 'unknown';
  }

  return { fullResponse, finishReason };
}

Track stream duration, chunk latency, error rate, and timeout frequency. Set alerts for elevated rates of ERR_STREAM_PREMATURE_CLOSE. Check prismix.dev/service/openai to determine whether a spike is API-wide or isolated to your infrastructure.

// Minimal stream monitoring wrapper
async function streamWithMonitoring(messages) {
  const start = Date.now();
  let chunkCount = 0;
  let totalBytes = 0;

  try {
    const stream = await client.chat.completions.create({
      model: 'gpt-4o', messages, stream: true,
    });
    for await (const chunk of stream) {
      const content = chunk.choices[0]?.delta?.content || '';
      chunkCount++;
      totalBytes += Buffer.byteLength(content);
      process.stdout.write(content);
    }
    console.log('Stream OK', { ms: Date.now() - start, chunkCount, totalBytes });
  } catch (error) {
    console.error('Stream failed', {
      code: error.code,
      ms: Date.now() - start,
      chunkCount,
      totalBytes,
    });
    throw error;
  }
}