OpenAI streaming hanging in production: the SSE runbook (2026)

Last verified: July 23, 2026

Short path: Start with the production LLM API guide, compare current gateway rates on AI API pricing, or review LumeAPI’s production plan.

Your OpenAI streaming response probably is not “randomly hanging.” In production, a stream that stops mid-token usually comes from one of four places: a client read timeout, a proxy buffering response bytes, a broken SSE parser, or a retry/cancellation path that leaves the original request alive. The fix is to measure time to first byte and time between chunks separately, then enforce an inter-chunk deadline instead of one oversized request timeout.

Quick Answer

In short

The useful fix for openai streaming hanging is not “increase the timeout to ten minutes.” Separate connection, first-byte, and inter-chunk timeouts. A stream that has delivered 40 tokens and then goes silent needs a different response from a request that has delivered no bytes at all. Put the deadline at the byte-reading layer, disable proxy buffering for SSE, and preserve enough state to avoid replaying a tool call twice.

What most guides get wrong

The common advice is to increase the HTTP timeout. That treats every failure as slow model generation.

A request can be connected successfully, receive its first token, and then sit silent because an intermediary is buffering the response. It can also sit silent because the upstream is still working, because the client is reading the wrong stream, or because your parser is waiting for \n\n while the transport has only delivered half of a UTF-8 sequence. Those cases look identical in an application log that records only “request started” and “request finished.”

A single 300-second timeout hides the distinction. It also keeps sockets, worker slots, and user requests occupied while your queue grows.

Measure three clocks:

1. Connect time: how long until the HTTP request is established.
2. Time to first byte: how long until any response bytes arrive.
3. Inter-chunk idle time: the longest acceptable silence after streaming begins.

The third clock is the one most production clients omit. It is also the one that catches a stalled connection without killing a legitimately slow initial response.

A realistic production scenario

A six-person support automation team runs Node.js workers behind Nginx and a Kubernetes ingress. Their endpoint uses an OpenAI-compatible client with streaming enabled. The UI receives text correctly in local development, but in production the answer appears frozen for 45 seconds and then arrives in one large burst.

The team initially raises the SDK timeout from 60 to 300 seconds. That makes the dashboard look healthier while tying up more worker memory. A packet capture shows the upstream is sending response bytes; the application does not see them until the proxy’s response buffer fills. On another route, the proxy is not the problem: the client receives 800 bytes, then waits forever because its code resets no timer after the first chunk.

The turning point is a three-field log entry: first_byte_ms, last_chunk_gap_ms, and bytes_received. After disabling buffering on the streaming route and adding a 30-second inter-chunk deadline, the team can distinguish “proxy hid the stream” from “upstream stopped sending.” They stop retrying every timeout automatically, which matters because some requests also contain tool calls.

Expert take

SSE streaming is a chain, not a single connection:

model/provider → gateway → ingress → reverse proxy → application server → client

Every hop can alter the timing. A proxy can buffer chunks. A load balancer can close an idle connection. Your web framework can wait for the full upstream body before writing to the browser. A client library can expose a high-level iterator that has no per-read timeout. Your parser can discard events because it assumes one network read equals one SSE event.

That last assumption is wrong. TCP reads do not preserve application message boundaries. One read may contain several data: events, or one event may be split over several reads. SSE fields are text lines, and events are separated by a blank line. Parse an accumulator, not individual chunks. Use a streaming UTF-8 decoder so a multibyte character split across reads is not corrupted.

The operational rule is simple:

• Allow a separate, usually longer deadline for the first byte.
• Once bytes arrive, require progress within an inter-chunk window.
• Cancel the request when that window expires.
• Retry only when the operation is safe to replay.

Do not follow this advice blindly for every workload. A long-running reasoning request may have a legitimate period before its first token, so an aggressive first-byte timeout creates false failures. Conversely, an agent tool call may produce a partial stream and then execute a side effect. Replaying it after a client timeout can create a duplicate ticket, payment, email, or database mutation. Streaming improves perceived latency; it does not make generation transactional.

LumeAPI is an independent third-party gateway, not OpenAI, Anthropic, or Google. Moving the base URL can help you compare model access and token rates, but it does not provide automatic parity with every provider-native feature. Validate streaming, tool calling, usage reporting, and cancellation behavior in your own integration before switching production traffic.

First, reproduce the hang outside your application

Start with the smallest possible request. This separates model, network, and application behavior.

curl -N --fail-with-body \
  --connect-timeout 10 \
  --max-time 180 \
  https://api.lumeapi.site/v1/chat/completions \
  -H "Authorization: Bearer $LUMEAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-terra",
    "messages": [
      {"role": "user", "content": "Reply with five short words."}
    ],
    "stream": true
  }'

-N tells curl not to buffer its output. The command is useful even when your production application calls OpenAI directly: run the same request against the direct provider and against the gateway, then from the same network location as the failing worker.

Record:

• Whether the first data: line appears.
• The time to the first byte.
• The largest gap between visible events.
• Whether the response ends with data: [DONE].
• Whether the TCP connection closes without a terminal event.
• The HTTP status and response headers.

Do not treat [DONE] as the only valid indicator of a completed user-visible answer. A connection can terminate after enough content has arrived to display a partial answer. Your application should record the stream as interrupted, not silently mark it successful.

The official OpenAI streaming guide is the reference point for the provider’s streaming format. For the wire-level event model, also consult the MDN Server-sent events documentation.

Diagnose the layer that is hanging

1. No response bytes ever arrive

This is a connect, DNS, TLS, authentication, routing, or upstream availability problem—not an SSE parsing problem.

Check the status code and headers before entering the body loop. A 401, 403, 404, or 429 may be hidden by a wrapper that reports only “stream stopped.” Log the request URL hostname, selected model, status, and a redacted request ID. Never log the bearer token or full prompt by default.

If direct OpenAI traffic works but the LumeAPI request does not, compare:

• Base URL, including the /v1 path.
• Authorization header.
• Model identifier.
• Whether the request is actually sent with stream: true.
• Egress firewall and DNS rules for the worker environment.
• Proxy environment variables such as HTTPS_PROXY and NO_PROXY.

A compatible endpoint still requires a compatible request shape. “The client initialized successfully” does not prove that it is calling the intended host.

2. The first byte arrives, then silence exceeds your limit

This is the classic openai api stream stuck production case. The upstream may be slow, the connection may be half-open, or an intermediary may have stopped forwarding bytes.

Log every received byte batch while debugging, not only parsed tokens. A parser can be wrong while the transport is healthy. Capture:

request_id
status
first_byte_ms
bytes_received
sse_events_parsed
last_chunk_at
max_inter_chunk_gap_ms
stream_end_reason

Use monotonic time for these measurements. Wall-clock time can jump when the host synchronizes its clock.

A read timeout should be reset after each successful read, not only after the request begins. If the socket returns a few bytes every 20 seconds, a total request timeout may be the appropriate control. If it returns nothing for 30 seconds, an inter-chunk timeout should terminate it.

3. Bytes arrive at the proxy but not at the browser

This points to buffering. The application may be receiving events correctly while its response layer waits for a larger body. Confirm by calling the application endpoint from inside the cluster and from outside it.

For an Nginx route, the relevant concepts are disabling response buffering and setting a read timeout long enough for your expected idle window. The exact configuration belongs in the proxy’s current documentation; see Nginx’s proxy module reference. Do not copy a generic setting into every route. A streaming route and a normal JSON route have different buffering needs.

Also inspect:

• CDN caching and compression behavior.
• Kubernetes ingress annotations.
• Framework response flushing.
• HTTP/2 or HTTP/1.1 behavior in the affected hop.
• Whether your server writes each parsed event immediately.
• Whether a middleware collects the response to add a Content-Length.

The downstream response must be treated as a stream all the way to the user. Fixing only the upstream client does not help if your API server collects the full answer before sending it.

4. The network is fine but the UI stops rendering

The UI may be receiving data that the renderer rejects. Common causes include assuming every event contains a content delta, failing on tool-call deltas, or decoding JSON before the complete SSE event has arrived.

Keep transport parsing and semantic parsing separate:

1. Decode bytes into text incrementally.
2. Split complete SSE events.
3. Extract data: lines.
4. Handle [DONE].
5. Parse JSON.
6. Inspect the event shape before reading content.

Do not assume every event has a non-empty text field. Some events may carry metadata, finish information, or tool-call fragments. Treat an empty content value as a valid event, not automatically as a hang.

A production-safe streaming loop

The following Node.js example uses the standard fetch interface rather than a provider-specific SDK. It demonstrates separate first-byte and inter-chunk deadlines, incremental UTF-8 decoding, and SSE event accumulation. Adapt the event field handling to the exact response shapes you use.

const BASE_URL = "https://api.lumeapi.site/v1";

function createDeadline(ms, label) {
  let timer;
  const promise = new Promise((_, reject) => {
    timer = setTimeout(() => {
      const error = new Error(`${label} after ${ms}ms`);
      error.code = label.toUpperCase().replaceAll(" ", "_");
      reject(error);
    }, ms);
  });

  return {
    promise,
    cancel() {
      clearTimeout(timer);
    }
  };
}

async function streamCompletion({ apiKey, messages }) {
  const controller = new AbortController();
  const response = await fetch(`${BASE_URL}/chat/completions`, {
    method: "POST",
    headers: {
      "authorization": `Bearer ${apiKey}`,
      "content-type": "application/json"
    },
    body: JSON.stringify({
      model: "gpt-5.6-terra",
      messages,
      stream: true
    }),
    signal: controller.signal
  });

  if (!response.ok || !response.body) {
    const body = await response.text().catch(() => "");
    throw new Error(`stream request failed: ${response.status} ${body}`);
  }

  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
  let sawFirstByte = false;

  async function readWithDeadline(ms, label) {
    const deadline = createDeadline(ms, label);
    try {
      return await Promise.race([reader.read(), deadline.promise]);
    } finally {
      deadline.cancel();
    }
  }

  try {
    while (true) {
      const result = await readWithDeadline(
        sawFirstByte ? 30_000 : 90_000,
        sawFirstByte ? "inter-chunk timeout" : "first-byte timeout"
      );

      if (result.done) {
        buffer += decoder.decode();
        break;
      }

      sawFirstByte = true;
      buffer += decoder.decode(result.value, { stream: true });

      // An SSE event ends at a blank line.
      const events = buffer.split(/\r?\n\r?\n/);
      buffer = events.pop() ?? "";

      for (const event of events) {
        const dataLines = event
          .split(/\r?\n/)
          .filter(line => line.startsWith("data:"))
          .map(line => line.slice(5).trimStart());

        if (dataLines.length === 0) continue;

        const data = dataLines.join("\n");
        if (data === "[DONE]") return;

        const parsed = JSON.parse(data);
        const choice = parsed.choices?.[0];
        const text = choice?.delta?.content;

        if (typeof text === "string" && text.length > 0) {
          process.stdout.write(text);
        }

        // Record finish_reason, tool-call fragments, and usage here.
      }
    }
  } catch (error) {
    controller.abort();

    // The output may be partial. Persist it as interrupted, not complete.
    error.partial = true;
    throw error;
  } finally {
    reader.releaseLock();
  }
}

There are several deliberate choices in this example:

• The first-byte deadline is longer than the inter-chunk deadline.
• TextDecoder receives { stream: true }, so split UTF-8 characters are handled.
• The buffer retains an incomplete event between reads.
• Multiple data: lines are joined before JSON parsing.
• A timeout aborts the underlying request instead of merely stopping your loop.
• Partial output is not presented as a successful final answer.

The example is not a universal retry implementation. Add request classification before retrying. A pure extraction request may be safe to replay. An agent request that already emitted a tool call requires a stronger policy.

Retry and timeout checklist

Use this checklist during an incident and then convert it into tests.

Transport checks

• [ ] Is the request reaching the intended base URL?
• [ ] Is stream actually set to true in the serialized JSON?
• [ ] Is the response status logged before body parsing?
• [ ] Is the first-byte time recorded?
• [ ] Is the maximum inter-chunk gap recorded?
• [ ] Does the reader have a deadline after every read?
• [ ] Does timeout cancellation close the underlying connection?
• [ ] Does curl with -N show incremental events?

SSE checks

• [ ] Does the parser retain incomplete lines between reads?
• [ ] Does it handle \n\n and \r\n\r\n event boundaries?
• [ ] Does it support several events in one network read?
• [ ] Does it handle [DONE]?
• [ ] Does it tolerate events without text content?
• [ ] Does it preserve partial UTF-8 characters?
• [ ] Does it record malformed events with a request ID?

Proxy checks

• [ ] Is response buffering disabled on the streaming route?
• [ ] Is compression delaying flushes?
• [ ] Is the ingress idle timeout shorter than your inter-chunk deadline?
• [ ] Does the application server flush downstream writes?
• [ ] Is a CDN caching or transforming the response?
• [ ] Does the browser-facing route use a different proxy chain than the worker?

Retry checks

• [ ] Is the failure before any output, after output, or after a tool call?
• [ ] Is the operation safe to replay?
• [ ] Is exponential backoff bounded?
• [ ] Are 429 and 5xx responses handled differently from read timeouts?
• [ ] Does the retry budget have a per-request and per-worker limit?
• [ ] Can a user cancel the original request before a fallback begins?

A retry should not start while the first request is still consuming a worker slot. Abort first, release the response body, and then decide whether a new attempt is justified.

Choosing timeout values from measurements

Avoid copying a timeout from a blog post. Derive it from your traffic.

Start by collecting first-byte and inter-chunk distributions for successful requests. Set the first-byte deadline above the normal tail for the specific model and prompt class, with room for bursts. Set the inter-chunk deadline based on the longest normal pause you are willing to expose to the user. Then add an overall wall-clock cap so a stream that drips one byte every few seconds cannot run forever.

Keep these values configurable by route or model. A short classification response and a long agent response should not share one hard-coded timeout.

A useful alert is not simply “stream failed.” Alert when:

• first-byte latency rises while inter-chunk gaps remain normal;
• inter-chunk gaps rise while first-byte latency remains normal;
• downstream flush latency exceeds upstream flush latency;
• timeout rates increase only behind one ingress or region;
• retries increase token usage without increasing completed requests.

That breakdown tells you whether to inspect provider latency, proxy behavior, or application code.

OpenAI direct versus an OpenAI-compatible gateway

Changing from a direct provider endpoint to LumeAPI should be a controlled variable, not the first emergency fix. Keep the request body and client behavior constant, change only the base URL and credential, and run the same stream probe from the same worker network.

LumeAPI’s catalog rates as of July 22, 2026 are below. Prices are USD per 1 million input and output tokens; they do not measure latency or guarantee feature parity with a provider-native endpoint.

The calculation for gpt-5.6-terra is:

Official: 10 × $2.50 + 2 × $15.00 = $55.00
LumeAPI:   10 × $0.75 + 2 × $4.50  = $16.50

That is a $38.50 difference for the stated traffic, not a promise about your invoice. Actual spend changes with prompt length, output length, retries, and the model selected. A cheaper route does not compensate for a retry loop that submits the same request seven times.

Use a gateway when the OpenAI-compatible request surface and catalog pricing fit your workload. Stay on a provider-native path when you depend on a provider-specific streaming event, Batch, prompt caching, a native moderation workflow, or an operational guarantee you have not verified through the gateway.

For a migration comparison, keep a direct-provider canary. Test first-byte latency, inter-chunk gaps, cancellation, tool-call streaming, malformed-event handling, and usage accounting before making the gateway your only route. See the related recovery runbook for interrupted LLM requests for the state-management side of this problem.

Why retries make a hanging stream expensive

A stream timeout is ambiguous. You know your client stopped waiting; you do not always know whether the upstream completed generation after the connection disappeared.

That creates two hazards:

1. Duplicate token spend: the original generation may continue or have already completed, while the retry starts another generation.
2. Duplicate side effects: an agent may have emitted a tool request before the stream broke.

Record an application-level request ID in your logs and database. If your workflow includes tools, persist the tool-call decision before executing it and make the tool operation idempotent where possible. A timeout after a tool call should usually enter a reconciliation state, not jump straight to a fresh agent run.

For user-facing chat, a safer fallback is often: preserve the partial text, mark it interrupted, and offer a “continue” action with explicit context. That is different from silently replaying the entire prompt.

FAQ

What does openai streaming hanging mean in a client log?

It means the client has not observed a completed stream, but it does not identify the failing layer. Check whether the client received zero bytes, partial bytes, or complete bytes that the parser rejected.

How do I fix an OpenAI API stream stuck in production?

Measure first-byte and inter-chunk timing, disable buffering on every streaming proxy route, parse SSE across arbitrary network reads, and abort stalled reads with a bounded deadline.

What is a practical SSE stream timeout for an LLM?

Use separate configurable values for connection, first byte, inter-chunk idle time, and total duration. Choose them from your successful latency distribution rather than assuming one value fits every model and prompt.

Can I retry after a chat completions stream times out?

Yes, but only after aborting the original request and classifying the operation. Pure text generation may be replayable; tool calls and other side effects require deduplication or reconciliation.

Does LumeAPI automatically fix proxy buffering or stream hangs?

No. LumeAPI is an independent third-party gateway. Your reverse proxy, ingress, application server, SSE parser, timeout policy, and cancellation behavior remain your responsibility.

Which model should I use while debugging a stream hang?

Start with one exact catalog model, such as gpt-5.6-terra, and keep the request unchanged while comparing routes. Switching models during diagnosis makes latency and event-shape differences harder to isolate.

Next steps

1. Run the minimal curl -N probe from the same network as the failing worker and save first-byte, inter-chunk, and completion observations.
2. Add incremental SSE parsing plus separate first-byte and inter-chunk aborts; test proxy buffering with a staging route.
3. Compare the measured route and token economics against the production LLM API options before migrating more traffic.

The core fix is operational, not cosmetic: do not turn every silent stream into a five-minute wait. Identify where progress stopped, abort the correct socket, and retry only when you can explain what the first request did.