Why Is My LLM API Response Cut Off? Diagnose Truncated Output

Last updated: August 6, 2026

An LLM API response can stop mid-sentence even when the HTTP request succeeds. First inspect the response's termination signal. In OpenAI-compatible Chat Completions, finish_reason="length" means generation reached the configured output limit; stop means a natural or configured stop point; tool_calls means the next action is a tool, not more prose; and content_filter means content was omitted. A broken stream or an input-length error is a different failure and needs a different fix.

Do not automatically save, execute or parse a cut-off answer. Classify the stop reason, validate that the requested artifact is complete, and either reject it or continue with a bounded protocol.

LumeAPI is an independent third-party OpenAI-compatible gateway. Exact model limits and parameter support vary by route, so verify the current model page before choosing an output budget.

Short path: Use the production LLM API, confirm the exact model, and keep the separate timeouts and provider failures guide for transport-level failures.

Quick Answer

In short

“The text looks cut off” is an observation, not a diagnosis. Preserve the HTTP status, request ID, finish_reason, usage, model ID, output limit and streaming terminal event. Then validate the artifact: JSON must parse and pass a schema, code must parse or compile, and a multi-section answer must contain its declared end marker. Only retry after the cause is known.

The truncation decision matrix

The highest-value distinction is between model termination and transport termination. A complete HTTP response with finish_reason="length" is not fixed by network retries. A stream that disappears before a terminal event is not fixed by raising max_tokens.

Reject incomplete output before your application uses it

This Python guard checks the model signal and a task-level completion marker. The marker is useful for prose and code-generation workflows where stop alone is not enough.

from dataclasses import dataclass


class IncompleteGeneration(RuntimeError):
    pass


@dataclass(frozen=True)
class CheckedText:
    text: str
    finish_reason: str


def require_complete_text(response, end_marker: str = "[[END]]") -> CheckedText:
    if not response.choices:
        raise IncompleteGeneration("No completion choice returned")

    choice = response.choices[0]
    reason = choice.finish_reason or "missing"
    text = choice.message.content or ""

    if reason == "length":
        raise IncompleteGeneration("Output token limit reached")
    if reason == "content_filter":
        raise IncompleteGeneration("Output withheld by content filter")
    if reason == "tool_calls":
        raise IncompleteGeneration("Tool execution is required before final text")
    if reason != "stop":
        raise IncompleteGeneration(f"Unexpected finish reason: {reason}")
    if not text.rstrip().endswith(end_marker):
        raise IncompleteGeneration("Task-level end marker is missing")

    return CheckedText(text=text.removesuffix(end_marker).rstrip(), finish_reason=reason)

An end marker is not a security control and does not prove correctness. It only verifies that the model reached the requested protocol boundary. Validate the actual artifact next.

JSON needs schema validation, not a continuation prompt

When a JSON object is cut in half, sending “continue” often produces a second fragment that cannot be joined safely. Even if the combined string parses, repeated keys, missing array items or changed ordering may corrupt the result.

Use bounded pagination instead:

Return at most 25 items.
Return one JSON object with:
- items: an array matching the supplied schema
- next_cursor: a stable source cursor or null
- complete: true only when no source items remain
Do not emit text outside the JSON object.

Your application should validate each page, reject duplicate item IDs and advance only from a cursor grounded in the source dataset. If the model is inventing the cursor, it is not reliable pagination; have the application assign chunks or source ranges instead.

import json


def validate_page(raw: str, seen_ids: set[str]) -> tuple[list[dict], str | None]:
    page = json.loads(raw)
    if set(page) != {"items", "next_cursor", "complete"}:
        raise ValueError("Unexpected page shape")
    if not isinstance(page["items"], list) or not isinstance(page["complete"], bool):
        raise TypeError("Invalid page field types")

    accepted = []
    for item in page["items"]:
        item_id = item.get("id")
        if not isinstance(item_id, str) or not item_id:
            raise ValueError("Every item needs a stable string id")
        if item_id in seen_ids:
            raise ValueError(f"Duplicate item id: {item_id}")
        seen_ids.add(item_id)
        accepted.append(item)

    if page["complete"] and page["next_cursor"] is not None:
        raise ValueError("A complete page cannot have a next cursor")
    return accepted, page["next_cursor"]

A reliable continuation protocol for prose or code

If a task cannot be regenerated as independent pages, use checkpoints instead of the one-word prompt “continue.”

1. Ask for an outline with stable section IDs before generating long text.
2. Generate one bounded section per request.
3. Save only sections that pass their validator.
4. On truncation, send the section ID, the last accepted checkpoint and a small overlap—not the entire partial document.
5. Ask the model to begin with the exact checkpoint ID and end with a section-specific marker.
6. Deduplicate the overlap and validate the assembled artifact.
7. Cap retries; if the same section fails twice, reduce its scope or switch model/route.

For code, the checkpoint should be a syntactic unit such as a function or file, not the last 20 characters. For prose, use a heading and paragraph ID. For a dataset, use application-owned source offsets.

Set an output budget from the task, not from habit

An oversized limit can reserve unnecessary throughput, increase worst-case cost and still fail to finish an unbounded task. An undersized limit guarantees truncation. Estimate:

required output budget =
  expected visible output
  + format overhead
  + model-specific reasoning reserve, when applicable
  + safety margin

OpenAI's current Responses reference defines max_output_tokens as including visible output and reasoning tokens. Chat Completions fields and compatible-provider behavior can differ. Use the parameter documented for the exact model and endpoint; do not blindly rename one field to another.

A retry also has a cost. If a failed attempt consumes F input tokens and G output tokens, and you repeat it R times, the avoidable model charge is approximately:

avoidable cost = R × ((F / 1,000,000 × input rate)
                    + (G / 1,000,000 × output rate))

Use current live rates. The better optimization is often smaller validated sections, not one giant request with a larger ceiling.

For scale only, assume a hypothetical rate of $0.50 per million input tokens and $2.00 per million output tokens. One failed attempt with 8,000 input and 1,200 output tokens costs roughly $0.0064; repeating the same failed request three times costs about $0.0192. Those are illustrative rates, not a LumeAPI or OpenAI quote. At high volume, the more important loss is often delayed jobs and corrupted downstream records, so optimize for validated completion rather than the cheapest-looking request.

Call an OpenAI-compatible route and keep the evidence

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LUMEAPI_KEY"],
    base_url="https://api.lumeapi.site/v1",
    timeout=60.0,
    max_retries=0,
)

response = client.chat.completions.create(
    model=os.environ["LUMEAPI_MODEL"],
    messages=[{
        "role": "user",
        "content": "Write section S1 in under 600 words. End with [[END]].",
    }],
    max_tokens=1200,
)

checked = require_complete_text(response)
print(checked.text)
print({
    "finish_reason": checked.finish_reason,
    "usage": response.usage,
    "request_id": getattr(response, "_request_id", None),
})

The numbers are example budgets, not universal settings. Confirm that the chosen model accepts the field and that 1,200 tokens cover your actual language, structure and reasoning behavior.

What to log without storing private prompts

Keep enough evidence to diagnose the failure:

timestamp
provider host and model ID
application request ID and provider request ID
HTTP status
streaming or non-streaming
configured output limit
finish_reason or terminal event
input/output/reasoning token usage when available
artifact validator result
retry count and retry cause

Avoid logging API keys, Authorization headers or unredacted customer content. A hash or internal document ID can correlate retries without copying the prompt into every log line.

Build a truncation test suite before production

Force each branch deliberately instead of waiting for users to discover it.

Attach a correlation ID to the request, validator and persistence event. Then assert that no incomplete fixture reaches the database or execution environment. This is more useful than checking whether the UI “usually looks complete,” because a single truncated SQL statement, configuration file or record batch can cause a disproportionate failure.

A realistic production scenario

A reporting service asks a model for 300 JSON records. The call returns HTTP 200, but finish_reason is length and the JSON ends inside record 187. A naive worker appends “continue,” receives a new array starting at an uncertain item and imports duplicates.

The corrected worker splits the source into application-owned batches of 25 IDs. Every response must match the schema and contain exactly the requested IDs. The worker commits a batch only after validation, then advances its own cursor. A failed batch is cheap to retry, and no model-generated cursor decides which business records were processed.

What most guides get wrong

“Raise max_tokens” is not a complete fix. It does not distinguish length from a broken stream, a tool call, a content filter or an early stop. It also turns an unbounded output request into a larger unbounded output request.

“Just say continue” is especially unsafe for JSON, code and extraction. Without stable checkpoints and validation, the continuation can repeat, skip or contradict the partial result. Completeness is an application invariant, not a visual impression.

Expert take

Never infer completion from HTTP 200 or from a final period. Require two signals: a model-level terminal reason and a task-level validator. Then make retries smaller and more deterministic than the failed request. This transforms truncation from a user-visible surprise into a controlled branch in the application state machine.

FAQ

What does finish_reason="length" mean?

In OpenAI Chat Completions, it means the configured maximum generation length was reached. Treat the content as incomplete unless your task-level validator independently proves otherwise.

Why is my answer cut off even when finish_reason is stop?

The model may have selected a natural stopping point, matched a configured stop sequence or misunderstood the requested scope. Remove accidental stop sequences and require bounded sections plus an explicit end marker.

Should I retry a truncated response automatically?

Only after classification. A length result can be retried as a smaller section; content_filter needs policy handling; tool_calls needs tool execution; a broken stream needs transport recovery. Cap all retries.

How do I continue truncated JSON?

Prefer regenerating application-defined pages and validating each object. Concatenating free-form continuations is fragile because valid syntax does not guarantee complete or non-duplicated data.

Is a missing [DONE] event the same as finish_reason="length"?

No. A missing terminal streaming event suggests an interrupted stream or client/proxy issue. A received length reason is a completed API response that reached its output ceiling.

Can reasoning tokens cause a short visible answer?

On interfaces where the documented output budget includes reasoning tokens, reasoning can consume part of that budget. Inspect the exact endpoint's usage fields and current model documentation instead of assuming all providers account identically.

Sources and verification

• OpenAI Chat Completions reference — definitions of finish_reason, checked August 6, 2026.
• OpenAI Responses reference — max_output_tokens boundary, checked August 6, 2026.
• OpenAI Developer Community: token-limit completion and handling length-truncated JSON — developer demand signals, not product authority.
• Reddit: truncated output reports and Reddit: short visible output under a completion budget — international user-problem signals, not factual authority.
• V2EX: multi-turn token pressure and SegmentFault: long output cut-off — adjacent Chinese developer demand signals; technical claims were checked against primary documentation.
• Baidu Tieba was searched for a specific LLM API truncation discussion on August 6, 2026, but no sufficiently specific, reliably indexed thread was used as evidence.