How to Fix 鈥淐ontext Length Exceeded鈥?Errors in an LLM API

Last verified: July 18, 2026

A context length exceeded error means the request plus the output capacity you asked the model to reserve does not fit within that model's context window. Fix it by reserving output tokens first, counting every input component, and removing low-value context in a deliberate order. Trim duplicated retrieval, verbose tool results and old conversational detail before deleting system instructions or the user's current request.

Switching to a larger-context model can help, but it should not be the only control. Without a token budget, growing chat history or retrieval results will eventually exceed the next limit too.

What counts toward an LLM context window?

Developers often count only the visible prompt. The effective input can include much more:

• system and developer instructions;
• the current user message;
• previous user and assistant turns;
• tool definitions and JSON schemas;
• tool calls and tool results;
• RAG documents, citations and metadata;
• images or files represented in the request;
• message roles, boundaries and provider-side formatting;
• output capacity reserved for the answer.

A useful planning equation is:

input budget = context window - reserved output - safety margin

input used = instructions + history + current turn + tools + retrieval + formatting

The safety margin absorbs tokenization differences and application changes. It is a reliability control, not wasted capacity. Do not publish one universal characters-per-token rule: language, code, structured data, images, files, tools and model tokenizers differ.

OpenAI's token-counting documentation explicitly notes that message structure adds tokens and that images, files, tools and schemas make local character estimates unreliable. Use the provider's exact counting endpoint when one is available; otherwise treat local counts as estimates.

Diagnose the oversized component before trimming

Record a size breakdown for the request builder, without logging sensitive content:

Do this before catching the error and deleting the oldest message. Blind trimming can remove a constraint while preserving five duplicate documents.

The safest order for reducing prompt tokens

1. Set a realistic output reserve

If the user asks for a short classification, reserving thousands of output tokens reduces available input for no benefit. Set the maximum for the task, not the largest value supported by the model. Keep enough headroom for reasoning or structured output where the selected model and API require it.

2. Remove exact and near duplicates

Deduplicate repeated system blocks, identical RAG chunks, quoted email chains and tool results already represented in history. This usually loses less information than summarization. Hash normalized chunks for exact duplicates, then use source ID and overlapping character ranges to collapse retrieval overlap.

3. Reduce retrieval noise

For RAG, lower top-k only after inspecting relevance. Better controls are:

• filter by tenant, product, date or document type before vector search;
• rerank candidates and enforce a minimum relevance score;
• cap tokens per source and per request;
• prefer a few complete, relevant passages over many overlapping fragments;
• include source identifiers so the answer remains auditable.

Do not truncate documents in the middle of a table or code block if structure carries meaning. Select whole passages within the budget.

4. Compact tool definitions and results

Tool schemas consume context even when the model does not call them. Expose only tools available and relevant in the current state. Shorten descriptions without removing behavioral distinctions, replace giant enums with validated IDs where appropriate, and omit unused optional fields.

Tool results should return what the model needs for the next decision. A search tool can return five ranked records with selected fields instead of a raw database response. A browser tool can return the extracted section instead of the full HTML document.

5. Summarize old conversation state

Keep the recent turns verbatim and replace older turns with a structured memory. A useful summary preserves:

• the user's goal and preferences;
• decisions already made and why;
• constraints and safety requirements;
• identifiers, dates and unresolved questions;
• facts that future turns must not silently change.

Mark summaries as derived state, not user-authored text. Regenerate them periodically from a trusted checkpoint to limit drift. OpenAI's conversation-state guide notes that multi-turn context can be managed by supplying earlier messages; whether you manage it manually or with provider state, it still requires a retention policy.

6. Ask the user to narrow a genuinely oversized task

If one user upload or request exceeds the available budget, silent truncation can produce a confident but incomplete answer. Tell the user what could not fit and offer a scoped choice: a specific section, a multi-pass analysis, or preprocessing into an index.

7. Route to a larger-context model when the task needs it

Some jobs legitimately require more source material. Route them to a model with a larger documented context window, then preserve the same preflight checks. Model limits and supported parameters change, so read the live [LumeAPI model documentation](/docs) rather than hard-coding a number from an old article.

Python preflight guard for an OpenAI-compatible request

The following guard uses a conservative local estimator. It is useful for rejecting obviously oversized text requests, but it is not an exact tokenizer and does not understand images, files or provider-specific formatting.

from dataclasses import dataclass
from typing import Iterable

@dataclass(frozen=True)
class Budget:
    context_window: int
    output_reserve: int
    safety_margin: int = 512

    @property
    def max_input(self) -> int:
        return self.context_window - self.output_reserve - self.safety_margin

def rough_text_tokens(text: str) -> int:
    # Conservative fallback only. Prefer the provider's counting API.
    return max(1, (len(text.encode("utf-8")) + 2) // 3)

def estimate_messages(messages: Iterable[dict]) -> int:
    total = 0
    for message in messages:
        total += 8  # estimated role and message framing
        total += rough_text_tokens(str(message.get("content", "")))
    return total

def enforce_budget(messages: list[dict], budget: Budget) -> None:
    estimated = estimate_messages(messages)
    if estimated > budget.max_input:
        raise ValueError(
            f"Estimated input {estimated} exceeds budget {budget.max_input}; "
            "compact history or retrieval before sending"
        )

Configure context_window from verified model metadata, not a value copied from another model. Set output_reserve from the task. For exact counts, use the provider's token-counting feature with the same messages, tools and attachments that will be sent.

A loss-aware history compaction pattern

Do not simply keep the last N messages: a critical decision may be older than a casual exchange. Split state into durable facts, recent verbatim turns and an older summary.

def build_compacted_messages(
    system_prompt: str,
    durable_facts: list[str],
    older_summary: str,
    recent_messages: list[dict],
    current_user_text: str,
) -> list[dict]:
    memory = "\n".join(f"- {fact}" for fact in durable_facts)
    return [
        {"role": "system", "content": system_prompt},
        {
            "role": "system",
            "content": (
                "Conversation memory (derived; verify against new user input):\n"
                f"Durable facts:\n{memory}\n\nOlder-turn summary:\n{older_summary}"
            ),
        },
        *recent_messages,
        {"role": "user", "content": current_user_text},
    ]

Treat new user input as authoritative when it conflicts with derived memory. Store the source message IDs used to create a summary so you can audit or rebuild it.

How to fix context errors in a RAG application

A practical token allocation might use explicit percentages or absolute caps for instructions, query, retrieved evidence, recent history and output. The exact split depends on the product; the important part is enforcing it before request construction.

Use this flow:

1. Normalize the query and apply permission filters.
2. Retrieve a wider candidate set.
3. Deduplicate overlapping chunks.
4. Rerank for the current question.
5. Select whole passages until the retrieval token cap is reached.
6. Preserve source IDs and titles in compact metadata.
7. Count the final request, including tools and output reserve.
8. If it still does not fit, compact history or ask for narrower scope.

This is more reliable than sending top 20 chunks and cutting the final string at an arbitrary character position.

Why automatic truncation can be dangerous

Some APIs can automatically drop older items when a request is too large; other configurations return a 400 error. Automatic truncation is convenient for low-risk chat but can remove the original user requirement, a safety constraint or a previous tool result.

If you enable it, define tests for:

• long conversations where the first turn contains the goal;
• system constraints larger than usual;
• tool schemas added by feature flags;
• RAG results with multilingual text and code;
• a maximum-size attachment;
• output near the reserved limit.

For production workloads, explicit compaction is easier to observe and audit.

Common fixes that do not solve the root problem

Increasing only the model limit: history keeps growing and the incident returns later.

Deleting the system message: the request may fit but lose policy, output format or task definition.

Using characters / 4 as an exact count: it ignores language differences, tools, images, files and message framing.

Cutting every retrieved chunk equally: irrelevant duplicates remain while relevant passages lose conclusions.

Retrying the same oversized payload: a deterministic size failure is not transient.

Production checklist

• Keep context-window metadata per model and verify it after model changes.
• Reserve output and a safety margin before allocating input.
• Measure instructions, history, retrieval, tools and current input separately.
• Prefer an exact provider count when available.
• Deduplicate before summarizing.
• Retain recent turns and durable facts explicitly.
• Cap tool-result and RAG payloads at their source.
• Log counts and component sizes, not sensitive prompt contents.
• Alert before requests reach the hard limit.
• Test oversized requests in CI and staging.

A [multi-model API](/multi-model-api) can route by task and request size, while a [production LLM API](/production-llm-api) still needs deterministic preflight controls. For client setup, start with the [Python](/research/openai-compatible-api-python) or [Node.js](/research/openai-compatible-api-nodejs) OpenAI-compatible guide.

Sources and limits

• [OpenAI token counting guide](https://developers.openai.com/api/docs/guides/token-counting)
• [OpenAI conversation state guide](https://developers.openai.com/api/docs/guides/conversation-state)
• [LumeAPI documentation](/docs)

Context windows, exact tokenization and truncation behavior are model- and provider-specific. Verify current model metadata and count the final wire request whenever the provider exposes that capability.