LLM API Token Limits: Choose Models That Survive Long Threads

Last verified: July 23, 2026

Short path: Start with the production LLM API guide, compare current rates on the production LLM API page, and check AI API pricing before choosing a model.

A context-limit failure is rarely caused by one enormous user message. In production, the usual culprit is an agent that keeps resending its entire history: system instructions, tool definitions, retrieved documents, previous tool results, failed attempts, and the latest request. Eventually, the next request cannot fit, even though each individual message looked reasonable.

The practical answer to llm api token limits is not “pick the model with the biggest advertised context window.” Pick a model whose documented input capacity leaves room for the output, tool calls, retries, and the next few turns. Then enforce a budget before the request reaches the API.

Quick Answer

In short

A long thread is a rolling budget, not a single conversation record. The winning production design is to keep a compact state, retrieve only what the current step needs, and reserve output space before every request. Use the exact documented context limit for each model, but choose based on usable headroom rather than the largest headline number.

LumeAPI can reduce per-token spend for the catalog models, but a cheaper request that fails at the context boundary is still wasted work. The fix is request shaping first, model selection second, and price comparison third.

What Most Guides Get Wrong

The common myth is: “If a model supports a large context window, I can keep sending the entire thread.”

That is not how an agent workload behaves. The request may contain four different kinds of tokens:

1. The stable system prompt and policy instructions.
2. Conversation history, including old assistant messages.
3. Tool schemas and tool results.
4. Retrieved or generated material for the current step.

The model also needs room for its response. A request that uses nearly the entire context window for input can leave no practical space for a useful completion. Depending on the API and SDK, you may see a context-length error, a validation error, truncation, or a response that stops before completing the task. Those are different failure modes, and the exact behavior must be checked in the official documentation for the provider and model.

A second mistake is treating “token limit” as one number. It is at least three numbers in your application:

• The documented context window for the exact model ID.
• Your input budget for one request.
• Your maximum output budget for that request.

Your application should calculate the last two. The first belongs in configuration, backed by the provider's current documentation.

For provider references, use the OpenAI API documentation, Anthropic API documentation, and Google Gemini API documentation. Provider documentation changes, and model aliases can have different limits from older or newer versions in the same family.

A Realistic Production Scenario

Consider a small support-automation team using Python, PostgreSQL, a vector store, and an OpenAI-compatible client. Their agent classifies a ticket, searches internal runbooks, calls a billing tool, and drafts a response. The conversation object contains every assistant message and every tool result.

The first request is fine. By the sixth tool call, the payload includes the original ticket, a 40-page runbook excerpt, three verbose JSON responses, the tool schemas, and several failed attempts. The team sees a context error in the middle of an otherwise successful run. They increase the output setting and retry the same payload. That makes the problem worse: more output is reserved, and the retry resends the same oversized history.

The turning point is not switching blindly to a larger model. They change the state representation. Tool results are stored in PostgreSQL and referenced by ID. Only the fields needed for the current decision are inserted into the prompt. Old turns become a summary with links to source records. Retrieved chunks are capped by token budget rather than document count. The agent now has a bounded request shape, so model selection and pricing become predictable.

The important detail is that the failure came from accumulation. No single ticket was unusually large.

Expert Take

Context management is an accounting problem with a control loop.

Before each call, estimate:

input tokens =
  system prompt
+ conversation state
+ tool definitions
+ retrieved context
+ current user request

Then enforce:

input tokens + reserved output tokens <= configured request budget

The configured request budget should be below the provider's documented maximum. The gap is operational headroom. It absorbs token-counting differences, serialization changes, new tools, and occasional larger-than-normal inputs.

Do not use a character count as a token count. JSON punctuation, code, URLs, identifiers, and non-English text can tokenize differently from ordinary prose. Use the tokenizer or usage data recommended by the relevant provider where available. If exact preflight tokenization is unavailable for a compatible route, use a conservative estimate and measure actual usage after the request.

The second control is state compression. Conversation history is useful for reasoning, but your database should remain the source of truth for durable facts. Store the full tool result outside the prompt. Insert a short structured representation into the next request:

{
  "tool": "lookup_invoice",
  "invoice_id": "inv_123",
  "status": "overdue",
  "balance_usd": 184.20,
  "source_record": "billing_events/evt_991"
}

The agent does not need the entire billing response if only status and balance affect the next action.

This advice does not apply unchanged when you require a provider-native feature that the gateway does not expose. Batch processing, prompt caching, provider-specific response controls, multimodal inputs, and special tool semantics may differ between an official endpoint and an OpenAI-compatible gateway. LumeAPI is an independent third-party gateway, not OpenAI, Anthropic, or Google. Verify the feature and parameter behavior against the route you intend to use.

Token Limits Are Not Just Context Windows

Teams often use these terms interchangeably:

• Input token limit: how much input a particular request can accept.
• Output token limit: how many tokens the model may generate for that request.
• Context window: the model's total working context, generally covering input and generated output according to the provider's API contract.
• Application budget: the lower limit your code enforces to keep requests reliable.
• Rate limit: how many requests or tokens the provider permits over a period. A rate-limit error is not a context-length error.
• Payload limit: an HTTP or API constraint on request size that may be expressed in bytes, files, or other units.

This distinction matters during incident response. A 429 caused by throughput requires backoff and queue control. A context-length error requires reducing or restructuring the request. Retrying the same context-limit payload is not recovery; it is repetition.

Model IDs matter as well. gpt-5.6-terra, claude-sonnet-4-6, and gemini-3.5-flash are separate catalog entries. You should not copy the context setting for one model to another simply because both are described as “fast” or “pro.” Maintain a model capability record with the exact ID, verified context information, tokenizer assumptions, output behavior, supported inputs, and last verification date.

How to Pick a Model for Long Threads

Use this decision order.

1. Establish the actual workload

Measure the largest normal request, not the average chat message. Include:

• The complete system prompt after template expansion.
• All tool schemas included on that turn.
• The largest retrieved context.
• Tool results after JSON serialization.
• The user's message.
• The maximum answer or structured output you will accept.
• Any retry or repair prompt added by your orchestration layer.

A support agent, coding agent, and document-analysis workflow can have very different token shapes. A model that works for a 20-turn chat may fail on a three-step tool loop if each tool returns a large payload.

2. Set a usable budget

Do not configure your application to operate continuously at the documented maximum. For example, if your internal request budget is 70% of the verified model capacity, the remaining space can absorb a larger tool result or a prompt change. The exact percentage is a policy choice; the principle is to leave headroom.

Your budget should be enforced at the boundary where the request is assembled. Logging a failure after the API rejects it is too late.

3. Match reasoning needs to the remaining context

For long threads, a model with enough context but inadequate task performance can produce a different failure: it technically fits, but loses the relevant instruction or misreads stale history. A concise state summary often improves quality more than adding another page of old transcript.

Use a more capable model for the step that needs difficult reasoning, not automatically for every hop. Use a smaller catalog model for classification, extraction, routing, or summarization when your evaluation supports it. The model choice should follow the task after the request has been bounded.

4. Test degradation behavior

A production agent needs a plan for a request that is too large:

1. Drop low-value retrieved chunks.
2. Replace verbose tool results with structured fields.
3. Summarize older conversation turns.
4. Reduce the output budget if the task permits it.
5. Escalate to a model with a verified larger capacity if one is available and compatible.
6. Ask the user for a narrower scope when the retained information is still too large.

The order matters. Escalating every oversized request can turn a prompt-construction bug into a billing problem.

Context Window Limit in Production: A Budgeting Pattern

A reliable conversation state has layers.

Stable instructions should be short and versioned. If the system prompt contains policy, product rules, tool usage instructions, examples, and duplicated formatting guidance, it becomes expensive on every hop. Remove repeated prose and move durable application logic into code.

Working memory should contain only facts needed for the current task. Keep it structured and replace stale values rather than appending every update.

Long-term memory belongs in a database or retrieval system. Store the full conversation, source documents, and tool outputs there. Retrieve a small, relevant slice for each decision.

Current evidence is the material needed to answer this turn. It may include a few search results, a tool response, or a file excerpt. Apply a token budget to evidence before inserting it.

Output space is reserved explicitly. A request that needs a 2,000-token answer cannot safely allocate the entire remaining window to input.

A simple policy can look like this:

MODEL_BUDGETS = {
    # Values are application budgets, not claims about provider maximums.
    "gpt-5.6-terra": 80_000,
    "claude-sonnet-4-6": 80_000,
    "gemini-3.5-flash": 80_000,
}

def fit_context(model_id, system_text, messages, tools, retrieved_text,
                output_budget, count_tokens):
    tool_text = serialize_tools(tools)
    sections = [
        ("system", system_text),
        ("messages", serialize_messages(messages)),
        ("tools", tool_text),
        ("retrieved", retrieved_text),
    ]

    budget = MODEL_BUDGETS[model_id]
    kept = []

    for name, text in sections:
        tokens = count_tokens(text)
        if name == "retrieved":
            remaining = budget - output_budget - sum(item[1] for item in kept)
            if remaining <= 0:
                continue
            text = trim_to_tokens(text, remaining, count_tokens)
            tokens = count_tokens(text)

        kept.append((name, tokens, text))

    input_tokens = sum(tokens for _, tokens, _ in kept)
    if input_tokens + output_budget > budget:
        raise ValueError("Request exceeds application token budget")

    return assemble_request(kept), input_tokens

This example deliberately separates MODEL_BUDGETS from provider limits. Populate those values from current official documentation and your own safety policy. Do not treat the numbers in the example as universal context-window specifications.

The code also shows a useful priority rule: trim retrieval before discarding system instructions or the current user request. In a real implementation, you would likely summarize old messages before reaching the exception and attach metadata showing which content was removed.

The Token Limit Exceeded Fix That Actually Works

When an incident occurs, inspect the assembled request rather than only the exception text.

Log token counts by category:

system_tokens=1,240
message_tokens=8,900
tool_schema_tokens=5,100
tool_result_tokens=21,400
retrieval_tokens=48,000
requested_output_tokens=8,000

This immediately tells you whether the problem is history, tools, retrieval, or an oversized output reservation. Redact secrets and personal data before logging payload samples. Counts are usually more useful than storing the entire prompt.

Then apply targeted fixes:

• History is too large: summarize old turns and preserve decisions, constraints, unresolved questions, and source references.
• Tools are too large: expose only tools relevant to the current stage and simplify schemas. Avoid embedding long examples in every schema.
• Tool results are too large: return a compact result plus a record ID. Paginate large results and let the agent request the next page.
• Retrieval is too large: rank by relevance, deduplicate overlapping chunks, and enforce a token cap.
• Output is too large: set an explicit output ceiling and use structured output where appropriate.
• One document is too large: process it in sections, extract facts, then run a separate synthesis step.
• The model's verified capacity is insufficient: select another exact catalog model only after checking quality, feature compatibility, and documented limits.

The important operational rule is idempotence. If your retry handler sends the same oversized request, it should fail fast locally instead of sending it again. A retry policy belongs around transient transport and rate-limit errors, not deterministic payload validation failures.

Comparing Catalog Models by Cost

The catalog below gives per-million-token rates supplied for this article. “Input” and “output” refer to the gateway's transparent per-token catalog rates. These prices do not establish context-window size, latency, native feature support, or quality.

For a concrete scenario, assume 10 million input tokens and 1 million output tokens in a month:

The calculation assumes all tokens are billed at the listed rates, with no provider-native Batch, cache, storage, or other pricing mechanism applied. It also assumes the gateway route supports the request shape you send. Confirm current terms before making a budget commitment.

The cost trap is repeated input. If an agent resends 100,000 input tokens on ten hops, that is 1 million input tokens before the final answer. A context-window fix that cuts the request to 20,000 tokens per hop can reduce cost and improve reliability at the same time.

A Decision Matrix for Long-Thread Workloads

This matrix is intentionally not a ranking of model context windows. The supplied catalog does not include verified context capacities, and those capacities can change by exact model ID or endpoint. Check the official model documentation before putting an ID into a long-context route.

Calling a Catalog Model Through LumeAPI

LumeAPI exposes an OpenAI-compatible Chat Completions endpoint at https://api.lumeapi.site/v1. The following Python example uses an exact catalog ID and keeps the request deliberately small:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LUMEAPI_KEY"],
    base_url="https://api.lumeapi.site/v1",
)

response = client.chat.completions.create(
    model="gpt-5.6-terra",
    messages=[
        {
            "role": "system",
            "content": (
                "Answer from the supplied evidence. "
                "If evidence is missing, say what is missing."
            ),
        },
        {
            "role": "user",
            "content": (
                "Ticket ID: 1842\n"
                "Relevant facts: invoice is overdue; balance is $184.20.\n"
                "Question: draft a concise customer reply."
            ),
        },
    ],
    max_tokens=500,
)

print(response.choices[0].message.content)

The base_url change is the key migration detail, but it is not a promise that every provider-native parameter will work unchanged. Confirm the OpenAI SDK version and the endpoint behavior in the relevant documentation before adding streaming, tool calls, structured outputs, multimodal inputs, or provider-specific fields.

For a long-running agent, put a preflight layer in front of this call. The layer should count or conservatively estimate tokens, apply the model-specific application budget, record the categories that were trimmed, and reject a request before it reaches the network when it cannot fit.

Context Management Checklist

Use this checklist during implementation and incident review:

• Record the exact model ID, endpoint, and verification date.
• Store documented provider limits separately from your internal safety budget.
• Count system instructions, messages, tools, retrieval, and output reservation independently.
• Keep full transcripts and tool payloads outside the prompt when they are not needed for the current decision.
• Summarize old turns around decisions and unresolved work, not generic prose.
• Deduplicate retrieval results before token budgeting.
• Cap tool-result size and paginate large responses.
• Limit tools supplied to the current agent stage.
• Reserve output capacity before adding retrieved context.
• Log token counts by category with sensitive content redacted.
• Fail locally on deterministic context overflow.
• Retry only errors that can plausibly change after waiting or reconnecting.
• Test the largest normal payload and an adversarial oversized payload.
• Recheck limits and pricing after model or provider changes.
• Test gateway compatibility for every feature your production route depends on.

The checklist is more valuable than a single “large context” setting because it continues to work when your prompt, tools, retrieval corpus, or orchestration graph changes.

FAQ

What are llm api token limits?

LLM API token limits are the constraints on input, output, and total request context for a specific model and endpoint. They determine whether a request fits and how much text the model can generate. Always verify the exact model ID in current official documentation rather than relying on a family-level label.

How do I fix a context window limit in production?

Fix a context window limit by measuring the assembled request, reserving output space, trimming retrieval, summarizing older messages, and reducing verbose tool results. If the bounded request still does not fit, choose another model with a verified suitable capacity or split the task into stages.

What is the max tokens setting for OpenAI, Claude, and Gemini?

The max-token parameter and its accepted name or behavior can vary by provider, endpoint, SDK version, and model. Do not assume one setting applies to OpenAI, Claude, and Gemini routes. Check the OpenAI text generation documentation, Anthropic messages documentation, and Gemini API reference for the exact route you use.

Does a larger context window eliminate the need for summarization?

No. A larger context window does not eliminate summarization because repeated input increases cost, slows request construction, and can bury the current evidence under stale history. Summarize when the old material is no longer needed verbatim, while retaining durable facts and source references outside the prompt.

Can I use provider-native Batch or prompt caching through LumeAPI?

Do not assume provider-native Batch or prompt caching is available through LumeAPI. Those features may depend on provider-specific endpoints, billing rules, request formats, or cache semantics. Verify support for the exact route before counting those mechanisms in a production cost model.

Which LumeAPI model is best for a long agent thread?

There is no responsible universal winner based on the catalog alone because the supplied rates do not specify context capacity or task quality. Start with a bounded-context evaluation using gpt-5.6-terra, claude-sonnet-4-6, and gemini-3.5-flash, then test the exact model IDs against your tools, retrieval data, and recovery behavior.

Next Steps

1. Instrument your request builder and record input tokens by system prompt, history, tools, retrieval, and output reservation.
2. Add a preflight budget, summarize old state, and make oversized tool results addressable by record ID instead of embedding their full payload.
3. Run your production eval against two or three exact catalog IDs, then compare the resulting token mix and rates on the LumeAPI production page.

The central decision is straightforward: first make the request bounded, then choose the model that handles the bounded task. A large context limit is useful headroom, but it is not a substitute for state management.