Claude API Slow: Production Latency Troubleshooting Runbook (2026)

Last verified: July 23, 2026

Short path: Start with the Claude API guide, compare current model costs on cheap Claude API, then check AI API pricing or the <code>claude-sonnet-4-6</code> model page.

If the Claude API is slow, do not start by replacing your agent framework. First find out whether the delay is before the request, before the first token, between tool calls, or in the final output. Most production latency problems come from measuring the whole agent loop as one number and then tuning the wrong component.

The practical fix is usually a combination of four changes: measure time to first byte separately from total completion time, stop resending unnecessary context, stream responses, and put a hard budget around retries and output length. Claude Sonnet can still be the right model; the slow part may be your orchestration layer.

Quick Answer

In short

A slow Claude call is often an agent-loop problem wearing a model-provider disguise. The biggest production win is to reduce the amount of work per request—especially repeated context and unnecessary sequential hops—then stream what remains. A gateway such as LumeAPI may help with routing and token cost, but it is an independent third-party path, not a promise of faster Anthropic inference.

What most guides get wrong

The common myth is: “The model is slow, so increase the timeout or switch providers.”

A longer timeout changes what your user sees; it does not reduce latency. A provider switch can also hide the real issue if your agent sends 100,000 tokens and waits for five sequential tool calls before displaying anything.

The more useful distinction is time to first byte versus time to completion. If first-byte latency is high while output is short, investigate connection reuse, queueing, concurrency, regional network paths, and upstream response time. If the first token arrives quickly but completion takes 20 seconds, inspect output length and generation settings. If each individual request looks acceptable but the user waits 45 seconds, inspect the agent graph: serial tools, repeated planning calls, and retries are probably dominating the experience.

Treat each request as a trace, not as a single stopwatch value.

A realistic production scenario

Consider a representative support-automation team running Python workers, PostgreSQL-backed conversation state, and an agent that calls a ticketing API and an internal search service.

Their Sonnet request appeared to take 32 seconds at p95. The team initially blamed Anthropic because a comparable GPT request completed sooner. A trace showed a different story:

• 1.4 seconds to establish a new HTTPS connection on each hop
• 8 seconds waiting for the first model response
• 6 seconds for an internal search call
• 9 seconds for a second Claude planning call
• 7 seconds generating the final answer
• one retry on a read timeout for roughly 10% of requests

The agent was also appending the full prior tool transcript to every call. The turning point was splitting the trace into model and orchestration spans. The team enabled connection pooling, streamed the final response, summarized old tool output, and made the search call concurrent with another independent lookup. They did not rewrite the stack or immediately replace Sonnet. The important change was measuring the seventh step instead of arguing about the first.

Expert take

Latency has three separate budgets:

1. Transport budget: DNS, TCP/TLS setup, proxy traversal, connection pool waits, and network transfer.
2. Provider/model budget: time before the first token and time spent generating tokens.
3. Application budget: tool calls, serialization, queueing, retries, and decisions made between model calls.

Only the second budget belongs directly to model inference. Your user experiences all three.

The mechanism that surprises teams most is context re-send. A multi-turn API call generally includes the conversation or the relevant message history again. In an agent loop, that can mean the same system instructions, retrieved documents, tool definitions, and failed tool payloads are transmitted on every hop. Even if the model produces only a short answer, the request can become expensive and slower to process.

The second non-obvious mechanism is serial dependency disguised as intelligence. An agent may call the model, wait for a search tool, call the model again, wait for a ticket lookup, and call the model a third time. Each hop adds its own first-token delay. Reducing three hops to two can matter more than shaving a small amount from one model request.

Do not follow this advice blindly. If your workload requires provider-native features, native prompt caching, Batch processing, special content blocks, or a provider-specific response format, an OpenAI-compatible gateway may not expose everything your application needs. Verify the feature surface before migrating. And if your trace shows the model begins responding quickly but generates a long, necessary answer, transport tuning will not make generation instantaneous; you need to change the output contract or accept the budget.

Build the latency trace before changing code

Add a request identifier that survives the complete agent loop. At minimum, capture these fields for every model call:

• provider path: Anthropic direct, gateway, or another route
• exact model ID
• agent run ID and hop number
• request start and end timestamps
• DNS, connection, TLS, and connection-pool wait time where available
• time to first byte or first streamed chunk
• total completion time
• input and output token counts returned by the API
• HTTP status and error type
• retry number and retry reason
• tool-call count and serialized tool-result size
• requested max_tokens
• whether streaming was enabled

Use p50, p95, and p99. Averages conceal the timeout tail that your users actually report.

A useful derived metric is:

agent latency =
  sum(model first-byte and generation time)
  + sum(tool latency)
  + queue time
  + retry delay
  + serialization and application overhead

For each request, also calculate:

output tokens / total duration
input tokens / model-call count
retry delay / total duration

These ratios point toward different fixes. High output tokens per second but long total time suggests too much output. High input tokens with repeated prompts suggests context growth. Large retry delay identifies policy trouble rather than model quality.

Add a trace around the gateway call

The following example uses the catalog’s OpenAI-compatible endpoint. It records first-byte timing with a streaming request and uses an explicit timeout. The exact model ID is important.

import json
import os
import time
import requests

BASE_URL = "https://api.lumeapi.site/v1"
MODEL = "claude-sonnet-4-6"

payload = {
    "model": MODEL,
    "messages": [
        {
            "role": "system",
            "content": "Answer briefly. Use a tool only when the request requires external data."
        },
        {
            "role": "user",
            "content": "Summarize the current ticket status in three bullets."
        }
    ],
    "max_tokens": 500,
    "temperature": 0,
    "stream": True,
}

started = time.perf_counter()
first_chunk_at = None
chunks = []

with requests.post(
    f"{BASE_URL}/chat/completions",
    headers={
        "Authorization": f"Bearer {os.environ['LUMEAPI_KEY']}",
        "Content-Type": "application/json",
    },
    json=payload,
    stream=True,
    timeout=(5, 45),  # connect timeout, read timeout
) as response:
    response.raise_for_status()

    for line in response.iter_lines(decode_unicode=True):
        if not line or not line.startswith("data: "):
            continue

        if first_chunk_at is None:
            first_chunk_at = time.perf_counter()

        data = line.removeprefix("data: ")
        if data == "[DONE]":
            break

        chunks.append(json.loads(data))

finished = time.perf_counter()

print({
    "model": MODEL,
    "time_to_first_chunk_ms": (
        (first_chunk_at - started) * 1000 if first_chunk_at else None
    ),
    "total_ms": (finished - started) * 1000,
    "chunks": len(chunks),
})

Use streaming for user-facing work, but do not confuse “the interface started drawing text” with “the request became faster.” Streaming improves time to visible output. It may not reduce total generation time.

The Claude API slow fix: work through this checklist

1. Separate first-token delay from generation time

Run the same request in two modes:

• non-streaming, to measure complete-response duration
• streaming, to measure time to first chunk and total duration

Interpret the result:

• High first chunk and high total time: investigate transport, queueing, upstream load, request size, and concurrency.
• Low first chunk and high total time: reduce output length, constrain the response format, or split the task.
• Good single-call latency but bad agent latency: inspect serial tools, retries, and repeated context.
• Only p99 is bad: inspect connection pools, worker saturation, rate limiting, and retry storms.

Do not compare a streamed Claude response with a non-streamed GPT response and call the comparison fair.

2. Reuse connections

A new connection for every model call adds avoidable work. Use an HTTP client with connection pooling, keep worker processes warm, and avoid creating a new client inside the request handler. Check whether your proxy or load balancer closes idle connections aggressively.

Connection reuse is especially important in agent loops because one user request may create several model calls. If every hop performs a fresh handshake, the overhead compounds.

Also check connection-pool limits. A pool that is smaller than your worker concurrency creates hidden queue time. A pool that is much larger than your provider allowance can create a burst of requests and trigger throttling.

3. Stream the final user-visible response

If the model has useful partial output, send it to the client as it arrives. This is usually the best answer to “the chatbot feels slow,” even when total completion time cannot be reduced.

Keep tool-selection and structured planning calls non-streamed if your application needs to parse them atomically. Stream the final natural-language answer only after the tool results are available. That keeps the UI responsive without forcing your parser to consume half a plan.

Your frontend and reverse proxy must also permit incremental flushing. A server can receive streamed chunks and still buffer them until the response ends.

4. Cap output deliberately

A generous output limit is not a performance strategy. Set max_tokens to the largest response your task actually needs. For a classification or routing decision, that may be a small structured response. For a customer explanation, use a clear length instruction and validate it.

Output tokens affect both cost and completion time. If a tool call needs a JSON decision, do not ask the model for a full explanation in the same call. Return the decision, execute the tool, and generate a concise user-facing message afterward.

A cap should not be so low that the model regularly truncates responses and forces a retry. Track finish reasons and truncated-output rates when tuning it.

5. Remove repeated context

Inspect the serialized request, not just the application object. Teams often discover that the prompt contains:

• the same system policy repeated by a wrapper
• old tool results that are no longer relevant
• full search documents after a summary has already been created
• failed tool calls and their stack traces
• duplicated conversation history
• tool schemas included on calls that cannot use tools

Set a context policy. For example:

• retain the last few conversational turns
• summarize older turns
• keep only the fields needed from tool results
• truncate logs and stack traces
• retrieve documents for the current task instead of carrying every prior result
• include only tools available in the current phase

Do not summarize blindly. Preserve identifiers, user constraints, and facts needed to complete the task. A bad summary can cause a second model call, which defeats the latency win.

6. Collapse serial agent hops

Draw the agent as a graph. Mark every edge that waits for a previous model or tool response. Then ask whether the dependencies are real.

Independent operations can often run concurrently:

import asyncio

async def run_independent_lookups(ticket_id, customer_id):
    ticket_task = asyncio.create_task(fetch_ticket(ticket_id))
    customer_task = asyncio.create_task(fetch_customer(customer_id))

    ticket, customer = await asyncio.gather(ticket_task, customer_task)
    return ticket, customer

Only parallelize calls that are truly independent and safe for your systems. Two writes to the same record, or two tools with conflicting side effects, should not be fired together merely to reduce latency.

A useful rule is to set a maximum model-hop budget per user request. If the agent reaches the limit, return a controlled fallback or ask for clarification instead of entering an open-ended loop.

7. Fix retry behavior

Retries are necessary for transient failures, but an unbounded retry loop turns a temporary problem into a user-visible outage.

Use exponential backoff with jitter, a small maximum attempt count, and a total request deadline. Retry based on the documented status and error semantics of the API rather than matching arbitrary error strings. Anthropic’s API error documentation is the right place to verify current error handling for direct integrations.

Be careful with read timeouts. A client can time out while the provider has already accepted and completed the request. Automatically retrying a request with side effects can duplicate work. For tool execution, make operations idempotent or place the side effect behind a durable job and status check.

import random
import time
import requests

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}

def post_with_deadline(session, url, headers, payload, deadline_s=45):
    started = time.monotonic()
    last_error = None

    for attempt in range(3):
        remaining = deadline_s - (time.monotonic() - started)
        if remaining <= 0:
            break

        try:
            response = session.post(
                url,
                headers=headers,
                json=payload,
                timeout=(5, min(remaining, 20)),
            )

            if response.status_code not in RETRYABLE_STATUS:
                response.raise_for_status()
                return response

            last_error = RuntimeError(
                f"retryable HTTP status {response.status_code}"
            )

        except (requests.Timeout, requests.ConnectionError) as exc:
            last_error = exc

        if attempt == 2:
            break

        backoff = min(8, 2 ** attempt) + random.uniform(0, 0.5)
        time.sleep(min(backoff, max(0, deadline_s - (time.monotonic() - started))))

    raise TimeoutError("model request exceeded retry/deadline budget") from last_error

This is a pattern, not a claim that every status should be retried in every application. Confirm the provider’s current guidance and your operation’s idempotency before shipping it.

Compare the direct path and a gateway fairly

If your current source is Anthropic direct, run a controlled comparison before migrating traffic:

1. Use the same prompt, model ID, output limit, and streaming mode.
2. Test at the same concurrency level.
3. Measure first-byte and total time separately.
4. Compare p50, p95, and error/retry rates.
5. Repeat with small and large contexts.
6. Test your real tool-loop shape, not only a single prompt.
7. Check response compatibility, logging, billing, and failure behavior.

LumeAPI is an independent third-party gateway, not Anthropic, OpenAI, or Google. Its endpoint is OpenAI-compatible, but that does not mean every provider-native feature behaves identically or is available through the gateway. It also does not provide a universal guarantee that a Claude request will be faster than the direct Anthropic route.

The reason to test it may still be compelling: the live catalog lists lower per-token rates. That is a cost decision, while latency is an empirical routing decision.

Pricing is not latency, but it changes the tuning choices

The following rates are from the LumeAPI catalog updated July 22, 2026. They are per 1 million input and output tokens. “Official” means the reference rate supplied in the assignment; verify the provider’s current pricing before committing production spend.

The monthly example is simple arithmetic:

input cost = 1 × input rate
output cost = 0.25 × output rate
monthly total = input cost + output cost

It excludes taxes, application infrastructure, retries beyond the stated token volume, and any provider-specific pricing features. It also says nothing about response time.

For a slow agent, lower rates can make it affordable to add a small fast-path model for bounded work, but do not route every request to a more expensive model just because you have budget. Likewise, do not switch to Opus solely because a long Sonnet response feels slow. First determine whether the request is oversized or the agent is looping.

See the official Anthropic Messages API documentation for the native API shape, and compare it with the compatibility requirements of your chosen route.

A practical runbook for an incident

Use this sequence when a latency alert fires.

Triage: first 15 minutes

• Confirm whether the issue affects all Claude requests or one model ID.
• Check p50 versus p95 and p99.
• Separate first-byte time from total time.
• Check request volume, concurrent workers, connection-pool waits, and HTTP status codes.
• Look for a jump in input or output tokens.
• Count retries per request and total retry delay.
• Compare a single direct request with a single gateway request using a fixed small prompt.
• Do not increase all timeouts or retry counts during the incident.

Containment

• Stop open-ended agent loops.
• Apply a maximum hop count and total deadline.
• Disable nonessential tool calls.
• Reduce output limits for clearly bounded tasks.
• Stream final responses where the UI supports it.
• Protect downstream services from a retry storm.
• Route only the affected workload to a tested fallback; do not assume model compatibility without an evaluation.

Root-cause investigation

Inspect a slow trace and answer these questions in order:

1. Did the request wait in your queue?
2. Did it wait for a connection?
3. How long until the first response bytes?
4. How many input tokens were sent?
5. How many output tokens were generated?
6. How many tools ran, and which were serial?
7. Did the client retry?
8. Did a proxy buffer the stream?
9. Did the application perform extra model calls after receiving a usable answer?

The answer usually falls into one of three buckets: transport, model generation, or orchestration. Assign the fix to the bucket instead of changing everything at once.

Regression protection

Add latency tests that cover the shapes your production system actually sends:

• short prompt, short answer
• long conversation with a short answer
• tool call followed by a final response
• two independent tools
• provider error followed by one permitted retry
• streamed response through the production proxy

Store token counts and hop counts with test results. A latency regression caused by a prompt wrapper may not appear in a tiny synthetic request.

When not to optimize for the lowest latency

Some workloads should remain on the direct provider path or use provider-native features. Examples include applications that depend on a native API feature not exposed by your compatibility layer, strict data-routing requirements, or a billing workflow built around provider-native discounts and batch processing.

Do not trade away correctness for a lower p95 on tasks where a wrong tool argument creates a financial or operational incident. Benchmark the actual task success rate alongside latency. A faster response that requires a corrective second call is not necessarily faster for the user.

Also watch for cold-start effects in your own infrastructure. A serverless worker that starts a process, loads a large prompt template, opens a database connection, and then calls Claude may report the entire startup as “Claude latency.” Put application spans around the model request so your dashboard does not assign your runtime’s work to the provider.

FAQ

Why is the Claude API slow only in my agent loop?

The loop may be adding serial model calls, tool waits, repeated context, and retries that do not appear in a single-call benchmark. Trace every hop and calculate the sum instead of timing only the outer request.

How do I apply a Claude API slow fix without rewriting my stack?

Start with instrumentation, connection reuse, streaming, output limits, context trimming, and bounded retries. These changes usually sit at the HTTP client and orchestration boundaries rather than requiring a new agent framework.

Does streaming reduce Claude API latency?

Streaming usually reduces time to visible output, but it does not necessarily reduce total completion time. Measure both first-chunk and final-chunk timestamps.

Should I use a smaller model to reduce Claude API latency?

Use a smaller model when the task has a clear quality boundary and your evaluation accepts the result. Do not use model substitution to hide a large prompt, excessive output, or a serial retry loop.

Can LumeAPI reduce Claude API latency compared with Anthropic direct?

It may produce a different observed latency because the network and routing path differ, but no gateway should be treated as a guaranteed latency improvement. Test the same model, prompt, concurrency, streaming mode, and timeout policy on both paths.

What is the main limitation of an OpenAI-compatible Claude route?

Compatibility at the chat-completions endpoint does not guarantee parity with every native provider feature, parameter, response block, or operational policy. Verify the features your application relies on before moving traffic.

Next steps

1. Add first-byte, total-duration, token, hop, and retry fields to your model-call traces.
2. Fix the largest measured bucket: transport, output generation, context size, or orchestration.
3. Run a small production-shaped comparison between Anthropic direct and a tested Claude API cost option, keeping the model and prompts constant.

The core lesson is simple: a slow Claude request is not automatically a slow Claude model. Find the repeated work, the serial dependency, or the retry that is stretching the user’s wait—and remove that before rewriting your stack.