How to Build an LLM API Fallback in Python

Last verified: July 20, 2026

An LLM API fallback sends a request to one tested alternative model only after the primary route has a retryable operational failure, such as a timeout, connection failure, rate limit, or server error. Do not fall back on authentication errors, invalid input, missing balance, or schema bugs: another model is unlikely to fix them and may create a second charge.

LumeAPI makes application-controlled fallback simpler because supported GPT, Claude and Gemini models share one OpenAI-compatible base URL, key and USD wallet. You still select the exact model ID and implement the policy. Current LumeAPI product text explicitly says customers implement retries and fallback routes in their applications; this guide does not claim managed automatic failover.

Retry and fallback are different

A retry repeats the same route after a temporary failure. A fallback changes the model or route. Use a small retry budget before one tested fallback, not a long chain.

A timeout is ambiguous: the upstream may have completed work even when the client saw no response. Read-only generation is easier to repeat safely than a tool call that sends an email or charges a card.

Configure one client and a tested pair

python -m pip install --upgrade openai
export LUMEAPI_KEY='your-lumeapi-key'

import json
import os
import time
import uuid
from dataclasses import asdict, dataclass

import openai
from openai import OpenAI

client = OpenAI(
    api_key=os.environ['LUMEAPI_KEY'],
    base_url='https://api.lumeapi.site/v1',
    timeout=20.0,
    max_retries=0,
)

PRIMARY_MODEL = 'gpt-5.4-mini'
FALLBACK_MODEL = 'gemini-3.5-flash'
ALLOWED_MODELS = {PRIMARY_MODEL, FALLBACK_MODEL}

SDK retries are disabled here so the application owns the total attempt budget. If you keep SDK retries enabled, count them when calculating worst-case latency and cost.

Classify only retryable failures

RETRYABLE_ERRORS = (
    openai.APITimeoutError,
    openai.APIConnectionError,
    openai.RateLimitError,
    openai.InternalServerError,
)

def is_retryable(exc: Exception) -> bool:
    return isinstance(exc, RETRYABLE_ERRORS)

Do not use except Exception as a signal to switch models. It would convert programmer errors, invalid schemas and authentication failures into duplicate calls.

Trace every attempt

@dataclass
class Attempt:
    request_id: str
    model: str
    attempt: int
    outcome: str
    latency_ms: int
    error_type: str | None = None

def write_attempt(record: Attempt) -> None:
    print(json.dumps(asdict(record)))

In production, send this record to structured logging. Do not include the API key, authorization header, full prompt or sensitive output. LumeAPI's console-side logs can then be reconciled with application attempts using time, model and request metadata.

Implement a bounded fallback

def complete_with_fallback(prompt: str) -> str:
    request_id = str(uuid.uuid4())
    last_error: Exception | None = None

    for attempt_number, model in enumerate(
        [PRIMARY_MODEL, FALLBACK_MODEL], start=1
    ):
        if model not in ALLOWED_MODELS:
            raise RuntimeError(f'Model is not allowlisted: {model}')

        started = time.perf_counter()
        try:
            response = client.chat.completions.create(
                model=model,
                messages=[{'role': 'user', 'content': prompt}],
                max_tokens=300,
            )
            latency_ms = round((time.perf_counter() - started) * 1000)
            write_attempt(Attempt(
                request_id=request_id,
                model=model,
                attempt=attempt_number,
                outcome='success',
                latency_ms=latency_ms,
            ))
            return response.choices[0].message.content or ''
        except Exception as exc:
            latency_ms = round((time.perf_counter() - started) * 1000)
            write_attempt(Attempt(
                request_id=request_id,
                model=model,
                attempt=attempt_number,
                outcome='failed',
                latency_ms=latency_ms,
                error_type=type(exc).__name__,
            ))
            if not is_retryable(exc):
                raise
            last_error = exc

    raise RuntimeError('Primary and fallback routes failed.') from last_error

This performs at most two billable attempts. It does not sleep and retry the primary first; add one jittered retry only if your latency budget and error data justify it.

Add backoff without creating a retry storm

A small delay helps with 429 or transient server errors. Respect Retry-After when exposed; otherwise use capped exponential backoff with jitter.

import random

def backoff_seconds(retry_number: int, cap: float = 4.0) -> float:
    base = min(cap, 0.5 * (2 ** retry_number))
    return random.uniform(0.0, base)

At scale, queue and concurrency controls matter more than aggressive per-request retries. If thousands of workers immediately retry, they can prolong an incident. Put a global rate limiter or circuit breaker in front of the fallback route.

Protect side effects with idempotency

The sample is safe only for read-only text generation. For a workflow with tools, separate generation from execution:

1. Generate or recover a proposed action.
2. Validate the action and user authorization.
3. Derive a stable idempotency key from the business operation.
4. Store a pending/executed record durably.
5. Execute once; return the stored result on repeats.

Do not let both the primary and fallback send an email or payment. The duplicate tool-call prevention guide covers the full pattern.

Test fallback before an outage

Inject failures instead of waiting for production:

• Make the primary client raise APITimeoutError; verify the fallback runs once.
• Return 401; verify the fallback does not run.
• Return 400; verify the original error surfaces.
• Make both models fail; verify the final error preserves the last cause.
• Simulate 100 concurrent failures; verify the fallback is protected from a traffic spike.
• Compare prompts, output validation and safety behavior on both models.
• Verify logs distinguish primary, fallback and final outcome.

A fallback model must pass the same task evaluation as the primary. Transport compatibility is useful, but prompt behavior, tool selection, structured output, context limits and safety responses can differ.

Set an operational policy

Define these values before enabling fallback:

FALLBACK_POLICY = {
    'max_total_attempts': 2,
    'deadline_seconds': 25,
    'allowed_tasks': {'classification', 'draft_reply'},
    'blocked_tasks': {'payment', 'send_email'},
    'primary': PRIMARY_MODEL,
    'fallback': FALLBACK_MODEL,
}

For each task, decide whether degraded success is better than a clear error. A legal or financial workflow may prefer to stop rather than silently use a model that has not passed the required evaluation.

Monitor the right metrics

Track:

• primary success rate;
• fallback activation rate;
• fallback success rate;
• total attempts per completed task;
• p50/p95 end-to-end latency;
• cost per successful task including failed attempts;
• output-validation failure rate by model;
• circuit-breaker state;
• duplicate-side-effect prevention events.

LumeAPI per-call logs expose the selected model, tokens, cost and latency. Combine them with the application trace because the gateway cannot know whether a business task was ultimately accepted by your validator.

Current model and cost examples

On July 20, 2026, LumeAPI listed gpt-5.4-mini at $0.225 input / $1.35 output per 1M tokens and gemini-3.5-flash at $0.90 / $5.40. The first is displayed 70% below its reference; the second is 40% below its reference. Rates change, and the cheaper token route is not automatically the better fallback. Measure cost per validated task.

One key and wallet make cross-family attempts operationally simpler, while a model allowlist prevents arbitrary selection. The same account also covers listed image and video models, but their asynchronous workflows need modality-specific fallback semantics.

When not to use LLM fallback

Do not use fallback to hide:

• a broken request schema;
• an invalid or leaked key;
• depleted account balance;
• missing authorization;
• a model that never passed your evaluation;
• an application bug;
• a side effect without idempotency;
• a cost overrun that should trigger a hard budget stop.

Fallback improves availability only when it is bounded, tested and observable. Otherwise it multiplies latency, cost and failure modes.

Next steps with LumeAPI

Review the LLM API gateway, multi-model API, and production LLM API checklist. For a wider resilience runbook covering timeouts, 429s and circuit breakers, see LLM API timeouts and provider failures.

Sources and limitations

• Official OpenAI Python library for error classes, retry behavior, timeouts and request IDs.
• LumeAPI gateway, pillar registry promises, model catalog, and pricing for the shared endpoint, transparent model choice, allowlists, logs and dated rates.

All Python blocks were syntax-checked on July 20, 2026. No customer inference key was used, so this article makes no observed availability or latency claim. LumeAPI does not currently document managed automatic failover; the implementation shown is application-controlled.