How to Test an OpenAI-Compatible API Endpoint

Last verified: July 20, 2026

Test an OpenAI-compatible API endpoint in layers: authentication, model discovery, a non-streaming chat request, streaming, error behavior, and only then the advanced features your application uses. Passing one chat.completions.create() call proves basic connectivity; it does not prove tool calling, structured output, retries, token accounting, or model behavior are interchangeable.

For LumeAPI, use https://api.lumeapi.site/v1 as the SDK base URL. The live /v1/models route returned 401 without a bearer key on July 20, 2026, confirming that model discovery is authenticated. No customer inference key was available for this article, so the test suite below is executable but does not claim billable end-to-end results.

Define what compatible means for your application

OpenAI compatibility is a request-and-response contract, not a quality guarantee. Create a pass/fail list from the features your production code actually depends on.

If your application only sends ordinary text, the first five rows may be enough for a trial. An agent or extraction pipeline needs the advanced tests too.

Install and configure the test client

python -m pip install --upgrade openai
export LUMEAPI_KEY='your-lumeapi-key'
export LUMEAPI_MODEL='gpt-5.4-mini'

Keep the key in an environment variable. Do not print the full authorization header in CI output.

import os
from openai import OpenAI

BASE_URL = os.getenv('LUMEAPI_BASE_URL', 'https://api.lumeapi.site/v1')
MODEL = os.getenv('LUMEAPI_MODEL', 'gpt-5.4-mini')

client = OpenAI(
    api_key=os.environ['LUMEAPI_KEY'],
    base_url=BASE_URL,
    timeout=20.0,
    max_retries=0,
)

Retries are disabled during the compatibility test so one failing request is not hidden by an automatic retry. Production settings can be different after you understand the errors.

Test authentication and list models

Start with model discovery before spending tokens.

def test_models() -> set[str]:
    page = client.models.list()
    ids = {item.id for item in page.data}
    if not ids:
        raise AssertionError('The endpoint returned no model IDs.')
    if MODEL not in ids:
        raise AssertionError(f'{MODEL!r} is not available to this key.')
    return ids

A model list can be account-scoped. Treat the response as the available set for that key at that time, not a permanent universal catalog. If the gateway does not expose model listing, require an authoritative, dated catalog instead. LumeAPI provides both the authenticated route and a public model catalog.

Test a minimal chat completion

Use a deterministic, low-output task. Do not start with a long production prompt.

def test_chat() -> None:
    response = client.chat.completions.create(
        model=MODEL,
        messages=[
            {'role': 'system', 'content': 'Return exactly the word READY.'},
            {'role': 'user', 'content': 'Connectivity check'},
        ],
        temperature=0,
        max_tokens=10,
    )
    if not response.choices:
        raise AssertionError('No choices returned.')
    text = response.choices[0].message.content or ''
    if not text.strip():
        raise AssertionError('The first choice contains no text.')
    print({'model': response.model, 'text': text.strip()})

Do not require the text to equal READY byte for byte unless exact formatting is part of your contract. The main connectivity test checks the response shape; a separate prompt regression suite should score output behavior.

Test streaming separately

A non-streaming success says nothing about Server-Sent Events, disconnects, or chunk parsing.

def test_stream() -> None:
    stream = client.chat.completions.create(
        model=MODEL,
        messages=[{'role': 'user', 'content': 'Count from one to three.'}],
        stream=True,
        max_tokens=30,
    )
    chunks = 0
    text_parts: list[str] = []
    for event in stream:
        chunks += 1
        if event.choices:
            delta = event.choices[0].delta.content
            if delta:
                text_parts.append(delta)
    if chunks == 0 or not ''.join(text_parts).strip():
        raise AssertionError('Streaming produced no usable text.')

For production, also force a client disconnect and confirm your application closes resources, records the incomplete result, and does not execute a partial tool call.

Test negative cases deliberately

A compatible endpoint should fail predictably. Run these in a non-production account or test environment:

1. Omit the bearer key and expect an authentication failure.
2. Send a clearly nonexistent model ID and expect a non-retryable request/model error.
3. Send an invalid message shape and expect validation failure.
4. Use a short timeout to confirm timeout classification.
5. If you can safely reach a rate limit, verify the 429 body and headers without creating a retry storm.

import openai

def test_unknown_model() -> None:
    try:
        client.chat.completions.create(
            model='definitely-not-a-real-model-id',
            messages=[{'role': 'user', 'content': 'test'}],
        )
    except (openai.BadRequestError, openai.NotFoundError):
        return
    raise AssertionError('Unknown model did not produce the expected error class.')

Do not assert that every compatible gateway uses the same error message text. Status classes and a stable machine-readable code are more useful contracts.

Run the smoke suite with useful exit codes

def main() -> int:
    tests = [test_models, test_chat, test_stream, test_unknown_model]
    failures: list[str] = []
    for test in tests:
        try:
            test()
            print(f'PASS {test.__name__}')
        except Exception as exc:
            failures.append(test.__name__)
            print(f'FAIL {test.__name__}: {type(exc).__name__}: {exc}')
    return 1 if failures else 0

if __name__ == '__main__':
    raise SystemExit(main())

A nonzero exit code lets CI stop a rollout. Keep secrets redacted and store only the model, test name, status, latency, request ID and sanitized error category.

Add feature-specific contract tests

Only test what you plan to use, but test it deeply:

• Tool calling: required vs automatic tool selection, argument JSON, parallel calls, result messages and loop termination.
• Structured output: supported response_format, strict JSON Schema, refusal cases and local validation.
• Vision or media inputs: exact input type, file limits and model support.
• Long context: token limits, truncation behavior and cost.
• Streaming tools: partial arguments and event assembly.

Read the dedicated tool-calling Python guide and structured-output Python guide rather than treating a basic chat pass as evidence for advanced parity.

Decide whether the endpoint passed

Use three outcomes:

• Pass: every required contract test passes for the exact model IDs you will deploy.
• Conditional pass: basic chat works, but optional features need an application workaround or a different model.
• Fail: authentication, required response fields, error handling, or a critical feature is incompatible.

Record the SDK version, base URL, model ID, date and test commit. Compatibility can change as SDKs, gateways and models evolve. Re-run the suite during model upgrades and before increasing traffic.

Why LumeAPI fits this workflow

LumeAPI uses one OpenAI-compatible base URL and one key for supported GPT, Claude and Gemini text models. The same account and USD wallet also cover listed image and video models, although their request flows are modality-specific. Exact model IDs are visible in the catalog, and per-call logs expose model, tokens, cost and latency for production review.

That reduces integration duplication, but it does not remove the need for tests. You still choose the model ID and verify the features your application relies on. Start with the OpenAI-compatible API overview, inspect current models, and use the Python quickstart for a first request.

Sources and limitations

• OpenAI Models API reference for the standard list shape.
• Official OpenAI Python library for custom base URLs, errors, retries and request IDs.
• LumeAPI docs, model catalog, and OpenAI-compatible API for the live gateway URL, authentication, catalog IDs and product behavior.

All Python blocks were syntax-checked on July 20, 2026. The public route and unauthenticated 401 behavior were checked; no customer inference key was used, so output, latency and advanced-feature results must be verified with your own key and selected models.