Semantic Search API Cost Too High? Cut Embedding Bills in 2026

Last verified: July 24, 2026

Short path: Start with the GPT API guide, compare current AI API pricing, then see the cheap OpenAI-compatible API option.

A semantic search API cost too high problem usually is not one bad embedding invoice. It is three meters running at once: document embeddings during ingestion, query embeddings on every search, and rerank or chat calls after retrieval. The expensive part is often the stage you send too much text to—not the model you blame first.

Quick Answer

In short

Treat semantic search as a pipeline cost problem, not an embedding-model problem. The quickest cut usually comes from reducing candidate text before reranking and preventing chat generation from becoming the default response to every query.

Once the pipeline is disciplined, route compatible answer-generation traffic through a lower-cost GPT tier. A 70% token-rate reduction is useful, but it will not save a retrieval design that sends 40 chunks into every reranker call.

What most guides get wrong

The common advice is: “Switch to a cheaper embedding model.” That can help, but it misses the meter that grows faster.

Embedding cost scales with tokens you embed. Reranking cost scales with queries × candidates × tokens per candidate. If you retrieve 50 chunks, each 350 tokens long, then rerank all of them for every query, you have created a 17,500-token evaluation job before you generate a single answer. Add a query, metadata, instructions, and duplicated chunk headers, and the reranker sees even more.

The second mistake is treating answer generation as mandatory. A search page that could return three cited passages often triggers a 1,000-token “helpful” answer anyway. That is not semantic search; that is a chat product attached to search economics.

Do not start by shaving fractions off embedding cost. First remove unnecessary tokens and unnecessary calls. Then negotiate model price.

A realistic production scenario

Priya owns a support-document search service for a B2B SaaS product. The stack is straightforward: documents are chunked into roughly 450-token passages, stored in a vector database, retrieved with top-40 similarity search, reranked, then passed into a GPT answer step.

By the second month, the team is handling 1.8 million searches a week. Their dashboard says “embeddings,” so that is where they look first. The actual request traces tell a different story: 96% of queries embed fewer than 30 tokens, but every query reranks 40 passages and sends the top eight into a chat completion. A typo search for “SAML scim provisioning” can produce 4,000 to 6,000 prompt tokens before the answer model writes anything.

The turning point is not a new vector database. Priya changes retrieval to top-80 only for recall, applies cheap metadata and score filters, reranks 12 candidates, and skips answer generation for navigational searches. The chat bill drops because the system stops asking a generation model to narrate every lookup.

Expert take

The useful way to investigate a vector search LLM bill is to draw the pipeline as four separate ledgers:

1. Ingestion embeddings: new or changed document tokens.
2. Query embeddings: query tokens, plus any rewritten-query tokens.
3. Reranking: candidate count multiplied by candidate length.
4. Answer generation: retrieved context plus output tokens.

Only the first two are fundamentally “embedding spend.” The other two can become much larger because they repeat on every search.

The non-obvious failure mode is duplicate work. Teams often re-embed an entire document set after changing chunk metadata, even when the text and embedding inputs did not change. They also rerank chunks that are nearly identical because chunk overlap, document versions, and access-control variants all survive into the candidate list. That gives the reranker three chances to score the same paragraph.

The decision rule I use is blunt: if a candidate cannot plausibly appear in the final answer, do not pay to rerank it. If a result page can satisfy the user with ranked links and snippets, do not pay for chat generation. If a document hash has not changed, do not embed it again.

This advice does not apply everywhere. Legal research, high-stakes support, and complex knowledge workflows may need broad candidate retrieval and a generative answer for every request. In those cases, preserve recall and audit quality first. Also keep native provider flows when you depend on provider-specific Batch processing, prompt caching, embedding endpoints, or reranking features that an OpenAI-compatible Chat Completions gateway does not expose.

LumeAPI is an independent third-party gateway, not OpenAI, Anthropic, or Google. Use it for compatible text-generation calls where its catalog and endpoint fit your workload; do not assume it replaces every native API capability.

Find the cost leak before changing vendors

Instrument one request record per semantic-search query. You need enough fields to reconstruct cost without reading raw user content:

A useful daily report is not “total AI spend.” It is this:

embedding_ingestion_cost
+ embedding_query_cost
+ rerank_cost
+ answer_generation_cost
= semantic_search_cost

Then divide each line by searches, documents ingested, and successful answer sessions. That tells you whether you have an ingestion issue, a query-volume issue, or an expensive post-retrieval design.

The math that makes reranking look cheap until it is not

Use these formulas before changing a model.

Embedding ingestion

monthly_embedding_tokens =
  tokens_in_new_or_changed_documents

monthly_embedding_cost =
  monthly_embedding_tokens / 1,000,000 × embedding_rate_per_1M

The phrase “new or changed” matters. Hash normalized chunk text before embedding. If the hash is unchanged, reuse the vector. Metadata changes such as a display title, a tag, or a UI collection label should not automatically trigger re-embedding unless that metadata is part of the embedding input.

Query embeddings

monthly_query_embedding_tokens =
  monthly_queries × average_query_tokens

Query embeddings are often inexpensive individually, but query rewriting can quietly multiply them. A pipeline that embeds the original query, two rewritten queries, and a hypothetical answer is doing four embedding calls per search. That may be justified for difficult research retrieval; it is wasteful for “reset my password.”

Reranking

monthly_rerank_tokens =
  monthly_queries
  × candidates_reranked_per_query
  × average_candidate_tokens

If you handle 20 million searches per month, rerank 20 candidates, and each candidate averages 300 tokens:

20,000,000 × 20 × 300 = 120,000,000,000 candidate tokens/month

That is 120 billion candidate tokens before query text, reranker instructions, or duplicate content. Reducing reranked candidates from 20 to 10 is not a tuning tweak. It cuts that stage roughly in half.

Answer generation

monthly_chat_input_tokens =
  generated_answers
  × average_prompt_tokens

monthly_chat_output_tokens =
  generated_answers
  × average_completion_tokens

Answer generation is where a GPT gateway discount can be easy to verify because Chat Completions input and output token meters are explicit.

For background on embedding workflows, see the official OpenAI embeddings guide. For current provider reference rates, check the official OpenAI API pricing page, then compare those rates with the LumeAPI catalog before deployment.

Cut the pipeline in the order that preserves retrieval quality

1. Make embedding work idempotent

Store a stable hash of the exact text sent for embedding. Include the embedding model identifier in the hash key so a deliberate model migration produces a clean re-index, while an unchanged chunk does not.

A practical key looks like:

embedding_key = sha256(model_id + normalized_chunk_text)

Normalize whitespace and remove volatile fields before hashing. Do not include timestamps, UI labels, indexing-job IDs, or random document ordering.

Also split ingestion into two queues:

• Content changed: chunk, embed, and upsert.
• Metadata changed: update metadata only.

That distinction prevents a permission-label update from becoming a million-document embedding job.

2. Retrieve broadly, then remove obvious losers cheaply

Approximate nearest-neighbor retrieval exists to find candidates efficiently. Use it for recall. Do not confuse “top 100 vectors are nearby” with “top 100 passages deserve expensive semantic evaluation.”

Before reranking, apply rules that do not require an LLM:

• Exclude documents the user cannot access.
• Deduplicate near-identical chunks and document revisions.
• Prefer current versions over archived copies.
• Drop empty, boilerplate-heavy, or navigation-only chunks.
• Limit candidates per source document when overlap creates clones.
• Use language, product area, and date metadata when available.

These filters improve economics and often improve answer quality because the reranker receives a cleaner contest.

3. Rerank only the decision boundary

A reranker is most valuable when vector similarity cannot distinguish between plausible candidates. It is less valuable when the top results are obviously irrelevant or obviously duplicates.

Start with a small evaluation set and test candidate counts such as 8, 12, and 20. Measure recall at the final context window, not just reranker scores. If top-12 preserves the same answer-supporting passage rate as top-30, keep 12.

A useful policy:

high vector-score gap → skip rerank
uncertain or mixed-intent query → rerank top 12
high-stakes or broad research query → rerank a larger set

The exact thresholds are application-specific. The point is to make reranking conditional instead of automatic.

4. Stop generating prose for navigational searches

Searches fall into different jobs:

A generative answer should be a product choice, not a tax attached to every search box. Put the “generate answer” control behind a confidence threshold, explicit user action, or a query classifier you have evaluated.

What a 20 million-query month looks like

The following example isolates the chat-generation portion of a semantic-search system. It does not claim that LumeAPI supplies embeddings or reranking. Embedding and rerank rates vary by provider and contract, so calculate those stages from your actual invoice and token logs.

Assumptions

• 20 million monthly searches
• 2% of searches require a generated answer: 400,000 answer calls
• 1,200 input tokens per generated answer after retrieval and context trimming
• 180 output tokens per generated answer
• Monthly chat input: 480 million tokens
• Monthly chat output: 72 million tokens
• Rates below are per 1 million input / output tokens from the July 22, 2026 LumeAPI catalog

For gpt-5.6-terra, the calculation is:

Official:
480M × $2.50 / 1M = $1,200.00 input
 72M × $15.00 / 1M = $1,080.00 output
Total = $2,280.00

LumeAPI:
480M × $0.75 / 1M = $360.00 input
 72M × $4.50 / 1M = $324.00 output
Total = $684.00

That is a real reduction, but note the larger architectural point: if you reduce generated-answer calls from 400,000 to 200,000 by returning search results when that is enough, you halve this entire line before applying a rate discount.

Do not choose gpt-5.4-mini solely because it is cheaper. Run it against your own answer-quality set, citation behavior, formatting requirements, and refusal patterns. A cheaper answer model that creates more follow-up searches can raise total cost.

Route compatible answer generation through LumeAPI

For compatible text-generation traffic, LumeAPI uses the OpenAI-style Chat Completions endpoint at:

https://api.lumeapi.site/v1

This curl request sends a retrieved context window to gpt-5.6-terra. Keep the context deliberately bounded; sending every retrieved chunk defeats the point of fixing retrieval economics.

curl https://api.lumeapi.site/v1/chat/completions \
  -H "Authorization: Bearer $LUMEAPI_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-5.6-terra",
    "messages": [
      {
        "role": "system",
        "content": "Answer only from the supplied search passages. If the passages do not support an answer, say so. Cite passage IDs."
      },
      {
        "role": "user",
        "content": "Question: How do I configure SCIM provisioning?\n\nPassage A [doc-104]: ...\n\nPassage B [doc-231]: ..."
      }
    ],
    "max_tokens": 220
  }'

For a production integration, log the model ID, input tokens, output tokens, retrieval count, and answer decision alongside the request ID. Without those fields, a cheaper gateway rate can hide a growing context problem.

You can also compare this approach with the embedding API cost guide and the RAG cost reduction guide. If repeated context is your dominant issue, review prompt caching and API costs before redesigning retrieval.

A practical cost-control checklist

Use this checklist before declaring an embedding plus rerank API expensive:

• [ ] Hash the exact embedding input and skip unchanged chunks.
• [ ] Separate content updates from metadata-only updates.
• [ ] Record embedding, rerank, and chat tokens as separate cost centers.
• [ ] Measure duplicate candidates after vector retrieval.
• [ ] Cap candidates per source document before reranking.
• [ ] Test rerank top-8, top-12, and top-20 against a relevance set.
• [ ] Trim chunk headers, repeated metadata, and irrelevant document sections.
• [ ] Require a confidence or intent reason before generating an answer.
• [ ] Set a completion-token ceiling for search answers.
• [ ] Evaluate a lower-cost chat model on retrieval-grounded answers, not general chat prompts.
• [ ] Keep native APIs where your workflow requires native embedding, Batch, or cache behavior.
• [ ] Compare monthly token math, not just per-million-token stickers.

FAQ

Why is my semantic search API cost too high even though query embeddings are small?

Your semantic search API cost too high issue is likely downstream of query embeddings. Reranking many long candidates and generating answers with oversized context can cost more than embedding short user queries by orders of magnitude.

Is embedding plus rerank API expensive because rerankers use too many candidates?

Usually, yes. Rerank spend grows with the number and length of candidate passages, so reducing a 40-candidate rerank stage to 12 relevant candidates can have a larger impact than switching embedding models.

How can I reduce semantic search API cost without hurting recall?

Retrieve broadly, then use access filters, deduplication, metadata constraints, and evaluated rerank limits before building the final context. Protect recall at retrieval time; control cost between retrieval and generation.

Should I re-embed my whole corpus after every document update?

No. Re-embed only changed chunk text, using a content hash tied to the embedding model ID. Metadata-only changes should normally update the index record without creating a new embedding request.

Can LumeAPI replace my embedding and rerank provider?

Not necessarily. LumeAPI is useful for OpenAI-compatible text-generation calls from the supplied catalog, but you should not assume native embedding, rerank, Batch, or prompt-cache feature parity without verifying the relevant service documentation.

When should I generate a chat answer instead of returning search results?

Generate an answer when the user needs synthesis, comparison, explanation, or a grounded multi-document response. For navigational queries, return the best document and snippet first; that is often faster, cheaper, and easier to verify.

Next steps

1. Export seven days of request logs and calculate embedding, rerank, and answer-generation cost separately.
2. Run an evaluation that compares reranking 8, 12, and 20 candidates while tracking final answer support and retrieval recall.
3. Move compatible answer-generation traffic to the LumeAPI OpenAI-compatible endpoint, then verify the savings against your actual input and output token mix.