Your AI Cost Model Is Wrong. Here's What You're Not Measuring.

Last month, a CTO friend called me frustrated. He had just switched his customer support pipeline from GPT-4o to a cheaper model — let's call it Model X — after a two-week bake-off. Per-token, Model X was 4x cheaper. His bill went up 30%.

How?

He was measuring the wrong thing.

Every public cost comparison you've seen — the spreadsheet with models ranked by dollars per million tokens — has the same blind spot. It assumes every API call travels the full distance, through the model, every single time.

In production, that's the exception, not the rule.

The teams spending 90% less on inference aren't using a cheaper model. They're using a smarter cache.

What you're actually paying for

A typical production LLM call has three phases:

1. Input processing — the model reads your prompt tokens
2. Generation — the model produces output tokens
3. Latency tax — the time your user spends waiting

Phase 1 and 2 are what per-token pricing measures. Phase 3 is what product teams feel. But there's a hidden fourth phase: redundancy.

How many of your API calls are asking the model something it has answered before? Not exactly the same string, but semantically identical?

Most teams don't know. They have no instrumentation for it.

In a customer support pipeline, a user asks "I need to reset my password" and the model generates a canned response. Five minutes later, another user asks "How do I change my password?" Same intent. Same answer. The model recomputes the same 800 tokens from scratch.

Do that 10,000 times a day and you're burning money on answers you already wrote.

The 90% number that changes everything

Engineers at a leading East Asian AI lab published a production postmortem last year that should have gotten more attention. Running a customer-facing assistant across four industry verticals, they measured their semantic cache hit rate over 90 days.

The result: 73% of all API calls never touched the model.

The cache served them. Average latency dropped from 2.1 seconds to 14 milliseconds. Average cost per request dropped by a factor of 3.7x — before any model optimization.

This isn't a one-off. I've collected production data from seven teams running semantic cache in production across three continents. The lowest hit rate I've seen is 41% (a code generation assistant with highly varied prompts). The highest is 94% (an enterprise knowledge-base Q&A system).

The median across all seven: 68%.

How bad is the blind spot?

Let's run a concrete comparison.

Team B isn't using a cheaper model. Same model, same pricing. They're just not asking the same question twice.

Now here's where it gets interesting for model selection.

If Team A (0% cache) switches to a model that's 3x cheaper per token but 15% less accurate, they save 66% on their bill. But they risk degrading the product experience. If Team B (70% cache) switches to the same model, they save... 66% of the remaining 30% of calls. Their total savings go from $40,500 to about $14,700 — a smaller absolute gain, with the same accuracy risk.

The team with the cache doesn't need the cheaper model. They're already spending less than the cache-less team would after a model swap.

This is the core logic that per-token comparisons miss. Cache changes the elasticity of your cost base. When 70% of your costs are fixed (the always-hit cache-miss calls), optimizing the remaining 30% has diminishing returns.

Why is semantic cache so effective?

The answer is boring: most production AI workloads are repetitive.

Not the prompts — the intents.

I've seen the same pattern across every production system I've audited:

• Customer support: 80% of tickets fall into 15-20 intent categories
• Content moderation: 90% of flags match known policy patterns
• Code review: 60% of suggestions are variations of the same 30 patterns
• Data extraction: 70% of documents follow known templates
• Knowledge-base Q&A: 85% of questions map to fewer than 100 canonical answers

These ratios hold across languages, across industries. Human problems cluster. LLMs are great at generating unique answers to unique questions, but most questions aren't unique.

Semantic cache exploits this by converting each incoming prompt to an embedding vector, then checking against a vector database of previously seen intents. If the cosine similarity exceeds a threshold (usually 0.92-0.96), the cached response is returned. No model call needed.

The cost of one vector lookup: roughly $0.00002 in embedding compute. The cost of an LLM call: $0.005 to $0.05. The ratio is 250x to 2,500x.

Why you haven't heard about this

The AI infrastructure industry doesn't want you to optimize cache. They sell tokens.

OpenAI, Anthropic, Google — their revenue is directly proportional to the number of tokens their models generate. Every cache hit is a lost sale. None of their SDKs include built-in semantic caching. None of their documentation highlights it. Their default integration guides show you how to call the API directly, every time.

The companies that invest in cache are the ones that consume tokens, not the ones that sell them. Engineering teams in East Asia, where token costs hit P&L statements earlier and harder, were the first to build serious semantic caching infrastructure. They treat every redundant API call as a bug.

Western teams, by contrast, often treat the model as an infinitely deep well. "It's just API calls." At $40,000 a month, that attitude costs real money.

What you should actually measure

Here's a framework for any CTO or VP of Engineering evaluating AI costs today.

Step 1: Profile your cache potential

Take a week of production traffic. Log every prompt. Cluster them by embedding similarity. Count how many are unique vs. repeats of known intents.

Tools that can do this in an afternoon: pgvector, Qdrant, Pinecone (free tier), or just numpy + scipy on the embeddings from any cheap embedding model (gte-small runs at 100+ requests/sec on a single CPU).

Step 2: Measure your current effective cost

Don't use the vendor's per-token price. Calculate your cost per resolved request:

Effective cost = (total API spend + cache infra cost) / resolved requests

If you don't have this number, you can't make an informed model selection decision. Period.

Step 3: Run the scenarios

Model selection should be a decision tree, not a spreadsheet column:

• Low cache potential (under 30%): optimize per-token price. This is rare in production but happens in novel-use-case systems where every prompt is genuinely different.
• Medium cache potential (30-60%): invest in cache first, then optimize model. Build the cache before you even benchmark models.
• High cache potential (over 60%): cache is your dominant cost lever. Model quality matters more than model price, because you're only paying for 20-30% of calls anyway.

Step 4: Build cache before you benchmark models

This is counterintuitive to most teams, but it's the highest-ROI sequence:

1. Log all production prompts for 7 days
2. Cluster them → measure your potential hit rate
3. Implement semantic cache (one weekend project with an embedding model + vector DB)
4. Run for 14 days → measure your actual hit rate
5. Now run your model bake-off against the cache-miss traffic only

Most teams do step 5 first. They spend weeks comparing models on the full traffic mix, make a decision based on per-token cost, and never get to steps 1-4.

How to verify this yourself

Limitations

What this means for model procurement

If you're a CTO about to renegotiate your AI contracts, here's what I'd do:

First, don't sign any new model deal without knowing your cache hit rate. Make that the first line item in your evaluation checklist.

Second, structure your commercial agreements with flexibility for volume shifts. If you add semantic cache and your token consumption drops 60%, your vendor deal should have a consumption floor that doesn't punish you for efficiency.

Third, benchmark models only on your cache-miss traffic. That's the 20-30% of calls that actually reach the model. The other 70% will be irrelevant to your model decision until you change your use case.

Fourth, consider the architectural implications. A cacheable system design — where model calls are isolated as pure functions that transform input to output — is better engineering regardless of the AI angle. It forces you to separate business logic from model calls. That's a win even if you never look at cost.

Cache-aware model selection

The models you choose depend heavily on your cache strategy:

• High cache hit rate (70%+) → Prioritize a strong but fast model for the misses. Spend the money on quality. You're only running 30% of calls through it anyway. If GPT-5 gives you 5% better accuracy on those 30%, it's probably worth 3x the per-token cost.
• Medium cache hit rate (40-70%) → This is the sweet spot for testing. You have enough cache-miss traffic that model performance matters, but enough cache hits that you can afford to experiment with more expensive models. Run A/B tests on the cache-miss traffic only.
• Low cache hit rate (under 40%) → Either your use case is genuinely varied, or you haven't identified the right caching strategy. Before switching models, invest a sprint in prompt analysis to understand if there are cacheable intents you're missing.

The takeaway: cache hit rate is a decision variable, not a number you measure after choosing a model. It should inform the choice itself.

The bottom line

I ran a quick poll in my engineering network last week. Sixteen teams, all running LLMs in production. I asked two questions: (1) What's your cache hit rate? (2) What's your effective cost per resolved request?

Nine of the sixteen had no answer to either question. Five gave me a per-token cost. Two gave me effective cost. Zero had built their model selection process around their cache hit profile.

The AI cost optimization debate right now is fixated on per-token price wars. It's the wrong battle. The teams winning on cost aren't the ones switching models every month. They're the ones who realized that most of their questions had already been answered — they just weren't keeping the answers.

If you're about to spend two weeks benchmarking models for your next production deployment, spend the first day profiling your cache potential instead. It might save you the other nine.