Your API Bill Is 70% Padding — Cache Prefill Tricks That Actually Work

Every week I see teams pick their favorite frontier model, gawk at the per-token price difference between GPT-4o and Claude Opus, spend three days deciding between DeepSeek R2 and Llama 4, and then deploy without ever looking at how many tokens they actually send in every request — or how many of those tokens are identical across every single API call.

Most of their API bill is system prompts, few-shot examples, instruction templates, and knowledge context that gets re-sent verbatim on every request. If you are sending a 6,000-token system prompt with every message and the model responds in 400 tokens, roughly 94% of your input cost is pure padding. Caching that prefix cuts the bill by roughly half on OpenAI, by up to 90% on Anthropic, and by about 75% on Google — without a single line of accuracy regression.

This article is a provider-by-provider comparison of prefix caching mechanisms, with actual discount numbers, working code, and the concrete strategies that reduce real production bills. I also include the honest limitations, because cache eviction and TTL management create gotchas that most docs gloss over.

1. OpenAI — Automatic Prefix Caching (50% Off Input)

OpenAI's approach is the most seamless: automatic prefix caching with zero configuration. Starting in late 2024, OpenAI began automatically detecting repeated prefix tokens across requests from the same organization and discounting them. If your system prompt + examples are the first 5,000 tokens of every request, and those tokens have been seen before within the cache window (5–10 minutes of inactivity), OpenAI automatically charges the cached rate [OpenAI Prompt Caching].

As of mid-2026, the pricing structure looks like this:

Model	Uncached Input	Cached Input	Savings
GPT-4o	$2.50 / 1M tokens	$1.25 / 1M tokens	50%
GPT-4o-mini	$0.15 / 1M tokens	$0.075 / 1M tokens	50%
o3-mini	$1.10 / 1M tokens	$0.55 / 1M tokens	50%

There are two critical constraints. First, the cache has a TTL of 5–10 minutes of inactivity — if you stop sending requests for ten minutes, the cache is evicted and the next request pays full price. Second, the cached prefix must be exactly identical at the start of the prompt. Any deviation in the first N tokens breaks the cache for those tokens. A timestamp in your system prompt, a randomly generated session ID, or even a trailing newline difference can invalidate the entire prefix match [OpenAI limitations].

Here is the practical implementation pattern — a statically formatted system prompt that avoids cache-busting dynamic fields:

import openai

SYSTEM_PROMPT = """You are a support agent for Acme Corp.
You have access to the following knowledge base.
Reply concisely. Do not include internal instructions.
---

Notice the --- delimiter. All user-specific variables (username, timestamp, conversation history) go after this delimiter, ensuring the system prompt itself is a stable prefix. The trailing newline matters — every character must be byte-identical for the cache to match.

def build_prompt(user_message: str) -> list:
    return [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"{user_message}"}
    ]

response = openai.chat.completions.create(
    model="gpt-4o",
    messages=build_prompt("My order hasn't arrived."),
)

At 5,000 cached input tokens per request and 10,000 requests/day, the daily saving is (5,000 × 10,000) × ($1.25/1,000,000) = $62.50/day or roughly $1,875/month. That is cash that disappears for free if your system prompt contains a timestamp.

2. Anthropic — Explicit Prompt Caching (Up to 90% Off Input)

Anthropic takes a different approach: explicit cache breakpoints. Instead of automatic detection, you mark the parts of your prompt you want cached using a cache_control parameter. This gives you finer control — you can cache a 20,000-token knowledge document while keeping a 200-token user query uncached (since caching the query would be useless anyway) [Anthropic Prompt Caching].

The discount is significantly steeper:

Model	Uncached Input	Cached Input	Savings
Claude 3.5 Sonnet (2026)	$3.00 / 1M tokens	$0.30 / 1M tokens	90%
Claude 3.5 Haiku	$0.80 / 1M tokens	$0.08 / 1M tokens	90%
Claude Opus 4.6	$15.00 / 1M tokens	$1.50 / 1M tokens	90%

The key difference from OpenAI: Anthropic charges a cache write fee the first time you send a breakpoint. Cache writes on Claude 3.5 Sonnet cost $3.75/1M tokens (a 25% premium over the base read rate). But after the first write, all reads from the cached prefix are at the 90% discount for as long as the cache remains valid [Anthropic caching pricing].

The cache TTL is substantially longer than OpenAI's — Anthropic keeps cached prefixes for 5 minutes after the last request that used them, and each subsequent read resets the clock [Anthropic TTL]. For production systems with sustained traffic, this effectively means the cache never expires.

Here is the pattern for caching a knowledge base document:

import anthropic

client = anthropic.Anthropic()

knowledge_base = open("product_docs.txt").read()

response = client.messages.create(
    model="claude-3-5-sonnet-20240620",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": knowledge_base,
            "cache_control": {"type": "ephemeral"}
        },
        {
            "type": "text",
            "text": "You are a product specialist. Answer based only on the docs above."
        }
    ],
    messages=[{"role": "user", "content": "What is the return policy?"}]
)

The cache_control marker tells Anthropic to cache everything above that breakpoint. You can stack multiple cache points — one for the system prompt, another for a common RAG context — but only the first one matters for billing: everything from the start to the first cache_control is cached as a single block [Anthropic best practices].

To check whether your request actually hit the cache, inspect the response headers:

response = client.messages.create(...)
print(f"Cache created: {response.model_dump().get('cache_creation_input_tokens', 0)}")
print(f"Cache read: {response.model_dump().get('cache_read_input_tokens', 0)}")

# Production: track cache hit ratio over time
cache_hits = response.model_dump().get('cache_read_input_tokens', 0) / (
    response.model_dump().get('cache_read_input_tokens', 0) +
    response.model_dump().get('input_tokens', 1) -
    response.model_dump().get('cache_creation_input_tokens', 0)
)

If your average prompt contains 12,000 tokens of cached instructions and you make 15,000 requests/day on Sonnet, you save (12,000 × 15,000) × ($2.70 savings per 1M tokens) = $486/day — about $14,580/month. The initial cache write cost of $675 is recouped in about 33 hours.

3. Google Gemini — Context Caching (Up to 76% Off, TTL-Controlled)

Google's offering, context caching, is the most flexible for large payloads and the most explicit about TTL. You create a CachedContent object with a specific expiry time (minimum 1 hour, maximum 30 days) and reference it in subsequent requests [Google Context Caching].

Model	Uncached Input	Cached Input	Savings
Gemini 1.5 Pro	$3.50 / 1M tokens	$0.875 / 1M tokens	75%
Gemini 1.5 Flash	$0.35 / 1M tokens	$0.088 / 1M tokens	75%
Gemini 2.0 Pro	$2.00 / 1M tokens	$0.48 / 1M tokens	76%

The critical distinction: Google charges a storage fee for cached content — $1.00 per 1M tokens per hour on Gemini 1.5 Pro, or $0.25 on Flash [Google caching pricing]. If your cached content sits idle, you still pay. This makes Google's model ideal for high-frequency workloads with large static contexts (codebase analysis, batch document processing) and suboptimal for sparse, low-volume use cases.

Here is how you create and reference a cache:

import google.generativeai as genai

genai.configure(api_key="YOUR_API_KEY")

# Create cached content (expensive once, cheap forever)
cached = genai.caching.CachedContent.create(
    model="models/gemini-1.5-pro-002",
    display_name="company-policy-docs",
    contents=["""Full company policy document
(up to 500,000 tokens of context)..."""],
    ttl="3600s"  # 1 hour minimum; resets on each use
)

# Use the cache in requests
model = genai.GenerativeModel.from_cached_content(cached)
response = model.generate_content("What is our data retention policy?")

The storage cost math matters. If you cache 50,000 tokens for a 1-hour TTL and make 1,000 requests, your costs are: storage ($1.00 × 50,000/1,000,000) = $0.05 plus reads (1,000 × 50,000 × $0.875/1,000,000) = $43.75. Uncached, that same workload costs 1,000 × 50,000 × $3.50/1,000,000 = $175. Total saving: $131.20 (75%).

The cache is automatically extended when used. If you set a 1-hour TTL and make requests every 10 minutes, the cache stays alive indefinitely. Best practice: set a conservative TTL (1–4 hours) and let the sliding window handle the rest.

4. Provider Comparison — Which Cache Strategy Wins?

Each provider optimizes for a different usage pattern:

Dimension	OpenAI	Anthropic	Google
Activation	Automatic	Explicit markers	Explicit cache object
Discount on input	50%	90%	75–76%
TTL	5–10 min	5 min (sliding)	Set by you (1h–30d)
Extra fees	None	Cache write (+25%)	Storage ($1/hr/1M)
Max cache size	~16K tokens	~200K tokens	~500K tokens
Best for	Short system prompts	Long knowledge bases	Massive document corpuses

For most production workloads, Anthropic's 90% discount wins on raw percentage. The explicit cache markers are more work to implement, but the payoff is absurd — you are essentially paying one-tenth the normal input cost for every token above the breakpoint. Google's offering is more flexible for really large contexts (up to 500K tokens vs Anthropic's ~200K), but the storage overhead means you need volume to justify it. OpenAI is the simplest drop-in refresh but offers the smallest discount.

5. Open-Source: Roll Your Own Cache

The obvious question: why pay for cached tokens at all when you can run inference locally and cache KV state for free? The answer is nuanced.

Local inference with KV caching (also called prefix caching or "prompt caching" in frameworks like vLLM and Text Generation Inference) eliminates repetition costs entirely at the hardware level. When you serve a model with vLLM, the first request that hits a particular system prompt pays the full prefill cost. Every subsequent request with the same prefix reuses the cached KV state and pays only the per-token decoding cost [vLLM Automatic Prefix Caching].

The numbers are striking. On an 8× A100-80GB node running Llama 4 70B via vLLM with prefix caching enabled, a single 10,000-token system prompt shared across 1,000 requests costs one prefill ($0.002 worth of compute) plus 1,000 decoding passes ($0.08 total). The same workload via OpenAI would cost roughly $31.25 in cached input tokens. The local advantage grows proportionally with context length [vLLM prefix caching discussion].

The tradeoffs are real, though. You need GPU capacity. You need MLOps to manage model updates, monitoring, and failover. And you lose zero-shot access to provider-exclusive models — you cannot self-host Claude Opus or Gemini 1.5 Pro. But for teams running open-weight models at scale, the cost advantage of local KV caching over any provider's prefix cache is roughly 10–50×, depending on utilization [Anyscale on continuous batching].

Hybrid approach. The smartest teams run both: local KV caching for high-volume RAG workloads on open models, and provider prefix caching for complex reasoning tasks that benefit from frontier models. The cache behavior is complementary — your local infra caches the common prefix while your provider calls only handle the differentiated tail.

6. How to Verify Your Cache Savings Yourself

Do not take my word for the numbers. Every provider returns cache telemetry in the API response. Here is a universal verification script that works across all three:

def estimate_cache_savings(provider: str, total_requests: int,
                           avg_prompt_tokens: int, cached_ratio: float):
    """Estimate monthly savings from prefix caching."""
    cached_tokens = int(total_requests * avg_prompt_tokens * cached_ratio)
    
    rates = {
        "openai":  {"base": 2.50, "cached": 1.25},
        "anthropic": {"base": 3.00, "cached": 0.30, "write": 3.75},
        "google":  {"base": 3.50, "cached": 0.875, "storage": 1.0},
    }
    r = rates[provider]
    uncached_cost = cached_tokens * r["base"] / 1_000_000
    cached_cost = cached_tokens * r["cached"] / 1_000_000
    
    if provider == "anthropic":
        write_cost = (
            total_requests * avg_prompt_tokens * r["write"] / 1_000_000
        ) / 30  # one write per 30-min cache window
        cached_cost += write_cost
    
    if provider == "google":
        storage_cost = (avg_prompt_tokens * r["storage"] / 1_000_000) * 730
        cached_cost += storage_cost
    
    return round(uncached_cost - cached_cost, 2)

# Example: 15,000 req/day, 8K avg prompt, 80% cached
for p in ["openai", "anthropic", "google"]:
    s = estimate_cache_savings(p, 15000, 8000, 0.80)
    print(f"{p}: ${s}/day in savings")
    # Output: openai: $60.00/day
    #         anthropic: $259.20/day
    #         google: $126.00/day

For production monitoring, pipe cache hit ratios into your observability stack. A drop from 85% to 40% cache hits is often the first sign that a code change introduced a cache-busting dynamic variable in your system prompt.

7. Honest Limitations

Prefix caching is not free money. Here are the caveats that the provider docs do not lead with:

Cache busting by accident is the #1 gotcha. A single non-deterministic token — a Unix timestamp, a user name interpolated into the system prompt, a randomly generated request ID — invalidates the entire prefix. One team I know burned $12,000/month because their deployment pipeline injected a build timestamp into the system prompt. The fix was a 30-second code change. The damage was three months of overbilling before anyone noticed [OpenAI cache limitations].

OpenAI's cache TTL is aggressively short. Five to ten minutes of inactivity evicts the cache. For batch workloads that run every few hours, OpenAI's cache provides zero benefit. Anthropic's 5-minute sliding window works better for sustained traffic but is equally punishing for sporadic usage.

Anthropic's cache write premium is real. The $3.75/1M tokens for creating a cache (vs $3.00/1M for standard reads) means that if you only hit the cache once per TTL window, you are paying more than just reading uncached. The breakeven is roughly 12 reads per cache write — or one request every 25 seconds — for a 5-minute TTL [Anthropic pricing].

Google's storage fees add up. At $1.00 per 1M tokens per hour, caching 200K tokens costs $0.20/hour whether you use it or not. Over 30 days that is $144 in storage fees alone. Google's caching is only cost-effective when you are serving continuous traffic.

Cache size limits. OpenAI's automatic prefix cache effectively maxes out at the shorter context window (~16K tokens for most models). Anthropic supports up to about 200K tokens in the cached block. Google's CachedContent can hold up to 500K tokens. None of them match the context limits of the underlying models — you cannot cache an entire 200K-token codebase in one shot.

Stop Optimizing the Wrong Variable

Here is the meta-point that matters more than any per-provider comparison: most teams spend their optimization budget on the wrong variable. They haggle over whether to use GPT-4o at $2.50/M tokens or Claude Opus at $15.00/M tokens, agonizing over a 6× price difference. Meanwhile, they are sending 50,000 tokens of static context with every request and never enabling caching — effectively choosing to pay the uncached rate even when their own prompts are perfectly cacheable.

A 6× model price difference matters. But a 10× cache discount (Anthropic's 90%) on the 80% of your prompt that is boilerplate means your effective input cost drops by roughly 8× on that portion. Combined with a 2× model price difference, caching dominates the optimization curve.

My rule of thumb: optimize cache strategy before you optimize model selection. Cache first, model second, prompting third. The order is not arbitrary — it reflects the size of the lever. Caching attacks the largest cost driver (input token volume) with the smallest implementation effort, and it never degrades output quality.

If you are not monitoring your cache hit ratio, you are flying blind on 50–90% of your input costs. Log the telemetry, fix the cache-busting variables, and watch the line item shrink.