A Seoul-based fintech company needed to process financial documents — contracts, disclosures, regulatory filings — with strict data sovereignty requirements. Korean financial regulations prohibit certain categories of customer data from leaving domestic infrastructure. The team had two options: pay for a closed API and navigate a complex compliance maze, or self-host an open-source model and keep everything onshore.
They chose Llama 3. They spent roughly one-third of what GPT-4 would have cost per transaction. They logged zero compliance incidents. The system has been running for over a year.
If you stopped reading here, you'd conclude this is an open-source victory story. It is. For them.
But here is the part no one tells you: the same decision, applied to a different use case, would have been a financial disaster.
The "free" illusion
"Open-source models are free" is one of the most expensive assumptions in AI infrastructure. The model weights cost nothing to download. But the infrastructure to run them is not free. And the engineering time to keep them running is not free. And the opportunity cost of every hour spent debugging a CUDA out-of-memory error instead of building features is not free.
The numbers tell a clear story. A single H100 GPU costs approximately \$2.85 to \$3.50 per hour on-demand from major cloud providers (as of mid-2026), down from \$8 per hour at peak scarcity in late 2024. AWS cut H100 prices roughly 44% in June 2025, bringing them to about \$3.90 per hour. Spot instances can go lower — GCP's spot H100 runs at \$2.25 per hour, AWS spot near \$2.50. Budget providers like Hyperbolic offer H100 at \$1.49 per hour. Long-term commitments can bring effective costs as low as \$1.90 to \$2.10 per GPU-hour.
That sounds cheap. Until you multiply it.
A 70-billion parameter model like Llama 3.1 requires at least two H100s for reasonable production throughput. At \$3.50 per hour, that's \$168 per day, \$5,040 per month — before you account for scaling, redundancy, or any of the other operational requirements of a production system. A 405-billion parameter model needs eight A100s, hardware costs exceeding \$200,000.
Now compare that to API pricing. Llama 3.1 70B through an API provider costs \$0.10 per million input tokens and \$0.28 per million output tokens. Llama 3.3 70B is even cheaper: \$0.04 per million input, \$0.12 per million output. GPT-4o: \$2.50 per million input, \$10 per million output. The per-token gap is enormous — Llama 3.3 is roughly 60x cheaper on input than GPT-4o.
But per-token cost is not total cost. And total cost is where the open-source tax lives.
The real ledger
Let's build the actual ledger for a production deployment. Not a pilot. Not an experiment. A real system handling real traffic.
Self-hosted open-source (70B model, moderate scale):
- GPU infrastructure: 2 H100s at \$3.50/hr = \$5,040/month
- Engineering overhead: at least one engineer spending 20-30% of their time on infrastructure — GPU memory management, request batching, scaling policies, health checks, failover, security patches, model upgrades. At a \$180,000 salary, that's \$3,000 to \$4,500 per month in allocated cost.
- Model upgrades: every new release requires testing, validation, and potentially a full redeploy. Llama 3 to Llama 3.1 to Llama 3.3 — each migration costs days of engineering time.
- Opportunity cost: every hour spent on infrastructure is an hour not spent on your product.
Total monthly cost (self-hosted): \$8,000 to \$10,000+
API-based (same workload):
- Per-token cost at \$0.10/\$0.28 per million (Llama 3.1 70B API) or similar
- Zero infrastructure engineering
- Zero upgrade migration cost
- Zero GPU provisioning
Which one is cheaper?
The answer depends entirely on your volume. At 10 million tokens per month, the API costs roughly \$1,900 (if using Llama 3.1 70B rates). The self-hosted option costs \$8,000+. At 100 million tokens per month, the API costs roughly \$19,000. The self-hosted option still costs \$8,000+, and now the math starts to flip — but only if you ignore the engineering overhead.
Multiple industry analysts have noted that commercial APIs often win at moderate scales because costs scale linearly, while open source models require step-function investments in infrastructure and talent.
The hidden cost no one talks about: token efficiency
Here is the open-source tax that most cost comparisons miss entirely.
A comprehensive study by Nous Research, published in 2025, examined 19 different AI models across three task categories: basic knowledge questions, mathematical problems, and logic puzzles. The findings were devastating for the "open source is cheaper" narrative.
Open-weight models use 1.5 to 4 times more tokens than closed models like OpenAI's for the same tasks. For simple knowledge questions, the gap widened dramatically — some open models used up to 10 times more tokens.
The researchers wrote: "While hosting open weight models may be cheaper, this cost advantage could be easily offset if they require more tokens to reason about a given problem".
This changes everything. A model that costs 1/10th as much per token but uses 4x as many tokens is only 2.5x cheaper in reality — and if it uses 10x the tokens, it's actually more expensive.
Large Reasoning Models are particularly inefficient. These models, designed to think through problems step-by-step, can consume thousands of tokens pondering simple questions. For basic knowledge questions like "What is the capital of Australia?" the study found reasoning models spending "hundreds of tokens pondering simple knowledge questions" that could be answered in a single word.
The Seoul fintech team got lucky — their document processing workload was structured enough that token efficiency wasn't a major factor. But for many teams, the open-source "savings" on per-token cost evaporate the moment you measure actual tokens consumed per completed task.
The breakeven point
Where is the actual breakeven? When does self-hosting open-source become cheaper than APIs?
The answer varies by workload, model size, and engineering cost. But a few data points help.
For a 70B model with moderate optimization, the breakeven typically sits between 50 million and 100 million tokens per month. Below that, APIs are cheaper. Above that, self-hosting can be cheaper — but only if you have the engineering team to maintain it and the workload is stable enough to justify the fixed infrastructure cost.
For smaller models — 7B to 14B parameters — the breakeven point is much higher because API pricing is already extremely low. GPT-4o-mini at \$0.15 per million input tokens is hard to beat with self-hosting unless you have massive scale.
For larger models — 70B and above — the breakeven point is lower because API prices are higher and self-hosting requires more GPUs. But the engineering overhead scales with model size, and token efficiency gaps become more pronounced.
The decision framework
Here are five questions to ask before you decide. Answer them honestly.
1. What is your actual monthly token volume? Not projected. Actual. If you're below 50 million tokens per month, the API is almost certainly cheaper. If you're above, do the full TCO calculation.
2. Do you have a dedicated infrastructure team? Self-hosting is not a "set it and forget it" proposition. You need someone who understands GPU memory management, request batching, model quantization, and inference optimization. If you don't have that person, you're about to hire one or burn engineering cycles you should be spending on your product.
3. How often do you upgrade models? Every upgrade is a migration. If you're iterating fast — trying new models, comparing performance, switching based on results — APIs give you that flexibility for free. Self-hosting locks you in. Each migration costs days of engineering time.
4. What is your workload's token efficiency? Run a test. Take 100 real production queries. Run them through both an open model and a closed model. Count the tokens. If the open model uses 2x, 3x, or 4x the tokens, factor that into your cost calculation. Many teams don't, and they pay for it.
5. What is the cost of being wrong? If you self-host and the math doesn't work, you're stuck with hardware commitments and a migration project. If you use APIs and the math doesn't work, you switch providers in an afternoon. The option value of APIs is real, and it has a dollar value.
The Seoul fintech team made the right call — for them.
They had compliance requirements that forced self-hosting. They had the engineering team to maintain it. Their workload was structured enough that token efficiency wasn't a major concern. They ran the numbers, and self-hosting came out ahead.
But that doesn't mean self-hosting is the right call for you.
The open-source tax is real. It shows up in GPU bills, in engineering salaries, in migration costs, in token inefficiency, in opportunity cost. "Free" models are rarely free. And the teams that understand this — that actually run the TCO calculation instead of assuming open source is cheaper — are the ones making better infrastructure decisions.
The next time someone says "let's use open source, it's free," ask them: have you run the numbers?
Data sources: Introl.com "GPU Cloud Prices Collapse" (December 2025); Introl.com "Spot Instances and Preemptible GPUs" (December 2025); NerdLevelTech "AI Costs 2026" (March 2026); VentureBeat "That 'cheap' open-source AI model is actually burning through your compute budget" (August 2025); Nous Research GitHub "Measuring Thinking Efficiency in Reasoning Models: The Missing Benchmark" (2025); Seoul fintech deployment case study (industry documentation, 2024–2025).