Number Research
← Back to Explore
Economics Economics of Information Systems AI Inference Cost Economics

The $340 Demo That Became a $61,000 Bill

Token prices fall while the real cost of AI hides in storage, chunk overlap, and vendor subsidies that could vanish overnight.

S.J. Nam 7 min read
The $340 Demo That Became a $61,000 Bill

A retrieval demo at a firm the analysts call NorthBank cost $340 a month. The production version of the same system cost $61,000 a month. Nobody who had watched this pattern before was surprised. The demo used a few hundred documents, a handful of test queries, and a fixed corpus; production had millions of documents, thousands of queries a day, a corpus that changed constantly, and users who found every edge case. The gap between those two numbers is not a rounding error or a bad negotiation. It is the whole story of where AI money actually goes — and almost none of it lives on the pricing page everyone stares at.

The Decoy on the Pricing Page

Ask a developer what an AI feature costs and they will quote you a token price, because that is the number the industry has trained everyone to watch. It is also the number designed to reassure you. GPT-4o launched in May 2024 at $5.00 input and $15.00 output per million tokens; in October 2024, OpenAI cut prices by 50% to $2.50 and $10.00. GPT-4o Mini arrived in July 2024 at $0.15 and $0.60 and has not moved since. Rewind further and the drop is vertiginous — early GPT-4 at 8K context billed $0.03 per thousand output tokens, or $30 per million. A model class got roughly three times cheaper in under two years while getting better.

So the sticker price falls, and the reassuring conclusion is that AI is racing toward free. But two structural facts hide inside that single number. The first is asymmetry: output tokens typically cost three to four times more than input tokens. A chatbot that reads a short question and writes a long, chatty answer is paying at the expensive end of the meter, and "reduce output verbosity" turns out to be a more powerful cost lever than switching models. The second is that the headline price is a ceiling almost no serious operator pays. OpenAI's Batch API processes requests asynchronously at 50% off, and prompt caching bills repeated prefixes at half rate — $1.25 instead of $2.50 per million for GPT-4o. The number on the page is real, and it is also theater. The people who obsess over it are optimizing the one line item that is both the smallest and the most negotiable.

The Bill That Never Appears

Here is the counterintuitive part. Embeddings — the numerical fingerprints that make search and retrieval work — are so cheap they are effectively free. Generating a billion embeddings with OpenAI's small model costs about $20, once. Text-embedding-3-small runs $0.02 per million tokens, or a penny per million through the Batch API. You could vectorize a small library for the price of lunch.

Then the meter you were never shown starts running. Storing those vectors in a managed vector database can cost $200 or more a month, forever; for a long-lived corpus, the recurring storage bill dwarfs the one-time generation cost within a few months. The physics are unforgiving: storage follows vectors times dimensions times four bytes, so a million 1,536-dimension vectors is about 5.7 GB. And the raw text you thought you were storing is not what you are billed for. Chunk overlap inflates token counts 10 to 25% over the raw text size — a 1 GB corpus that is 750 million raw tokens becomes 830 to 940 million billed tokens after overlapping chunking.

Managed vector databases turn this into a subscription you cannot easily cancel. Pinecone's bill is not one number but four. It charges write units, read units, storage at roughly $3.60 per GB per month, and capacity fees of $50 to $150 at sustained load. A ten-agent system on 10 million vectors runs $99 to $199 a month — and without compression enabled, storage alone hits $221. There is also a floor most calculators skip: a $50 monthly minimum applies on the Standard plan, so a month of $20 in actual usage still invoices at $50. The one-time cost you were warned to budget for is trivial. The recurring cost nobody mentioned is the real bill, and it accrues whether or not a single user ever runs a query.

That is how a $340 demo becomes a $61,000 production system. The tokens were never the problem. Re-embedding a corpus of 10 million 500-token documents at $0.13 per million costs about $650 — acceptable once. It stops being acceptable when the corpus changes constantly and you pay it again and again, on top of storage that scales with your data, retrieval that scales with your traffic, and orchestration that scales with your ambition.

The Price Is Not the Cost

Now follow the dollar all the way down, past your invoice, into the data center — because the strangest fact about the token price is that it does not reflect what the compute actually costs. It reflects a strategy. OpenAI posted a 33% gross margin in 2025, constrained by inference costs that reached $8.4 billion and are projected to hit $14.1 billion in 2026. Projected cash burn runs to roughly $27 billion in 2026 and $63 billion in 2027. When you pay $2.50 per million input tokens, part of that dollar is a real cost of serving you, and part of it is a bet — placed by investors — that the cost of serving you will collapse before the subsidy runs out.

There is good reason to believe it will, which cuts against the idea that the price reflects anything fixed. OpenAI cut ChatGPT guest traffic from tens of thousands of Nvidia GPUs to a couple hundred using software optimization alone — more than 50% savings from techniques like KV cache reuse, quantization, and smarter request routing. FP8 quantization on Nvidia's H100 has been shown to deliver 1.3 to 2 times higher throughput over FP16 at under 2% quality loss. The company sketches a path from a 39% gross margin toward a 52% target. Translate that: the underlying cost of a token can be halved overnight by an engineering change you will never see, while the price you pay does not move at all. The sticker price is decoupled from the compute in both directions — marked up by a margin, and marked down by a subsidy, until the two happen to meet.

What the Meter Actually Measures

The honest accounting, then, inverts the standard explainer. The token price you optimize is the cheapest, fastest-falling, most-discounted, and least-honest number in your stack — a decoy that is simultaneously too high, because it carries a margin, and too low, because it is propped up by billions in investor cash betting on efficiency gains that have not fully arrived. The dollars that determine whether your product survives contact with real users hide in the places no pricing page lists: the three-to-one output tax, the chunk overlap you didn't count, the vector storage that bills forever, the monthly minimums and capacity fees that switch on silently at scale.

Which raises the question worth sitting with. If the price of a token tells you almost nothing about what your system costs, and even less about what it costs to run — what exactly are you agreeing to when you build a business on a number set by a company losing tens of billions of dollars a year to keep it low?

References