Number Research
← Back to Explore
Information Systems Technology Adoption and Management AI Infrastructure Cost Economics

The Real Local-vs-Cloud AI Math Nobody Runs Correctly

Hardware cost, not electricity, dwarfs the local-AI ledger — and cloud bills quietly climb 25x by message fifty

S.J. Nam 8 min read
The Real Local-vs-Cloud AI Math Nobody Runs Correctly

The Engineer Who Broke Samsung's Trust

In March 2023, an engineer at Samsung's semiconductor division pasted a piece of proprietary source code into ChatGPT to help debug it. Weeks later, Bloomberg reported that Samsung had discovered the leak and, within days,
banned the use of ChatGPT and other AI-powered chatbots by its employees amid concerns about sensitive internal information being leaked on such platforms
. The story became the go-to cautionary tale for the cloud-versus-local debate: send your data to someone else's server, and you no longer control where it goes.

That story is true, and it's also the least interesting part of this argument now. Two years on, open-weight models have gotten good enough that the real fight isn't about whether a chatbot might memorize your code. It's about arithmetic — specifically, an arithmetic almost nobody on either side is doing correctly.

The Three-Legged Stool That Isn't

The standard framing treats privacy, cost, and speed as three independent dials you trade off against each other: cloud is fast and cheap but risky, local is private and slow but pricier upfront. That framing is tidy, and it's wrong. Privacy, once you've decided sensitive material can't leave the building, isn't really a dial — it's a gate that either closes the cloud option entirely (as it now does across parts of healthcare, where
patient data and regulatory compliance demand that models run inside hospital infrastructure rather than in third-party cloud environments
) or doesn't matter much at all. Speed and cost, meanwhile, turn out to be governed by the same underlying variable: how many tokens you actually push through hardware you've already paid for, whether that hardware sits in a data center billing you per call or on your desk humming at 30 watts. Get that variable right, and the "which is better" question mostly answers itself. Get it wrong, and you'll cheerfully overpay no matter which side you're on.

The Myth of the Free Electron

Local-AI advocates love to cite electricity costs, and the numbers are genuinely startling. A Mac Mini M4 Pro running an 8B–22B parameter model draws roughly 30–40 watts against the
350–450W for an RTX 4090 rig
, and one detailed cost breakdown puts the
annual electricity cost at roughly $14 versus $160–210 for a comparable GPU build
. A cluster of five Mac Minis running flat out draws about 200 watts combined — less than a single gaming GPU under load.

But this is close to a rounding error, and treating it as the headline number is the mistake almost everyone makes. One rigorous teardown of an M5 Max MacBook running local inference makes the point bluntly:
The power bill is not why local inference is expensive. The real cost is the $4,299 sitting on your desk, and how few tokens you actually push through it
. The trap, as the same analysis notes, is imagining you use it eight hours a day when
actual token generation for a heavy individual user is closer to 1–2 hours of wall-clock generation per day
— the rest of the time you're reading, thinking, or in meetings, and an expensive machine sits mostly idle. Amortized against real usage, hardware cost dwarfs power cost by orders of magnitude. The honest local-AI pitch isn't "free electricity." It's "you already own a good computer, so your marginal cost really is a few cents per million tokens" — a much narrower and more conditional claim than the enthusiasts usually make.

There's a second wrinkle generic explainers skip: local speed is bottlenecked by memory bandwidth, not raw compute.
An RTX 4090 has 1,008 GB/s bandwidth. The M4 Max tops out at 546 GB/s
, and because
LLM inference is memory-bandwidth bound, not compute bound, this bandwidth gap directly affects tokens/second
. In practice, local inference on consumer hardware is fine for short conversational bursts — a Mac Mini Pro can push
20–30 tok/s on 8B–22B local LLMs
— but slows noticeably on the 70B-class models closest to frontier quality, where users report something closer to four or five tokens per second, a pace you can feel while reading along.

The Tax Nobody Sees Until Message Fifty

Cloud pricing looks simple until you notice what a multi-turn conversation does to a bill. Because each new message resends the entire prior context, cost per turn climbs as a conversation lengthens — one analysis calculates that
the 50th message in a conversation costs 25.5 times a standalone message's cost
, a dynamic the researcher behind it describes as a tax that
compounds automatically, silently, and invisibly
. Flat-rate subscriptions hide this from casual users, but the underlying compute is real:
ChatGPT is estimated to cost OpenAI on the order of $17 billion per year to operate globally
, well beyond subscription revenue, and one heavy Claude Max user reportedly
used ~10 billion tokens in 8 months
, usage that
at list API pricing would be ~$15,000
against a $200 monthly plan.

Provider-to-provider pricing also varies by an order of magnitude for reasons that have little to do with quality:
xAI's Grok 4.1 models charge only $0.20 per 1 million input tokens and $0.50 per 1 million output tokens, whereas OpenAI's flagship GPT-5.2 is priced at $1.75 per 1 M input and $14.00 per 1 M output tokens
, and Anthropic's
introductory pricing of $2/$10 per million input/output tokens is in effect through August 31, 2026, after which the standard pricing of $3/$15 per million input/output tokens will take effect
. None of that spread tracks cleanly with capability — it tracks subsidy strategy, and subsidy strategies change on a company's schedule, not yours.

Parity on Paper, Not in Practice

Here is the genuinely counterintuitive finding: open-weight models have already closed the knowledge gap with closed frontier models, but not the reliability gap — and those are not the same thing.
At the end of 2023, the best closed model scored around 88% on MMLU while the best open alternative managed roughly 70.5%, a gap of 17.5 percentage points. By early 2026, that gap is effectively zero on knowledge benchmarks, and single digits on most reasoning tasks
, a convergence the Stanford AI Index report confirmed across evaluation suites. Among specific models,
DeepSeek R1 dominates MATH-500 at 97.3%, which is near-perfect
.

But benchmark parity masks a real gap in dependability. A hands-on 2025 evaluation of open and closed models found
30% of models failed basic sentence counting, highlighting fundamental gaps
, and that
only 63% solved logic puzzles, with failures spanning all model sizes and families
— open and closed alike. And where a gap does persist by consensus, it persists precisely where reliability matters most for unattended use:
closed models maintain a lead on production coding (SWE-bench), overall human preference (Chatbot Arena), and complex agentic tasks
. That's a meaningful asterisk for anyone planning to hand a local model a long autonomous task rather than a single, well-scoped question.

Where the Line Actually Falls

None of this points toward a universal winner, and that's the actual finding. It points toward a threshold, and the threshold isn't privacy or cost or speed in isolation — it's how saturated your usage already is. For someone firing off a few dozen prompts a week, a subsidized $20-a-month subscription to a frontier model is an absurdly good deal, propped up by a company eating the marginal cost. For a hospital system bound by HIPAA and GDPR, the privacy gate closes the conversation before cost or speed even enter it. And for a small agency running thousands of queries a month, the math flips fast: one workload analysis found that a
$599–$999 investment typically breaks even in 6–12 months for agencies running 1,500–2,500+ monthly queries
compared with paying per API call.

The uncomfortable implication is that "local models are finally good enough" is less a story about model quality catching up and more a story about agentic workflows quietly multiplying everyone's token consumption until they cross a line most people didn't know existed. The Samsung engineer's mistake was pasting code into the wrong box. The mistake most people are about to make is never checking which side of that line their own usage has already landed on.

References