Nobody buys a $40,000 GPU and then asks how many hours a day it’s actually computing. They ask how many tokens per second it can do at peak, divide the purchase price by a made-up annual token volume, compare that to an API price sheet, and put the winning number on a slide.
The number on that slide is wrong by roughly an order of magnitude. Not because the arithmetic is bad — because it’s missing a term.
The term everyone omits
Per-token pricing and per-hour ownership are different units. An API bills you for work performed. A GPU bills you for time elapsed, whether work happens or not. To compare them you need a conversion factor, and that factor is utilization.
Utilization isn’t one number, it’s two multiplied together:
Duty cycle — what fraction of wall-clock hours the box has any inference in flight. A GPU serving an internal dev team runs during business hours in one timezone. Call it 6 hours a day, five days a week. That’s 30 hours out of 168. Duty cycle: 18%.
Saturation — during those busy hours, what fraction of peak throughput you actually sustain. This is where the damage is. An H100 running a 70B-class model with continuous batching wants 25 to 40 concurrent requests to hit its throughput ceiling. Interactive users generate three or four. You’re running at maybe 20% of what the silicon can do, while paying for 100% of it.
Multiply them. 18% × 20% = 3.6% effective utilization. That’s the conversion factor between your capex and your token bill, and virtually no procurement model contains it.
I’ve measured this before on much smaller hardware — a Mac mini doing local inference for a real workload sat busy 1.7% of the time over three weeks. The absolute number scales with the box. The shape doesn’t.
Running the math both ways
Take a single-GPU node, honestly costed. $40K for the card, three-year straight-line depreciation: $13,333/year. Power at a 1.4 kW node draw and $0.15/kWh commercial rate: about $1,840/year running flat out. Rack and connectivity in a colo: $2,400. Ten percent of one platform engineer to own the thing — patching, driver hell, model upgrades, on-call: $18,000 at a fully loaded $180K.
Annual owned cost: ~$35,500. The staffing line is the biggest single item and it’s the one most spreadsheets leave blank. The operational bleed around open-weight deployment is not the GPU. It’s everything attached to it.
Now throughput. Assume that node sustains 2,000 output tokens/second aggregate under healthy batching. At 100% utilization that’s 63 billion tokens a year.
| Effective utilization | Tokens/year | Cost per M tokens |
|---|---|---|
| 100% | 63B | $0.56 |
| 25% | 15.8B | $2.25 |
| 10% | 6.3B | $5.63 |
| 3.6% | 2.3B | $15.65 |
At full tilt the box is spectacular — half a dollar per million tokens beats every commercial endpoint on the market. At the utilization your team will actually produce, it costs more than Claude Sonnet. Same hardware. Same model. Same electricity bill. The only variable that moved was the one nobody modeled.
Put a percentage on the slide, not a payback date
Here’s the reframe I’d argue for, and it fits in one line of a deck.
Stop computing break-even as a token volume. Compute it as a break-even utilization — the fraction of peak throughput you must sustain for the owned box to beat the API you’d otherwise call.
U* = annual_owned_cost / (api_price_per_Mtok × peak_Mtok_per_year)For the node above, against a $3/M blended API price: $35,500 / (3 × 63,000) = 18.8%. You need to hold nearly a fifth of peak throughput, all year, to win.
Against a $0.60/M hosted open-weights endpoint, the same node needs 94% utilization. That’s not a hard target. That’s a physically unreachable one for anything but a batch queue.
U* is better than a payback date for three reasons that matter to the person approving the purchase: it’s a single number, it’s measurable before you buy, and it’s falsifiable. “We break even in 14 months” can’t be checked until month 14. “We need 19% sustained utilization” can be checked next Tuesday against a week of Prometheus data.
If you’re already serving inference somewhere, you can measure both terms today:
# duty cycle — fraction of the last 7d with inference in flight
avg_over_time((DCGM_FI_DEV_GPU_UTIL > bool 5)[7d:1m])
# saturation — how full the batch actually runs (vLLM)
avg_over_time(vllm:num_requests_running[7d]) / <max_num_seqs>Multiply. That’s your real conversion factor. Divide your annual owned cost by it. Then compare.
The denominator keeps moving
There’s a second problem, and it’s worse than the first. U* isn’t a constant. It rises over the life of the asset, because the API price in the denominator keeps falling.
Epoch AI’s analysis of inference pricing found that the cost of reaching a fixed capability level has been dropping between 9x and 900x per year depending on the task. Take the conservative end. A 9x annual decline means the $3/M endpoint you benchmarked against is a $0.33/M endpoint a year later.
Your break-even utilization went from 19% to 171%. In year one. Your depreciation schedule did not move.
That’s the structural problem with capitalizing inference hardware: you’re making a three-year fixed commitment against a price curve that resets quarterly. The academic cost-benefit work on on-prem deployment generally models break-even against today’s commercial prices. Fine as a snapshot. Dangerous as a purchase justification, because it treats the comparison baseline as static when it’s the fastest-moving number in the stack.
What I’d actually do
Measure before you buy. One week of duty-cycle and saturation data from a rented instance running your real traffic costs a few hundred dollars and answers the question the spreadsheet is guessing at. Rent the exact GPU for a month first. Every cloud has them by the hour.
If your measured U is below 40%, don’t buy the box for cost reasons. There aren’t any. Route the interactive traffic to an API and keep looking.
If you have batch work — nightly document extraction, embedding regeneration, eval suites, synthetic data generation — that’s the workload that fills the trough. Owned hardware pays off when you can schedule against it, because scheduled work is the only kind that saturates. A queue that runs 2am to 6am at full batch depth does more for U* than doubling your user count.
And if the real driver is data residency, a contractual requirement, or a latency floor you can’t hit over the public internet — buy it. Those are good reasons. Just price the utilization gap and call it what it is: a control premium, with a dollar figure next to it. That’s a defensible line item. “We’ll save money on tokens” is not, and it falls apart the first time someone opens Grafana.
The finance team will accept a premium they can see. What they won’t forgive is a payback model that assumed 63 billion tokens and delivered 2.3 billion.