When to Use Per-Token Inference vs Renting a GPU by the Hour

A dedicated H100 sounds like the grown-up choice. Your own card, your own model, no shared queue. And at $1.99 an hour on RunPod, it doesn't even sound expensive.

It is, for almost everyone reading this. The card bills 24 hours a day and your users don't send requests 24 hours a day. The per-token API only charges when a token comes out. That asymmetry decides the question, and the Hugging Face alternatives post gave it one paragraph. It deserves the math.

How Much Does Each Option Cost?

Every number below was checked against the vendor's pricing page on September 19, 2026.

Two things jump out. The same 70B model costs ten times more per output token on Together than on DeepInfra, so "per-token" is not one price. And the cheapest dedicated H100 still costs about $53 a day whether you serve one request or a million.

Where Is the Break-Even?

Take DeepInfra, since it sells both models and the comparison is clean. A dedicated H100 at $2.20 an hour is $52.80 a day. Llama 3.3 70B output tokens cost $0.32 per million. Divide one by the other and the card pays for itself at 165 million output tokens a day.

That's 1,900 tokens a second. Every second. For 24 hours.

Whether one H100 can even produce that depends on your model, quantization and batching, and every vendor's throughput chart flatters its own hardware. Measure your own model with vLLM before you believe any of them. But the arithmetic alone gives the shape of the answer. A side project doing 50,000 requests a day at 500 output tokens each is 25 million tokens, about $8 on per-token. The card would cost $53 for the same day.

Run the same math on an 8B model at $0.04 per million and the break-even is 1.3 billion tokens a day. Small models never win on hourly. The serverless price is too low to beat.

When Does Hourly Win Anyway?

Cost is one axis. Four situations put you on a rented card before the break-even, and they have nothing to do with tokens.

Your model isn't on anyone's catalog. A LoRA fine-tune of a 70B, a custom vision head, an architecture that came out last week. Serverless vendors host what sells. If your model isn't on the list, there's no per-token price to compare against.

Your customers ask where the data lives. A shared multi-tenant API is a hard sell to a European healthcare buyer. A dedicated card in a named region, under a contract you can show them, is a different conversation.

You need latency you control. Per-token APIs put you behind a shared queue. Most of the time that's fine. If your product is a voice agent or a live coding assistant, p99 matters more than the bill.

You want a fixed number on the invoice. Some founders would rather pay $1,584 a month, flat, than explain a usage spike to an accountant. That's a legitimate reason, as long as you know you're paying for predictability, not compute.

What About the Middle Ground?

Modal sits between the two. You bring your own code and weights, like a rented GPU. But it bills per second at $0.001097 for an H100, scales to zero between requests, and boots a container in about a second. The Modal vs Replicate vs Baseten comparison worked through a real example: 1,600 seconds of actual GPU work a day came to roughly $53 a month, and the same work with a 20 minute scaledown window came to about $1,000. One config value, twenty times the bill.

So Modal covers the custom-model case without the 24 hour meter. It doesn't cover data residency or fixed billing, and at $3.95 an hour of active time against DeepInfra's $2.20 dedicated, it loses once you're steady. Right answer for bursty custom models, wrong one for anything running flat out.

Which Should You Pick?

Per-token by default. If your model is on the DeepInfra or Together catalog and your traffic is spiky, you'll spend a fraction of the hourly rate and never think about capacity. Route through a gateway if you want the option to switch vendors later, which the OpenRouter alternatives post covers.

Modal when the model is yours and the traffic is bursty. Hourly, on DeepInfra or RunPod, when the traffic is steady enough that the card is busy most of the day, or when residency, latency or a contract forces your hand. Do the division with your own token counts before you commit, because the vendors won't do it for you.

Found a better option? Let me know on Twitter @devtoolpicks.