Most teams pick a language model the way they pick a laptop: whatever topped the latest review. Then the invoice arrives, or the p95 latency does, and the decision gets made again under pressure.
The better question is not "which model is best?" but "which model is cheapest at the quality bar this task actually needs?" That turns a debate about impressions into arithmetic. This guide shows how to price a task rather than a token, how to set a quality bar before testing, how routing changes the economics, and gives you a worksheet you can reuse for every model decision.
Why leaderboards are the wrong starting point
Leaderboards measure general capability on public tasks. Your product has specific tasks, a specific tolerance for errors and a specific budget. Three gaps follow:
- The tasks differ. A model that leads on competition maths may not lead on extracting fields from your invoices.
- The conditions differ. Leaderboard scores use their own prompts and settings, often with many attempts or long reasoning that you would not pay for in production.
- Cost and latency are missing. Most rankings ignore price, speed and consistency, which are exactly what constrain real products.
Leaderboards are useful for making a shortlist. The decision should come from your own evaluation; see how to evaluate a new AI model in two weeks.
Price the task, not the token
Token prices are a misleading unit. What you pay for is a completed task: the prompt, the response, any reasoning tokens, the retries when output is malformed, the tool calls, the validation pass, and the human who fixes the output when it is wrong.
def cost_per_task(price_in, price_out, tok_in, tok_out, retry_rate,
error_rate, fix_minutes, hourly_rate):
"""Prices per million tokens; rates as fractions (0.05 = 5%)."""
model = (tok_in * price_in + tok_out * price_out) / 1_000_000
model *= 1 + retry_rate # retries are not free
human = error_rate * fix_minutes / 60 * hourly_rate # someone fixes the misses
return model + humanWork through an illustrative example. Suppose a document-processing task takes 3,000 input tokens and 500 output tokens, and a human spends 6 minutes, at $50 an hour, fixing each wrong output.
| Item | Model A (cheap) | Model B (premium) |
|---|---|---|
| Price per 1M tokens (in / out) | $0.50 / $1.50 | $3 / $15 |
| Token cost per attempt | $0.00225 | $0.0165 |
| Retry rate | 8% | 2% |
| Error rate after validation | 10% | 4% |
| Human fix cost per task | $0.50 | $0.20 |
| **Total cost per task** | **≈ $0.50** | **≈ $0.22** |
The numbers are made up, but the pattern is common. When a human must repair errors, a few percentage points of quality outweigh large differences in token price. Conversely, when errors are cheap or caught automatically, the cheap model usually wins. You cannot know which situation you are in without measuring error rates on your own tasks.
Set the quality bar first
Write down what "good enough" means before testing. Without a bar, teams drift towards the most capable option every time, because the bigger model always looks a little better in side-by-side demos.
- Hard bar: non-negotiable checks. Output must parse, cite a source, stay under a length, never include personal data.
- Soft bar: a rubric score from 1 to 5 with a minimum average, graded blind.
- Latency bar: p95 time that users tolerate on that screen. A chat reply, a background job and an autocomplete suggestion have very different budgets.
- Consistency bar: maximum acceptable variation across repeated runs for the same input.
A model that clears every bar is good enough. Among the models that are good enough, choose on cost.
Include the hidden costs
Some costs do not appear in the per-token price:
- Reasoning tokens. Reasoning models can bill several times more output than the visible answer. See what reasoning tokens actually cost.
- Context size. Long prompts with large retrieved documents dominate input cost. Better retrieval often saves more than a cheaper model.
- Caching. Many providers discount repeated prompt prefixes. A model with good caching support can be much cheaper for workloads with long, stable system prompts.
- Rate limits and capacity. A cheaper model you cannot get enough throughput on is not cheaper.
- Switching cost. Prompts, parsers and evals tuned to one model take time to port. Factor it in, but do not let it lock you in forever.
Latency is part of the budget
Speed has a price too, even if it does not appear on the invoice. A slower model can cost you conversions, user patience or the throughput of a batch pipeline. Treat latency as a constraint with three parts:
- Time to first token decides whether an interactive feature feels responsive. Streaming hides some of the total time, but not this part.
- Tokens per second decides how long long answers take, and how many requests a fixed-capacity deployment can serve.
- Tail latency (p95, p99) decides how often users see something slow. A model with a good median and a long tail generates more complaints than its average suggests.
Measure all three under realistic load, with your real prompt lengths. Our guide to designing a latency benchmark explains how. When two models both clear the quality bar, the faster one is often worth a modest price premium on interactive screens and not worth it at all for overnight batch jobs.
Route instead of choosing once
Very few products need one model for everything. A small model can classify the request and handle the easy majority; a larger one gets the hard tail. Routing is usually the single biggest cost win available.
- Classify difficulty with rules, a small model or a confidence signal.
- Send easy requests to the cheap model, with validation.
- Escalate when the request is classified hard or the cheap model's output fails validation.
Research such as RouteLLM (Ong et al., 2024) has shown that learned routers can preserve most of a strong model's quality while sending a large share of traffic to a cheaper one. Our article on where small models quietly win covers practical routing patterns.
Log a week of real requests, label 100 as easy or hard, and check how often a small model handles the easy ones correctly. That single number tells you whether routing is worth building.
The worksheet
- Collect 50 to 100 real inputs with expected outputs or a rubric. If you do not have them, start with build an eval set in an afternoon.
- Define the hard, soft, latency and consistency bars.
- Shortlist three to five candidates from public results and pricing pages.
- Measure each candidate: pass rate, error types, retry rate, p50 and p95 latency, tokens per task.
- Compute cost per completed task, including human fix time.
- Pick the cheapest model that clears every bar, or a routed combination.
- Write the decision down with the numbers, and re-run the worksheet quarterly or when a significant new model ships.
The best model is the least expensive one that still clears your bar.
Frequently asked questions
How do I choose the right LLM for my application?
Define a quality bar for your task, test a shortlist of models on real examples, and compute cost per completed task including retries and human fixes. Choose the cheapest model, or routed combination, that clears every bar.
Is the most expensive model always the most cost-effective?
No. It is cost-effective when errors are expensive to fix and the premium model makes meaningfully fewer of them. When errors are cheap or caught automatically, a smaller model usually costs less per task.
What is LLM routing?
Routing sends each request to a different model depending on its difficulty or type, typically sending easy requests to a cheap model and escalating hard ones or failures to a more capable model.
How often should I revisit my model choice?
Quarterly is a sensible default, plus whenever a new model or price change could plausibly alter the decision. Keeping your eval set and worksheet ready makes each review fast.
