A new frontier model ships, the launch post shows it topping a handful of benchmarks, and within hours someone on your team asks whether you should switch. The honest answer is almost always "we don't know yet." Leaderboard numbers are measured on someone else's tasks, with someone else's prompts, often under conditions you will never reproduce.
This guide is a two-week framework for reviewing a new AI model against your own workload. It is designed to be run by one or two people alongside their normal work, and to end in a decision you can defend: switch, route some traffic, or stay put.
What a model review should answer
Before you open a playground, write down the questions the review must answer. A useful review answers five:
- Quality: does it pass more of *our* tasks than the current model, at our quality bar?
- Cost: what is the cost per completed task, including retries and human fixes?
- Latency: is time-to-first-token and total response time acceptable for each surface?
- Failure modes: when it fails, how does it fail, and is that worse or better than today?
- Operational fit: rate limits, context window, tool calling, structured output and data-handling terms.
Notice that "is it smarter?" is not on the list. Capability matters only as far as it moves one of these five. Our article on treating model choice as a budget decision explains the cost arithmetic in detail.
Week one: build the test bed and run the baseline
Days 1–2: assemble a representative task set
Pull 50 to 150 real inputs from production logs or support tickets, covering your main use cases in proportion to traffic. Add a deliberate slice of hard cases: long inputs, ambiguous requests, adversarial users, edge-case formats. For each item, record what a good output looks like, either a reference answer or a rubric.
If you do not yet have an eval set, follow build an eval set in an afternoon; it is the single highest-leverage asset for every future model decision.
Day 3: read the model card critically
Read the model card and pricing page, but read them as claims to verify. Note the context window, output limits, knowledge cutoff, supported modalities, tool-calling format and any stated weaknesses. Our guide to reading a model card critically lists the questions to ask.
Days 4–5: run your current model as the baseline
Run your existing production prompt and model over the whole set and score it. This is the number to beat. Without a fresh baseline, you will compare a new model in a clean test against your memory of the old one in production, which flatters the newcomer.
Week two: test the candidate fairly
Days 6–7: port the prompts, then tune lightly
Run the candidate with your production prompts unchanged first. Then allow a fixed, small tuning budget, for example two hours of prompt changes, applied to both models. Many "upgrades" disappear when the incumbent gets the same attention as the newcomer.
Day 8: measure quality
Score both models blind with the same rubric. Use several runs per item if your tasks are open-ended, and calculate confidence intervals before reading anything into small differences.
| Metric | How to measure | Why it matters |
|---|---|---|
| Pass rate | Rubric or exact match on the task set | Core quality signal |
| Hard-case pass rate | Same, on the hard slice only | Upgrades usually show up here first |
| Format validity | Share of outputs that parse or validate | Broken formats cost retries |
| Cost per task | Tokens × price × (1 + retry rate) + fix time | The number finance will ask about |
| p50 / p95 latency | Time to first token and total time | What users actually feel |
| Refusal rate | Share of legitimate requests declined | Over-refusal silently drops tasks |
Day 9: measure latency and cost under realistic load
Run a few hundred requests at your real concurrency and input lengths. Record time to first token, tokens per second and total time at p50 and p95. Launch-week infrastructure is often congested, so repeat the test a few days later before drawing conclusions. The method is explained in designing a latency benchmark.
Day 10: study the failures
Read every failed item for both models. Classify each failure: wrong fact, ignored instruction, bad format, refusal, hallucinated tool call, unsafe output. Two models with the same pass rate can fail in very different ways, and one failure type may be cheap to catch while another is invisible until a customer complains.
Common traps in model reviews
- Vibes from the playground. Ten impressive chats prove nothing. Write the impressions down, then check them against the task set.
- Testing only the happy path. New models often improve on easy items where your current model already succeeds.
- Ignoring verbosity. Longer answers cost more and score better with human and model judges. Normalise for length or instruct both models to the same length.
- Changing two things at once. If you switch model and rewrite the prompt in one go, you cannot tell which change helped.
- Forgetting the tail. p95 latency and the worst 5% of outputs drive user complaints, not the median.
Who should run the review
A model review works best with two roles. The operator builds the task set, runs both models and keeps the scripts reproducible. The reviewer grades outputs blind and owns the failure analysis. One person can do both, but separating them reduces the chance that enthusiasm for the new model leaks into the grades.
Involve the people who live with the outputs. Support agents, analysts or engineers who use the current model daily know which failures are tolerable and which ones generate tickets. Ask two or three of them to grade a sample of outputs from both models without knowing which is which; their preferences are often more predictive of user satisfaction than any automated metric.
Finally, set a timebox and stick to it. Model reviews expand to fill available time, and a good-enough decision in two weeks beats a perfect one after the next release has made it obsolete.
Making the decision
At the end of week two, you should be able to fill in a one-page decision record.
Routing is underrated. Very few products need one model for everything, and a router that sends hard requests to the stronger model and routine ones to a cheaper model often beats any single choice. See where small models quietly win for patterns.
Whatever you decide, keep the task set and scripts. The next model will ship in a few weeks, and the second review takes a day instead of two weeks.
Frequently asked questions
How many test cases do I need to evaluate an AI model?
For most product decisions, 50 to 150 representative cases are enough to detect large differences. Small differences of a few percentage points need hundreds of items or repeated runs, plus a confidence interval, before they can be trusted.
Should I trust public benchmarks when choosing a model?
Use them to build a shortlist, not to make the decision. Public benchmarks can be contaminated, saturated or unrelated to your tasks, and they rarely reflect your prompts, formats or latency needs.
How often should I re-evaluate models?
Re-run your eval whenever a candidate model is released that could plausibly change your decision, and at least quarterly for your production model, since providers update models and infrastructure without always announcing behavioural changes.
What is the most common mistake in model evaluations?
Comparing a carefully prompted new model against an untuned or remembered baseline. Always re-run the incumbent on the same set, with the same tuning budget, at the same time.
