Every serious model release comes with a model card or technical report: a document describing what the model is, how it was trained, how it performs and where it falls short. Model cards are among the most useful sources of information about a model. They are also written by the people who want you to use it.
Reading one well means separating facts you can rely on from numbers that depend on choices you cannot see. This guide explains what a model card is for, what each section should tell you, the most common ways results are presented favourably, and a checklist you can use on any release.
What a model card is supposed to do
The idea comes from Mitchell et al. (2019), "Model Cards for Model Reporting," which proposed short documents accompanying trained models, describing intended uses, evaluation results across relevant conditions and groups, and known limitations. The goal was transparency: helping people decide whether a model fits their use case, and warning them where it does not.
Modern cards for large language models vary widely. Some are detailed technical reports with dozens of pages of evaluations. Others are a short page with a benchmark table. Either way, the same questions apply.
Section by section: what to look for
Model details
Look for: parameter count (or an honest statement that it is not disclosed), architecture type (dense or mixture of experts), context window, output limits, supported modalities, knowledge cutoff date and release date. Missing basics are not necessarily a problem, but they limit what you can infer.
Intended use and out-of-scope use
This section tells you what the developer designed and tested for. If your use case is listed as out of scope, such as medical advice or automated decisions about people, take it seriously: it usually means the developer did not evaluate it, or evaluated it and did not like the results.
Training data
Most frontier developers disclose little about training data. Look for at least: the general sources, the cutoff date, any filtering for quality or safety, and whether benchmark decontamination was performed. Without decontamination information, treat scores on older public benchmarks with extra caution; see how to check for benchmark contamination.
Evaluation results
This is the section most readers jump to, and the one that needs the most scrutiny. The next section covers it in detail.
Limitations and risks
A good card names specific weaknesses: languages that perform worse, task types with high error rates, known failure modes. A limitations section made only of generic disclaimers ("the model may produce inaccurate information") tells you nothing.
Safety evaluations
Look for which risks were tested (misuse, harmful content, privacy, security), how (internal red-teaming, external testers, automated evaluations), and what mitigations were applied. Note whether results are reported as numbers or only described.
Reading benchmark tables without being misled
Benchmark tables can be accurate in every number and still give a misleading impression. These are the details to check.
| Detail | Why it matters |
|---|---|
| Number of attempts (pass@1 vs. best of many) | Best-of-k scores can be far higher than single-attempt performance |
| Few-shot vs. zero-shot | Examples in the prompt can raise scores substantially |
| Reasoning or thinking enabled | Extended reasoning raises scores and cost; check which setting was used |
| Prompting harness | Custom prompts or scaffolding can differ from what you get via the API |
| Which comparison models and versions | Older or weaker versions of competitors flatter the new model |
| Benchmark subset | Reporting a subset (e.g. "verified" or "hard") changes difficulty |
| Error bars or number of runs | Without them, small differences may be noise |
Small differences deserve particular scepticism. On a benchmark of a few hundred questions, a difference of one or two percentage points is often within the range of random variation; see why your 2-point improvement might be noise.
Also check the benchmark itself. Scores near the top of an old benchmark say little, because the test no longer separates strong models; see what should replace a saturated benchmark.
Common ways results are presented favourably
None of these are necessarily dishonest, but each one shifts the impression:
- Cherry-picked benchmarks: showing the benchmarks where the model leads and omitting the rest.
- Mismatched settings: the new model with reasoning or many samples, competitors without.
- Self-reported competitor numbers: taken from different papers with different setups.
- Charts with truncated axes that make small gaps look large.
- Internal benchmarks that cannot be reproduced or inspected.
- "Up to" claims for speed or cost that apply only in ideal conditions.
When a claim matters for your decision, look for independent evaluations, or better, test it yourself on your own tasks; see how to evaluate a new AI model in two weeks.
The model card checklist
- Do I know the context window, output limit, knowledge cutoff and modalities?
- Is my use case inside the stated intended use?
- Is benchmark decontamination described?
- For each benchmark I care about: how many attempts, how many examples in the prompt, was reasoning enabled, which harness?
- Are competitor results measured under the same conditions, on current versions?
- Are there error bars, or at least the number of runs?
- Are limitations specific to tasks, languages or domains?
- Are safety evaluations described with methods and results?
- Is there independent evidence for the claims that matter most to me?
- What is missing, and does it matter for my use case?
Reading system cards and technical reports
Large releases increasingly come with several documents: a short model card, a longer technical report and sometimes a separate system card focused on safety testing. They serve different readers.
- The technical report explains architecture, training and evaluation methods. It is where you find the settings behind benchmark numbers, often in appendices that are worth the extra scrolling.
- The system card describes risk assessments, red-teaming, and the safeguards applied before release. If you deploy in a sensitive domain, it tells you what the developer tested and what they decided was acceptable.
- Release notes and changelogs for API models record behaviour changes between versions. They are short, easy to miss and often more relevant to production systems than the original launch materials.
Read the documents together. Claims in the launch post should be traceable to evidence in the report; if they are not, treat them as marketing.
Turning the card into a test plan
The most useful output of reading a model card is not a verdict; it is a list of things to verify. Each claim that matters to you, such as "better at long documents," "improved tool use" or "fewer refusals," becomes a test on your own data. Each limitation that touches your use case becomes a risk to check before launch.
Keep notes on each card you read. Over time you will learn which developers' claims tend to hold up in your tests, which is far more valuable than any single benchmark table.
Frequently asked questions
What is an AI model card?
A model card is a document published with a machine learning model that describes its intended uses, training approach, evaluation results, limitations and risks. The format was proposed by Mitchell et al. in 2019 to improve transparency.
Can I trust the benchmark scores in a model card?
The numbers are usually accurate for the conditions used, but those conditions, such as number of attempts, prompting and reasoning settings, may differ from how you will use the model. Check the settings and confirm important claims on your own tasks.
What does pass@1 mean in a model card?
pass@1 is the share of problems the model solves on its first attempt. Scores like pass@k measure whether at least one of k attempts succeeds, which can be much higher than single-attempt performance.
What should I do if a model card does not mention my use case?
Treat it as untested. Build a small evaluation with real examples from your use case and test the model directly before relying on it.
