Most LLM evaluation starts as a script: loop over some examples, call the model, check the outputs, print a score. That is the right way to start. Within a few months, though, the script has grown into something several people depend on, and nobody can say exactly which prompt, model version or dataset produced last month's numbers.
An eval harness is the infrastructure that runs evaluations reliably and reproducibly: it manages datasets, calls models, applies graders, stores results and reports comparisons. This checklist covers what a good harness needs, grouped by area, with notes on why each item matters. You do not need everything on day one, but knowing the list helps you avoid painting yourself into a corner.
Why a harness, not a script
Evaluation results drive decisions: shipping a prompt, switching a model, signing a contract. Those decisions are only as good as the numbers, and numbers from an unreproducible process cannot be trusted or audited. A harness provides three things a script usually lacks: reproducibility (you can re-create any result), comparability (two runs differ only in what you intended to change) and visibility (anyone can see what was tested and why it failed).
Datasets
- Version every dataset. Store eval sets in version control or a dataset store with immutable versions. A result without a dataset version is not reproducible.
- Record provenance for each example: where it came from, when, and who wrote the reference.
- Tag examples by category and difficulty, so results can be broken down.
- Keep a held-out split that is never used while tuning prompts.
- Handle sensitive data properly: remove or replace personal information, and control who can see the data.
If you are starting from scratch, see build an eval set in an afternoon.
Runs and reproducibility
- Pin the model version explicitly, not an alias that silently updates.
- Store the full configuration of every run: prompt templates, system prompt, temperature and other sampling settings, tool definitions, retrieval settings and code version.
- Save every raw output, not only scores. You will want to re-grade with an improved grader, and to read failures later.
- Make runs resumable and idempotent. API errors and rate limits will interrupt long runs; restarting should not duplicate or lose work.
- Cache model responses keyed by the full request, so re-grading or re-analysing does not re-pay for generation.
Graders
- Layer graders: deterministic code checks first, model graders for judgement, human review for calibration.
- Version graders like code, including grading prompts and grader model versions.
- Calibrate model graders against human labels and re-calibrate when anything changes; see model graders are useful, if you calibrate them first.
- Store grader reasoning alongside verdicts, so disagreements can be investigated.
Statistics and reporting
- Report confidence intervals and sample sizes with every score; see your 2-point improvement might be noise.
- Use paired comparisons when comparing two configurations on the same examples, and show the examples that changed outcome.
- Break down by category, so regressions in one area are not hidden by gains in another.
- Support repeated runs at non-zero temperature and aggregate per example correctly.
Cost, speed and operations
- Track cost and latency per run alongside quality. A configuration that is 1 point better and 3 times more expensive is a business decision, not an automatic win.
- Respect rate limits with concurrency controls and backoff, and keep a separate quota for evaluation so it does not compete with production.
Integration with development
- Run a fast smoke eval in CI on every change to prompts, tools or model configuration: a small, representative subset that finishes in minutes.
- Run the full eval before releases and model upgrades, and on a schedule to catch silent provider changes.
- Set gates carefully. Block merges on hard failures (format errors, safety cases, critical regressions), but treat small score movements as information, not automatic blockers, given the noise.
- Make results easy to browse. A simple web page listing failures with inputs, outputs and grader notes gets used; a JSON file in a bucket does not.
Signs your harness needs work
You may already have a harness that technically works. These symptoms suggest it is not doing its job:
- Nobody can reproduce last month's numbers. Usually a missing model pin, an unversioned dataset or an untracked prompt change.
- Every comparison starts with an argument about settings. If people debate whether two runs used the same temperature, the configuration is not being recorded.
- People re-run evals to look at outputs. Raw outputs should be stored and browsable; re-running costs money and may give different results.
- Score changes are celebrated or panicked over without intervals. The harness should show uncertainty by default.
- The eval only runs before big launches. If running it is painful, it will not run often enough to catch regressions. Make the smoke eval fast and automatic.
- Failures are invisible. If seeing which examples failed requires writing a query, people will look at the score and nothing else.
Each of these maps to items on the checklist above. Fixing the most painful one first usually does more for evaluation quality than any new metric.
What good looks like
A mature harness is mostly invisible. Engineers change a prompt, the CI smoke eval runs in a few minutes, and the pull request shows a short summary: score with interval, categories that changed, a link to the examples that flipped, and the cost of the run. Before a release, someone triggers the full suite and reads the failures. When a provider ships a new model, the comparison is a configuration change and an afternoon of reading outputs, not a project.
Tools to start from
You do not have to build everything. Several open-source projects cover parts of this checklist:
| Tool | Focus | Good for |
|---|---|---|
| EleutherAI lm-evaluation-harness | Academic benchmarks across many models | Running standard benchmarks reproducibly |
| Inspect (UK AI Security Institute) | Evals including agents, tools and sandboxes | Complex and agentic evaluations |
| promptfoo | Prompt and model comparisons, red-teaming | Product teams iterating on prompts |
| OpenAI Evals | Eval framework and registry | Teams in that ecosystem |
Many teams combine a framework for running and grading with their own lightweight storage and reporting. Choose based on what you evaluate: academic benchmarks, product prompts or agents. For agents, the requirements grow considerably; see evaluating agents end to end.
A staged plan
- Week 1: versioned dataset, pinned model, stored outputs and configs, a results table per run.
- Month 1: code and model graders with calibration, category breakdowns, confidence intervals, response caching.
- Quarter 1: CI smoke eval, scheduled full runs, cost and latency tracking, a browsable results page, automatic addition of production failures to the dataset.
Frequently asked questions
What is an LLM evaluation harness?
It is the software that runs evaluations of language model systems: it loads versioned datasets, calls models with recorded configurations, applies graders, stores outputs and scores, and reports comparisons between runs.
Which open-source tools can I use to evaluate LLMs?
Popular options include EleutherAI's lm-evaluation-harness for standard benchmarks, Inspect for complex and agentic evaluations, promptfoo for prompt comparisons and red-teaming, and OpenAI Evals.
Should LLM evals run in CI?
Yes, a small, fast smoke eval should run on changes to prompts, tools or model settings, with hard gates for critical failures. Run the full, slower eval before releases and on a schedule.
Why should I store raw model outputs from evals?
Raw outputs let you re-grade with improved graders without paying for generation again, investigate failures, compare runs example by example and audit past decisions.
