An evaluation set, or eval set, is a collection of example inputs with a way to judge whether a model's outputs are good. It is the single most valuable asset for anyone building with language models. It tells you whether a prompt change helped, whether a new model is worth switching to, and whether last week's fix broke something else.
Most teams put off building one because it sounds like a research project. It is not. A useful first eval set takes about an afternoon. This guide gives you a four-hour plan: where to find examples, how to define "good," how to grade automatically, and the traps to avoid.
What you need before you start
- A clear description of one task your product performs, for example "answer customer questions about billing using our help centre."
- Access to real inputs: logs, support tickets, emails, search queries or user research notes.
- A spreadsheet or a simple file format such as JSONL.
- Someone who knows what a good answer looks like, ideally the person who would handle the task without AI.
Do not try to evaluate everything at once. One task, done well, is far more useful than a sprawling set that covers everything superficially.
Hour 1: collect 50 real examples
Pull real inputs, not invented ones. Invented examples tend to be clean, typical and easy; real ones are messy, ambiguous and occasionally strange, which is exactly what you need to test.
Aim for a mix that roughly reflects real traffic, plus deliberate coverage of difficult cases:
- Common cases (about 60%): the requests you see every day.
- Hard cases (about 25%): ambiguous wording, long inputs, multiple questions in one, missing information, typos, other languages.
- Should-refuse or escalate cases (about 15%): out-of-scope requests, requests for things the system must not do, inputs where the right answer is "I don't know."
Remove personal data or replace it with realistic fakes. Record each example with an ID, the input and a category tag.
{"id": "bill-001", "category": "common", "input": "Why was I charged twice this month?", "reference": "Explain duplicate pending authorisations; real double charges are refunded in 5–7 days; offer to check the account.", "must_include": ["pending", "refund"], "must_not": ["guaranteed"]}
{"id": "bill-031", "category": "hard", "input": "cancel but keep my data and also why is VAT different now", "reference": "Address both: cancellation keeps data for 30 days; VAT changed due to new billing address rules.", "must_include": ["30 days"], "must_not": []}Hour 2: define what "good" means
For each example, write down what a correct output must contain. The format depends on the task:
- Exact answers (a category, a number, a date): store the correct value.
- Structured outputs: store the expected object, or rules each field must satisfy.
- Free text: write a short reference answer plus a few checkable criteria, such as facts that must appear, claims that must not, length limits and tone.
Then write a rubric for open-ended quality, in plain language with three to five levels. Keep it specific to your task.
| Score | Meaning |
|---|---|
| 3 | Correct, complete, follows policy, appropriate length and tone |
| 2 | Correct but incomplete or with minor issues (too long, missing a helpful detail) |
| 1 | Partly wrong, misleading, or misses the main question |
| 0 | Wrong, harmful, invents policy, or should have escalated and did not |
Binary pass/fail is often better than a scale for decisions. If you use a scale, decide in advance what counts as a pass (for example, a score of 3, or 2 or higher).
Hour 3: choose graders and run a baseline
Use the cheapest grader that is reliable for each criterion:
- Code checks first. Exact match, schema validation, required phrases, forbidden phrases, length limits. Fast, free and deterministic.
- Model graders for judgement. Use a strong model with your rubric for criteria code cannot check, such as correctness of explanation or tone. Calibrate it against human judgement before trusting it; see model graders are useful, if you calibrate them first.
- Humans for the final word on a sample, and always for new or high-stakes tasks.
Write a short script that runs your current prompt and model over every example, applies the graders, and saves the outputs and scores to a file with a timestamp. Run it. That first result is your baseline.
import json, datetime
def run(model, prompt, path="eval.jsonl"):
rows = [json.loads(l) for l in open(path)]
results = []
for r in rows:
out = model.complete(prompt.format(input=r["input"]), temperature=0)
ok = all(s.lower() in out.lower() for s in r["must_include"]) and \
not any(s.lower() in out.lower() for s in r["must_not"])
results.append({"id": r["id"], "category": r["category"], "output": out, "pass_rules": ok})
stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M")
json.dump(results, open(f"results-{stamp}.json", "w"), indent=1)
return sum(x["pass_rules"] for x in results) / len(results)Hour 4: read the outputs
Do not stop at the number. Read every failure, and a sample of passes. You will find:
- grader errors: cases where the check was too strict or too loose; fix the check;
- bad references: cases where your expected answer was wrong or ambiguous; fix the reference;
- real model failures: the reason you built the set; tag them by type.
This hour is often the most valuable of the afternoon. It turns a vague sense that "the bot is sometimes wrong" into a concrete list of failure types.
Common traps
- Only easy examples. If your baseline scores near 100%, the set cannot detect improvements. Add harder cases.
- Testing on the examples you tuned with. Keep a portion of the set aside that you never look at while editing prompts.
- Too few examples to compare. Fifty examples detect big differences, not small ones; see why your 2-point improvement might be noise.
- One number for everything. Report results by category; improvements on common cases can hide regressions on hard ones.
- A set that never changes. Add every production failure as a new example. Retire examples that no longer reflect real use.
Who should own the eval set
An eval set decays without an owner. Make one person responsible for keeping it current: adding new failure cases, retiring outdated examples, and checking that references still match current policies and products. Ownership is a small, ongoing job, perhaps an hour a week, and it is what turns a one-off exercise into a reliable decision tool.
Involve domain experts in writing and reviewing references. The support lead knows which answers generate follow-up tickets; the compliance officer knows which phrasing is risky. Their judgement, captured in references and rubrics, is what makes the eval set reflect real quality rather than the engineer's guess.
What to do next
Once the set exists, use it every time something changes: prompts, models, retrieval settings, tools. Store results so you can compare runs over time. When the set grows beyond a few hundred examples, or several people run it, invest in a proper harness; the eval harness checklist lists what to build. And when a new model launches, your eval set turns a two-week debate into a one-day test; see how to evaluate a new AI model in two weeks.
Frequently asked questions
What is an LLM eval set?
An LLM eval set is a collection of example inputs with reference answers or grading criteria, used to measure how well a model and prompt perform on a specific task and to compare changes over time.
How many examples does an LLM eval set need?
Start with 50 to 100 real examples for one task. That is enough to catch large problems and compare prompts roughly. Grow to several hundred as you need to detect smaller differences.
Should I use synthetic examples in my eval set?
Prefer real inputs. Synthetic examples can fill gaps for rare cases, but they tend to be cleaner and more typical than real data, so they often overestimate quality.
Can I use an LLM to grade my eval set?
Yes, for criteria that code cannot check, but calibrate the model grader against human labels first and spot-check its decisions regularly. Use simple code checks wherever possible.
