Model graders are useful, if you calibrate them first

Grading the outputs of a language model is slow and expensive when people do it. Using another model as the grader, known as LLM-as-a-judge, makes evaluation fast and cheap enough to run on every change. It has become the default way to evaluate open-ended outputs such as answers, summaries and conversations.

It also introduces a new source of error. A model grader has its own biases, can be fooled by style, and can drift when you change its prompt or model version. Used without checks, it produces precise-looking numbers that measure the wrong thing. This guide explains the known biases, how to write a grading prompt, how to measure agreement with human judgement, and a calibration workflow that keeps the grader trustworthy.

What the research found

Zheng et al. (2023), "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," is the reference study. They found that strong models used as judges agreed with human preferences at a rate comparable to the agreement between different human annotators, which is why the method took off. They also documented systematic biases:

• Position bias: in pairwise comparisons, judges favour the answer in a particular position, often the first.
• Verbosity bias: judges tend to prefer longer, more detailed answers, even when the extra length adds nothing.
• Self-enhancement bias: judges may favour answers written by themselves or by similar models.
• Limited reasoning on hard questions: judges can be misled on maths and logic when they do not solve the problem themselves.

None of these make model graders unusable. They make calibration mandatory.

Choose the grading format

For product decisions, pass/fail against specific criteria is usually the most reliable. Pairwise comparison is useful for choosing between two prompts or models, as long as you control position bias.

Write the grading prompt like a rubric for a new hire

A good grading prompt is specific enough that two different people would apply it the same way.

Key practices:

• Grade one criterion at a time rather than "overall quality."
• Require evidence: quoting the relevant text makes judgements checkable and tends to improve accuracy.
• Give the grader the same source material the model had, so it can check grounding.
• For pairwise grading, run both orders (A then B, and B then A) and count only consistent verdicts.
• Control length: tell the judge not to reward length, or normalise answer lengths before comparing.

Calibrate against human labels

Calibration means measuring how often the grader agrees with careful human judgement, and improving it until agreement is acceptable.

1. Sample 100 to 200 outputs that cover passes, failures and borderline cases.
2. Have humans label them with the same rubric, independently. Ideally two people label each item; disagreements between them show where the rubric is unclear.
3. Run the model grader on the same items.
4. Measure agreement: raw agreement rate, plus Cohen's kappa, which corrects for agreement expected by chance. Also check the confusion matrix: false passes are usually more dangerous than false fails.
5. Read every disagreement. Is the grader wrong, the human wrong, or the rubric ambiguous? Fix the rubric or prompt and repeat.

A reasonable target is grader-human agreement close to human-human agreement on the same items. If two humans only agree 80% of the time, expecting 95% from a model is unrealistic, and the problem is the rubric.

from sklearn.metrics import cohen_kappa_score, confusion_matrix

human = [1, 0, 1, 1, 0, 1, 0, 1]   # 1 = pass
judge = [1, 0, 1, 0, 0, 1, 1, 1]

agreement = sum(h == j for h, j in zip(human, judge)) / len(human)
print("agreement", agreement)
print("kappa", round(cohen_kappa_score(human, judge), 2))
print(confusion_matrix(human, judge))   # rows: human, cols: judge

Keep the grader honest over time

• Pin the grader model version and the grading prompt. Changing either means re-calibrating.
• Keep a calibration set of human-labelled items and re-run it whenever anything changes.
• Spot-check weekly: have a human review a small random sample of grader decisions, especially passes.
• Watch for score inflation: if scores rise sharply after a prompt change in the system being tested, check whether outputs improved or just got longer or more confident.
• Avoid self-grading for comparisons: if you compare models from different families, use a grader from a third family or validate with humans.

Self-critique within the system being evaluated has related pitfalls; see the red-team yourself prompt.

Cost and speed trade-offs

Model grading is cheap compared with human review, but at scale it is not free. A few choices keep it affordable without losing reliability:

• Use code checks first and send only outputs that pass them to the model grader. There is no point paying a judge to evaluate an answer that fails to parse.
• Grade criteria together when they share context, in one call with structured output, rather than one call per criterion, once you have confirmed this does not reduce agreement.
• Use a smaller judge for simple criteria such as "does the answer mention a refund timeline," and reserve the strongest judge for nuanced ones such as factual correctness. Calibrate each judge separately.
• Cache grades for outputs that have not changed between runs, which is common when only part of a system was modified.

Track the grader's cost per eval run alongside the system's cost. If grading costs more than generation, it is usually a sign that the grading prompt or criteria can be simplified.

How model graders fit into an eval

Model graders are one layer in a stack. Use code checks for everything objective (formats, exact values, required phrases), model graders for judgement calls, and humans to calibrate and review. Report grader-based scores with confidence intervals; see why your 2-point improvement might be noise. For setting up the eval set itself, start with build an eval set in an afternoon, and for the infrastructure around it see the eval harness checklist.

Frequently asked questions

What is LLM-as-a-judge?

LLM-as-a-judge is the practice of using a language model to evaluate the outputs of another model or system, typically by scoring them against a rubric or choosing the better of two outputs.

How accurate are LLM judges?

Strong models can agree with human judgements about as often as humans agree with each other on many tasks, but they have known biases, such as favouring longer answers or a particular position, and they need calibration against human labels for each new task.

How do I reduce position bias in pairwise LLM grading?

Run each comparison twice with the order swapped and keep only verdicts that are consistent across both orders; treat inconsistent results as ties.

Should the judge model be the same as the model being evaluated?

Preferably not. Models can favour outputs similar to their own. Use a different model family as the judge when comparing models, or validate results against human labels.