Your 2-point improvement might be noise: confidence intervals for LLM evals

YYou change a prompt, re-run your eval set, and the score goes from 81% to 83%. Is the new prompt better? Very often, nobody can say, because nobody computed how much the score would move by chance alone.

Evaluation scores are estimates. They come from a finite sample of questions and, if you sample at non-zero temperature, from random model outputs. Both sources of randomness mean the score would differ if you ran a different sample. This guide explains how big that variation is, how to put confidence intervals on eval scores, how paired comparisons make small differences detectable, and how many examples you need.

Where the noise comes from

• Question sampling. Your eval set is a sample of possible inputs. A different 200 questions would give a different score.
• Model sampling. At temperature above zero, the same model and prompt give different outputs on each run.
• Grader noise. Model graders and human graders are not perfectly consistent; see calibrating model graders.

Each adds variance. A difference between two systems is only meaningful if it is large compared with that variance.

The basic formula

For a pass/fail eval with *n* independent questions and an observed pass rate *p*, the standard error is approximately:

SE = sqrt( p × (1 − p) / n )
95% confidence interval ≈ p ± 1.96 × SE

Some numbers to build intuition:

With 200 questions, an 80% score is consistent with a true pass rate anywhere from roughly 74.5% to 85.5%. A 2-point difference between two prompts sits comfortably inside that range.

For more precise intervals with small samples or scores near 0% or 100%, use the Wilson score interval, available in most statistics libraries.

Compare systems with paired tests

The table above describes a single score. When comparing two prompts or models on the same questions, you can do much better than comparing two independent intervals, because many questions are easy or hard for both systems. What matters is the questions where they disagree.

Count:

• b = questions system A passes and B fails;
• c = questions B passes and A fails.

If the systems were equally good, b and c would be similar. McNemar's test checks whether the difference between b and c is larger than chance would explain.

from statsmodels.stats.contingency_tables import mcnemar

def compare(a_pass, b_pass):
    """a_pass, b_pass: lists of booleans, same questions in the same order."""
    both = sum(a and b for a, b in zip(a_pass, b_pass))
    a_only = sum(a and not b for a, b in zip(a_pass, b_pass))
    b_only = sum(b and not a for a, b in zip(a_pass, b_pass))
    neither = len(a_pass) - both - a_only - b_only
    result = mcnemar([[both, a_only], [b_only, neither]], exact=True)
    return {"a_only": a_only, "b_only": b_only, "p_value": result.pvalue}

A related approach is the paired bootstrap: resample questions with replacement many times, compute the score difference each time, and take the 2.5th and 97.5th percentiles as a 95% interval for the difference. If the interval excludes zero, the difference is unlikely to be noise. The bootstrap works for any metric, including averages of rubric scores.

Handle repeated runs and grouped questions

If you run each question several times (for example at temperature 0.7), average the runs per question first, then compute the standard error across questions. Treating every run as an independent question overstates your precision.

Similarly, if questions come in groups, such as several questions about the same document, they are not independent. Errors cluster: if the model misreads the document, it fails several questions at once. Use clustered standard errors or bootstrap by resampling whole groups. Miller (2024), "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations," discusses these corrections in detail and is a good reference for teams formalising their evaluation practice.

How many questions do you need?

The number depends on the smallest difference you care about detecting. Roughly, for paired comparisons, what matters is the number of disagreements between systems; for independent comparisons, the rule of thumb below holds.

Paired designs typically need far fewer questions than this, because the shared easy and hard questions cancel out, but the order of magnitude is a useful warning: detecting a 2-point improvement reliably needs a large set. If your set has 100 questions, aim to detect large improvements and treat small ones as hypotheses.

Worked example: is the new prompt better?

Suppose you run two prompts on the same 300 questions. Prompt A passes 243 (81%); prompt B passes 249 (83%). Looking only at totals, B is two points ahead.

Now look at the disagreements. Say 21 questions are passed by B but not A, and 15 by A but not B; the remaining 264 questions get the same result from both. McNemar's test asks whether a split of 21 to 15 is unusual if the prompts were equally good. It is not: with 36 disagreements, a split this uneven or more happens by chance roughly 40% of the time, so the p-value is about 0.4.

The conclusion is not that B is no better. It is that this eval cannot tell yet. The options are to accept that the prompts are roughly equivalent and choose on other grounds such as cost or simplicity; to add more questions, focusing on the categories where the prompts disagreed; or to run each question several times to reduce sampling noise.

Notice how much more informative the disagreement count is than the totals. Thirty-six questions changed outcome; reading those 36 outputs will usually tell you more about the difference between the prompts than any statistic.

Practical rules

1. Report every score with an interval or at least the number of questions.
2. Compare on the same questions and use paired tests.
3. Fix temperature and settings across compared runs; see LLM temperature explained.
4. Decide what difference matters before running, so you are not tempted to celebrate noise.
5. Look at the disagreements, not only the totals. Ten questions that changed tell you more than a percentage.
6. Grow the set where decisions are close and the stakes are high. The guide to building an eval set covers how to add examples efficiently.
7. Automate the statistics in your harness so nobody has to remember; see the eval harness checklist.

Frequently asked questions

How do I calculate a confidence interval for an LLM eval score?

For a pass rate p on n independent questions, the standard error is sqrt(p × (1 − p) / n), and a 95% confidence interval is roughly p ± 1.96 × SE. Use the Wilson interval for small samples or extreme scores.

Is a 2% improvement on my eval significant?

Usually not on a few hundred questions. A 95% interval on 200 questions at 80% is about ±5.5 points. Use a paired test such as McNemar's on the same questions to check whether the difference is likely real.

What is a paired comparison in model evaluation?

Running two systems on the same questions and analysing the questions where they disagree. Because shared easy and hard questions cancel out, paired comparisons detect real differences with fewer questions than comparing separate scores.

How should I handle multiple runs per question?

Average the runs for each question first, then compute the standard error across questions. Treating each run as a separate question makes your results look more precise than they are.