BENCHMARKSBuild an LLM eval set in an afternoon
Fifty real examples, a clear rubric and a script beat any public benchmark for your product. A four-hour plan to build your first eval set.
BENCHMARKSBenchmark contamination: how to check whether a model has seen the test
Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.
BENCHMARKSModel graders are useful, if you calibrate them first
Position bias, verbosity bias and self-preference. How to measure a model grader against human labels, fix its rubric and keep it honest over time.
BENCHMARKSLost in the middle: why long-context models still forget, and how to test it
Needle tests are easy to pass. Real documents need several facts combined from different places. Here is how to measure long-context recall properly.
BENCHMARKSYour 2-point improvement might be noise: confidence intervals for LLM evals
A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.
BENCHMARKSWhen a benchmark saturates, what should replace it?
When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.
BENCHMARKSThe LLM eval harness checklist we wish we had on day one
Versioned datasets, pinned models, stored outputs, error bars and a CI gate. A checklist for turning ad hoc eval scripts into infrastructure you trust.