BENCHMARKSBuild an LLM eval set in an afternoon
Fifty real examples, a clear rubric and a script beat any public benchmark for your product. A four-hour plan to build your first eval set.
BENCHMARKSBenchmark contamination: how to check whether a model has seen the test
Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.
BENCHMARKSModel graders are useful, if you calibrate them first
Position bias, verbosity bias and self-preference. How to measure a model grader against human labels, fix its rubric and keep it honest over time.
BENCHMARKSYour 2-point improvement might be noise: confidence intervals for LLM evals
A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.
MODELSHow to read a model card critically
Model cards are marketing and documentation at once. A checklist for separating the useful facts from the flattering ones.
BENCHMARKSWhen a benchmark saturates, what should replace it?
When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.