BENCHMARKSBenchmark contamination: how to check whether a model has seen the test
Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.
BENCHMARKSLost in the middle: why long-context models still forget, and how to test it
Needle tests are easy to pass. Real documents need several facts combined from different places. Here is how to measure long-context recall properly.
BENCHMARKSYour 2-point improvement might be noise: confidence intervals for LLM evals
A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.
BENCHMARKSDesigning an LLM latency benchmark that matches what users feel
Time to first token, tokens per second and p95 under real load. A method for measuring LLM speed that predicts user experience, not lab conditions.
BENCHMARKSWhen a benchmark saturates, what should replace it?
When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.
AGENTSEvaluating AI agents end to end
Outcome checks, trajectory review, sandboxes, simulated users and pass^k. A complete method for knowing whether your agent really works.