Showing 1–5 of 5 · #benchmarks
BENCHMARKSBenchmark contamination: how to check whether a model has seen the test
Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.
SPSana PatelNov 05 · 6 min
BENCHMARKSLost in the middle: why long-context models still forget, and how to test it
Needle tests are easy to pass. Real documents need several facts combined from different places. Here is how to measure long-context recall properly.
MOMira OkaforOct 22 · 6 min
BENCHMARKSYour 2-point improvement might be noise: confidence intervals for LLM evals
A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.
JLJonas LindOct 01 · 6 min
BENCHMARKSDesigning an LLM latency benchmark that matches what users feel
Time to first token, tokens per second and p95 under real load. A method for measuring LLM speed that predicts user experience, not lab conditions.
RKRafael KimSep 08 · 6 min
BENCHMARKSWhen a benchmark saturates, what should replace it?
When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.
SPSana PatelAug 05 · 6 min