TESTSHow to test whether an AI model really reasons: the contradiction-trap eval
A planted contradiction, blind judges and zero partial credit. A step-by-step method for finding out whether a model tracks state or just sounds like it does.
MODELSHow to evaluate a new AI model in two weeks: a practical review framework
Launch-day benchmarks tell you little about your workload. A day-by-day plan for testing a new model on real tasks and reaching a defensible decision.
BENCHMARKSBuild an LLM eval set in an afternoon
Fifty real examples, a clear rubric and a script beat any public benchmark for your product. A four-hour plan to build your first eval set.
BENCHMARKSBenchmark contamination: how to check whether a model has seen the test
Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.
BENCHMARKSModel graders are useful, if you calibrate them first
Position bias, verbosity bias and self-preference. How to measure a model grader against human labels, fix its rubric and keep it honest over time.
BENCHMARKSLost in the middle: why long-context models still forget, and how to test it
Needle tests are easy to pass. Real documents need several facts combined from different places. Here is how to measure long-context recall properly.
AGENTSTool-use evals are broken. Here's how to fix them.
Exact-match grading, single runs and perfect mock APIs make tool-use scores look better than reality. A better harness design, step by step.
BENCHMARKSYour 2-point improvement might be noise: confidence intervals for LLM evals
A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.
BENCHMARKSWhen a benchmark saturates, what should replace it?
When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.