Showing 1–5 of 5 · #methodology

BENCHMARKS

Build an LLM eval set in an afternoon

Fifty real examples, a clear rubric and a script beat any public benchmark for your product. A four-hour plan to build your first eval set.

MOMira OkaforNov 06 · 6 min

BENCHMARKS

Benchmark contamination: how to check whether a model has seen the test

Canary strings, n-gram overlap, perturbed questions and fresh held-out sets. How to tell real capability from a memorised test.

SPSana PatelNov 05 · 6 min

BENCHMARKS

Model graders are useful, if you calibrate them first

Position bias, verbosity bias and self-preference. How to measure a model grader against human labels, fix its rubric and keep it honest over time.

SPSana PatelOct 24 · 6 min

BENCHMARKS

Your 2-point improvement might be noise: confidence intervals for LLM evals

A 2-point jump on 200 questions is usually noise. How to compute error bars, run paired comparisons and size your eval set to detect real gains.

JLJonas LindOct 01 · 6 min

BENCHMARKS

When a benchmark saturates, what should replace it?

When every frontier model scores in the 90s, the test has stopped telling them apart. Why benchmarks saturate and what comes after them.

SPSana PatelAug 05 · 6 min