Showing 1–3 of 3 · #performance

MODELS

Quantization without tears: what you lose at 8, 4 and 3 bits

Smaller weights, faster inference, and a quality cost that depends far more on the task than on the bit count. A practical guide to choosing a level.

JLJonas LindOct 28 · 6 min

PAPERS

Speculative decoding: faster answers without changing them

A small model drafts, a big model checks several tokens at once, and the output distribution stays exactly the same. How and when it speeds things up.

JLJonas LindOct 11 · 6 min

BENCHMARKS

Designing an LLM latency benchmark that matches what users feel

Time to first token, tokens per second and p95 under real load. A method for measuring LLM speed that predicts user experience, not lab conditions.

RKRafael KimSep 08 · 6 min