Showing 1–4 of 4 · #inference
MODELSQuantization without tears: what you lose at 8, 4 and 3 bits
Smaller weights, faster inference, and a quality cost that depends far more on the task than on the bit count. A practical guide to choosing a level.
JLJonas LindOct 28 · 6 min
TESTSHow to run an 8B LLM on a laptop: hardware, quantization and tools
Memory maths, quantization choices, tool options and a quick way to predict speed. Everything you need to run a capable model locally.
JLJonas LindOct 26 · 6 min
PAPERSSpeculative decoding: faster answers without changing them
A small model drafts, a big model checks several tokens at once, and the output distribution stays exactly the same. How and when it speeds things up.
JLJonas LindOct 11 · 6 min
PAPERSMixture of experts, without the hype
Many parameters, few used per token. How MoE routing works, what "active parameters" really means, and the trade-offs for anyone running or buying these models.
MOMira OkaforSep 14 · 6 min