Home/Archive/Papers

The archive

48 posts
Showing 1–1 of 1 · #cost
Tokenizers: the layer everyone ignores until it bitesPAPERS

Tokenizers: the layer everyone ignores until it bites

Why models struggle to count letters, why some languages cost more, and why numbers behave strangely. The tokenizer explains more than you think.

JLJonas LindJul 26 · 6 min

Get the Sunday Diff — every post worth reading, once a week.