Before a language model sees your text, a tokenizer chops it into pieces called tokens and converts each into a number. The model never sees letters or words directly, only sequences of token IDs. It is the least glamorous part of the stack, and it quietly explains a surprising amount of model behaviour: why models struggle to count letters, why the same text costs more in some languages, why arithmetic with long numbers is error-prone, and why context limits and prices are expressed in tokens rather than words.
This explainer covers how modern tokenizers work, the practical effects on cost and quality, and how to work with them instead of against them.
Why not just use characters or words?
- Words produce an enormous vocabulary, still miss new words, typos and names, and handle languages without spaces poorly.
- Characters have a tiny vocabulary but make sequences very long, which makes models slower and more expensive, since computation grows with sequence length.
Subword tokenization sits between the two: common words become single tokens, rarer words are split into pieces, and anything can be represented by falling back to smaller units.
How BPE works
The most common approach is byte-pair encoding (BPE), adapted for neural language processing by Sennrich, Haddow and Birch (2016).
- Start with a base vocabulary of individual characters or bytes.
- Count every adjacent pair of symbols in a large training corpus.
- Merge the most frequent pair into a new symbol and add it to the vocabulary.
- Repeat until the vocabulary reaches the target size, often tens of thousands to a couple of hundred thousand tokens.
Frequent words end up as single tokens ("the", " house"), while rare words are composed of several ("tokenization" might become "token" + "ization"). Byte-level BPE, used since GPT-2, starts from raw bytes, so any text, including emoji and any script, can be encoded without unknown tokens.
Related methods include WordPiece and SentencePiece (Kudo and Richardson, 2018). SentencePiece treats input as a raw stream, including spaces, which works well for languages without whitespace between words.
Quirks you will notice
Leading spaces are part of tokens
" Paris" and "Paris" are usually different tokens. That is why prompts and stop sequences behave differently with and without a trailing space, and why a structured format can break on whitespace details.
Letters are hidden inside tokens
When "strawberry" is one or two tokens, the model does not directly see its letters. Counting characters, reversing strings or spelling-based puzzles require the model to have learned what each token contains, which is why such tasks are error-prone. Asking the model to spell the word out first, character by character, often helps because each character becomes its own token.
Numbers are split inconsistently
Depending on the tokenizer, "1234567" might be split as "123" + "456" + "7" or in other chunks, and similar numbers may split differently. This complicates arithmetic and digit-level comparison. Some newer tokenizers split numbers into single digits or fixed groups for this reason. For exact calculations, give the model a calculator or code tool.
Rare strings can trigger odd behaviour
Tokens that were in the vocabulary but rare in training data can produce strange outputs; researchers documented "glitch tokens" such as " SolidGoldMagikarp" in earlier models in 2023. Today's models handle this better, but unusual identifiers and encodings still occasionally cause surprises.
Tokens and cost
Prices, rate limits and context windows are measured in tokens. A common rule of thumb for English with modern tokenizers is roughly four characters or about three-quarters of a word per token, so 1,000 tokens is around 750 English words. Treat this as a rough estimate only: code, numbers, rare names and especially non-English text can differ substantially.
# Count tokens with the tokenizer your model actually uses.
# Many providers offer a token-counting endpoint or library; open models ship their tokenizer.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<model-id>")
for text in ["Hello, world!", "Bonjour tout le monde", "こんにちは世界", "SKU-88412-B"]:
print(len(tok.encode(text)), "tokens:", text)Different model families use different tokenizers, so the same text can produce different token counts. When comparing prices between models, compare cost per task on your real inputs, not price per token; see choosing a model is a budget decision.
Multilingual fairness
Tokenizers are trained on corpora dominated by some languages, so those languages get efficient tokens and others are split into many small pieces. Petrov et al. (2023), "Language Model Tokenizers Introduce Unfairness Between Languages," measured large differences in how many tokens the same content requires across languages, with some languages needing many times more tokens than English.
Consequences:
- higher cost for the same content in underrepresented languages;
- less effective context, since the same window holds less text;
- slower responses, because more tokens must be generated;
- potentially lower quality, as heavily fragmented text is harder to model.
If you serve multiple languages, measure token counts per language on real text and include them in cost estimates.
Special tokens and chat templates
Beyond ordinary text, tokenizers define special tokens that mark structure: the start and end of a message, the boundary between system, user and assistant turns, and tool calls. Chat APIs insert these for you through a chat template. When you run open models yourself, using the wrong template, for example one from a different model family, is a common cause of degraded output, endless generation or ignored instructions. Use the template shipped with the model, and never let user-supplied text inject strings that look like special tokens into raw prompts.
Practical advice
| Situation | What to do |
|---|---|
| Estimating cost | Count tokens with the real tokenizer on real inputs |
| Counting letters, reversing strings | Ask the model to spell out characters first, or use code |
| Arithmetic with long numbers | Use a calculator or code-execution tool |
| Stop sequences and delimiters | Test exact strings, including leading spaces and newlines |
| Structured output | Prefer common key names; validate in code |
| Multilingual products | Budget and benchmark per language |
| Long prompts | Watch token growth from IDs, URLs, JSON and whitespace |
Some formatting choices are surprisingly expensive: long UUIDs, base64 data, deeply indented JSON and repeated boilerplate can consume many tokens. Removing them from prompts saves cost and leaves more room for useful context; see context windows are not memory. For reasoning models, remember that hidden reasoning tokens count too; see what reasoning tokens actually cost you. And when parsing model output, the tips in structured output without JSON mode help avoid whitespace-related surprises.
Frequently asked questions
What is a token in an LLM?
A token is a unit of text, often a word, part of a word, a punctuation mark or a byte sequence, that the model processes. Text is converted into a sequence of token IDs before the model sees it, and outputs are generated one token at a time.
How many words are in 1,000 tokens?
For typical English text with modern tokenizers, roughly 700 to 800 words. Code, numbers, rare names and many non-English languages use more tokens per word, so count with the actual tokenizer for accurate estimates.
Why can't language models count the letters in a word?
Because they see tokens rather than individual characters. A word may be a single token whose spelling the model must infer. Asking the model to spell the word out character by character first, or using code, usually fixes it.
Why does the same text cost more in some languages?
Tokenizers are trained mostly on certain languages, which get efficient tokens. Other languages are split into more pieces, so the same content needs more tokens, increasing cost and reducing how much fits in the context window.
