Tokens & tokenization

~7 min read

Models never see words. Before anything else happens, your text is cut into pieces called tokens, and each piece is replaced by its ID number from a fixed list (the vocabulary). Everything downstream — context limits, attention, pricing — counts in tokens, not words or characters.

Why pieces, not words?

Three options were on the table, and pieces won on tradeoffs:

Whole words — vocab explodes (every name, typo, and plural needs an entry), and unknown words are unrepresentable.
Characters — tiny vocab, but sequences get ~4× longer, and the model must re-learn spelling before meaning.
Subwords — common words stay whole (the), rare words split into reusable parts. Vocabularies land around 30K–200K entries. Best of both.

The dominant algorithm is a Byte-Pair Encoding (BPE) variant: start from bytes, repeatedly merge the most frequent adjacent pair, stop at the target vocab size. The result is deterministic — the same string always tokenizes the same way for a given tokenizer.

What it looks like

Illustrative split (exact boundaries differ per tokenizer):

unbelievable  →  [un] [believ] [able]     (3 tokens)
hello        →  [hello]                 (1 token)
संभव         →  [सं] [भव]                (2 tokens, often more)

Notice the last row: scripts the tokenizer saw rarely during training split into many small pieces. Same meaning, more tokens — which is why some languages cost noticeably more per sentence than English.

Fertility

Fertility = tokens per word. English prose sits near 1.3; code and non-Latin scripts run higher. When someone says a model has a "200K context window," that is ~150K English words but far fewer characters of dense code or Hindi text.

Why you should care

Limits are token limits. A 200K window fits fewer real words than the number suggests.
Bills are token bills. Every API prices input and output per million tokens — output tokens usually cost 3–5× input.
Tokenizers differ per model. The same prompt is a different token count (and a different bill) on each provider.

The one number to remember

Rough estimate: 1 token ≈ 4 characters of English text. Good for back-of-envelope math; never for billing disputes — always count with the real tokenizer.

Check your understanding

Progress saves on this device only

1.Why do most models use subword tokens instead of whole words?
2.A 200K-token window holds roughly how many English words?
3.The same sentence in Hindi often costs more tokens than in English. Why?