Between your sentence and the language model sits a step almost nobody explains: converting text into numbers. Tokenisation is that step, and understanding it demystifies several odd behaviours at once.
What tokenisation actually produces
Notice that tokens are not words. “Tokenisation” splits into “Token” + “isation”. “Unbelievably” splits into three pieces. Common words stay whole; rarer ones fragment into parts the tokeniser has seen before.
A rough rule for English: one token is about four characters, or ¾ of a word. 100 tokens is roughly 75 words.
How the vocabulary is built
Most models use Byte Pair Encoding, and the algorithm is simple enough to describe in a sentence: start with individual characters, repeatedly find the most frequent adjacent pair, and merge it into a new token — until you reach your target vocabulary size.
🧪 Byte Pair Encoding in miniature
Start with characters:
l o w e rThe pair
l o occurs most often → merge into loNow
lo w occurs most often → merge into lowThen
e r appears often → merge into erYou end up with tokens like
low, er, est. “Lower” is two tokens, “lowest” is two — and a word never seen before, like “lowness”, still tokenises fine as low + ness.Frequency in the training text decides everything. That is why common English words are single tokens and rarer words fragment.
Five behaviours tokenisation explains
Models miscount letters. Ask how many r’s are in “strawberry” and a model may get it wrong — not from stupidity but because it never sees the letters. It sees two or three token IDs. Counting characters inside a token is like counting the letters in a word you only ever heard as a single sound.
Reversing words is hard for exactly the same reason.
Some languages cost far more. Tokenisers are trained mostly on English, so English is efficient. The same sentence in Thai, Hindi or Burmese can take two to four times as many tokens — meaning higher cost, slower responses, and less room in the context window, for identical content.
Whitespace matters. ” the” and “the” are usually different tokens. A stray space can change tokenisation and, occasionally, output.
You are billed per token. Not per word or character. This is also why the context window is measured in tokens — “128k context” means 128,000 tokens, roughly 96,000 English words.
Tokenisation and cost
| Content | Roughly |
|---|---|
| 1 token | ~4 characters of English |
| 100 tokens | ~75 words |
| 1,000 tokens | ~1.5 pages of prose |
| Code | More tokens per character — punctuation and indentation add up |
| Numbers | Often split digit by digit, which is part of why arithmetic is hard |
⚠️ Do not estimate tokens by counting words
tiktoken for OpenAI models, or the tokeniser that ships with whichever model you are using. Word counts can be out by 50% or more on code, non-English text, or anything with unusual formatting.What happens next
Token IDs are not yet meaningful — ID 5771 has no more relationship to ID 5772 than any other pair. The next step converts each ID into a vector that does carry meaning, and that is embeddings, the following post in this sequence.
🔑 Key Takeaways
- Tokenisation splits text into sub-word chunks and maps each to an integer ID.
- Byte Pair Encoding builds the vocabulary by repeatedly merging the most frequent pair.
- Roughly 4 characters per token in English; other languages can cost several times more.
- Models cannot see letters, which is why they miscount characters and struggle with arithmetic.
- Cost and context windows are measured in tokens — count them properly, do not estimate.
Further reading
OpenAI’s tiktoken library lets you tokenise text yourself and see exactly how a string splits, and Hugging Face’s Byte Pair Encoding chapter walks through the merge algorithm step by step.
Where to go next
- Embeddings explained — the next post: turning IDs into meaning
- What is a large language model? — the previous post
- Encoding categorical variables — the same problem in classical ML
- Start Here — the full learning path