What Is a Tokenizer? – Definition and How It Works
Learn what a tokenizer does, how subword algorithms work, why token counts vary by content type, and what that means for cost and context.
A tokenizer converts text into the discrete units a model actually processes. Every context limit and every price is expressed in those units, so tokenization determines both what fits and what it costs.
1. Why Not Words or Characters
- Words produce an unbounded vocabulary — every name, typo, and compound needs an entry.
- Characters produce sequences too long to model efficiently.
- Subwords compromise: common words become single tokens, rare ones split into fragments.
Subword tokenization means no input is ever out-of-vocabulary, because anything can be assembled from smaller pieces.
2. How Subword Vocabularies Are Built
The dominant approach starts from characters and repeatedly merges the most frequent adjacent pair, recording each merge, until a target vocabulary size is reached.
The result reflects the training corpus: text resembling that corpus tokenizes efficiently, and text unlike it fragments.
3. Why Counts Vary So Much
| Content | Efficiency | Reason |
|---|---|---|
| Common English prose | Best — roughly ¾ of a token per word | Whole words are single tokens |
| Technical vocabulary | Lower | Rare terms split into pieces |
| Code | Lower still | Punctuation and indentation tokenize separately |
| JSON with long keys | Poor | Keys repeat in every record |
| Non-Latin scripts | Often worst | Frequently several tokens per character |
The last row has real consequences: the same message can cost several times more in one language than another.
4. Practical Implications
- Measure, do not estimate. Use the model's own tokenizer before shipping a prompt template.
- Send clean text. Markup is pure overhead in the context window.
- Watch structured payloads. Compact keys and omitted nulls reduce cost materially at volume.
- Budget for output. Generated tokens share the same window and usually cost more.
5. Tokenization Explains Odd Behaviour
- Character counting is unreliable, because the model does not see characters.
- Rhyming and spelling tasks are harder than they look for the same reason.
- Rare names are handled inconsistently, since they fragment differently by context.
- Numbers split in ways that make arithmetic error-prone without tools.
None of these are reasoning failures; they follow from the input representation.
6. Different Models, Different Tokenizers
Token counts are not portable. The same text can differ substantially between model families, so cost comparisons must use each model's own tokenizer.
Context limits are equally non-comparable: two models advertising the same window can hold different amounts of the same text.
7. Working With It
- Count with the target model's tokenizer.
- Convert documents to clean text or Markdown before sending.
- Compact structured payloads.
- Measure prompt and completion tokens separately in cost tracking.
- Re-measure when changing model families.
8. Summary
A tokenizer maps text to the subword units a model processes, built by merging frequent character pairs over a training corpus.
Efficiency varies by content and language, counts are not portable between model families, and both context limits and billing are denominated in these units — which makes clean, compact input a direct cost control.