Tokenization / Detailed public lesson
How text becomes model input.
A tokenizer is the adapter between human text and model computation. It does not understand the request; it converts a string into a model-specific sequence of token IDs. That sequence is what the model can actually receive, budget, cache, bill, and continue.
The objective is not to memorize one tokenizer's output. It is to understand the boundary, predict why token counts vary, and connect that variation to context, cost, and latency.
- Why this comes next
- The request trace established tokens and IDs as shared terms. This lesson now opens that boundary without jumping ahead to model internals.
- What success looks like
- Explain why visible words and tokens differ, what token IDs represent, and why token count affects system constraints.
- Evidence mode
- Compare examples, explain the differences, then apply the idea to an unfamiliar string.
Tokens are model-specific pieces, not universal words.
Lesson 0001 gave the whole request trace. This lesson zooms into the first hard boundary. The operational win is that we stop estimating by words and start asking what the model-visible sequence really is.
The mechanism has four moves: preserve the exact text, segment it into token pieces, map those pieces to vocabulary IDs, then budget the resulting sequence.
Visible text → token pieces → integer token IDs
The IDs are vocabulary references. They are not meaning scores, embeddings, or the original characters. Understanding begins downstream when those IDs become vectors and pass through learned weights.
Misconception trap: “Tokens are just words.”
Challenge it. Tokens can be spaces, bytes, punctuation, subwords, whole words, or emoji. Even the same visible word can receive a different ID when capitalization or a leading space changes.
The shape of the text changes the token path.
Common sequences often fit vocabulary entries efficiently. Unusual names, compressed identifiers, source code, mixed scripts, repeated punctuation, and long numbers may require more pieces. Whitespace can matter because some tokenizers encode a leading space together with the following text.
This is why character count and word count are only rough intuitions. The tokenizer, vocabulary, and exact bytes determine the actual token count.
| Text | Pieces or IDs | What changed |
|---|---|---|
AI backend | AI 17527 + backend 38775 | Two tokens. |
AI backend | AI 20837 + backend 38775 | The leading space changed the first ID. |
red / red | 1291 / 3592 | A space can be part of a token. |
Red / Red | 7805 / 5952 | Capitalization and position can change IDs. |
antidisestablishmentarianism | 6 tokens | A rare long word splits into reusable subwords. |
{"query":"vector search","top_k":3} | 11 tokens | JSON punctuation and key names consume budget. |
お誕生日おめでとう | 8 tokens | Visible character count does not predict multilingual token count. |
The real lesson poster keeps the entire boundary visible.
It traces human-visible text through segmentation, token pieces, vocabulary lookup, embedding handoff, and serving consequences. It also preserves the practical controls and misconception checks used in the original lesson.
Subwords balance three bad extremes.
Whole-word vocabularies are readable but explode for rare words, names, code, typos, and new terms. Character or byte sequences cover nearly everything but create long sequences. Subword methods keep common pieces compact while decomposing rare strings into known pieces.
- Whole words
- Compact for common vocabulary, brittle and enormous for the long tail.
- Characters or bytes
- Broad coverage, but longer sequences increase context pressure and serving work.
- Subwords
- The practical compromise used by families such as BPE, WordPiece, Unigram, and SentencePiece.
- Operational result
- Formatting is not cosmetic. Prose, code, JSON, markdown, multilingual text, and tool definitions can change backend cost.
Token count becomes a systems constraint.
Models operate within token-based context limits. More input tokens consume more of that capacity. They also increase the amount of input work during prefill, and billed APIs commonly measure input and output usage in tokens. Output tokens are generated repeatedly, so longer generation can add decode steps and latency.
Token count does not explain every performance or cost difference, but it is one of the first quantities a backend engineer should inspect.
Reviewed public sources
The source record is also retained in the linked LLM Wiki knowledge layer so later sessions can retrieve the concepts and provenance. Private machine paths and internal workspace links are intentionally omitted from this public projection.
Apply the boundary, not just the vocabulary.
A tokenizer converts text into what model-visible form?
Token IDs. It does not directly produce word meanings, final logits, or training rows.
Why can red and red have different IDs?
The leading space can be part of the token piece, so the vocabulary lookup changes.
What should you do before trusting a prompt budget?
Count tokens with the active model's tokenizer rather than estimating from pages, words, ideas, or tasks.
What does tokenization not do by itself?
Understand meaning. It segments strings and maps pieces to IDs; learned interpretation happens downstream.
Why do subwords help with rare strings?
They decompose unfamiliar strings into reusable known pieces instead of requiring a whole new vocabulary entry.
A strong response predicts backend consequences without memorizing exact IDs. The durable skill is inspecting the active tokenizer when context fit, cost, latency, or exact behavior matters.
Evidence decides whether to deepen or remediate.
If the learner can apply the boundary to a new string, the map can move toward embeddings, context pressure, or tokenizer tradeoffs. If the learner treats IDs as meanings or assumes one token per word, Curio keeps the frontier here and proposes a smaller comparison exercise.
Return to the topic atlas