A tokenizer maps text to token IDs before a language model sees it. Token budgets constrain context, output, latency, and cost.
Tokens are model-specific
A token may be a word, a word fragment, punctuation, whitespace, or bytes. The same text can have different counts across tokenizers. Use the exact tokenizer and chat template paired with the model revision.
Budget the whole request
Count system instructions, conversation history, tool schemas, retrieved passages, user input, and reserved output. If the total exceeds the context window, choose an explicit truncation or summarization policy.
Operational checks
Pin model and tokenizer revisions. Test multilingual text, code, emoji, special tokens, and truncation direction. Measure counts with the real tokenizer; character-based estimates are planning aids only.
Quick check
Can two models assign different token counts to the same sentence?
Show answer
Yes. Vocabulary, normalization, and segmentation rules differ by tokenizer.
What must fit in the context window?
Show answer
The complete serialized request plus the output tokens reserved for generation.