A tokenizer maps text to token IDs before a language model sees it. Token budgets constrain context, output, latency, and cost.
Tokens are model-specific
A token may be a word, a word fragment, punctuation, whitespace, or bytes. The same text can have different counts across tokenizers. Use the exact tokenizer and chat template paired with the model revision.
Budget the whole request
Count system instructions, conversation history, tool schemas, retrieved passages, user input, and reserved output. If the total exceeds the context window, choose an explicit truncation or summarization policy.
Operational checks
Pin model and tokenizer revisions. Test multilingual text, code, emoji, special tokens, and truncation direction. Measure counts with the real tokenizer; character-based estimates are planning aids only.
Approximation boundary
Demonstrate one English sentence, one French sentence, and one code sample. Do not present the approximation as an exact tokenizer.