How LLMs split text into tokens with subword methods like BPE, why counts differ by model family, and how Exact vs Approx labeling works in TokenCALC.
Try it in TokenCALC
Paste a prompt on the tokenizer to see Exact or Approx counts for your model.
Tokenization is how text becomes model input
Tokenization is the process of splitting raw text into the discrete units a language model consumes. Modern LLMs usually use subword tokenization, often based on byte-pair encoding (BPE) or related methods, rather than pure word or pure character splits.
That design is a compromise. Pure word tokens struggle with rare words and typos. Pure character tokens make sequences very long. Subwords keep vocabulary manageable while still representing almost any string.
What a tokenizer vocabulary is
A tokenizer ships with a vocabulary: the set of pieces it knows how to emit. Common English fragments often become single tokens. Rare or long words may split into several pieces. Spaces are frequently attached to the start of the next word rather than standing alone.
Because vocabularies differ, the same sentence can tokenize differently on OpenAI, Anthropic, Google, DeepSeek, or open-weight hosts. Tokenization is model-family specific, not universal.
Why providers disagree on counts
If two APIs use different vocabularies, they will disagree on token count even when the UTF-8 text is identical. That is expected. It is also why you should not budget Claude spend with an OpenAI tokenizer, or assume Gemini matches GPT on a pasted paragraph.
TokenCALC shows Exact only when we run a matching browser tokenizer for that family. Today that path is strongest for OpenAI encodings such as o200k_base via gpt-tokenizer. Other providers stay Approx unless or until a trustworthy offline tokenizer exists.
English heuristics vs real tokenization
Rules like “four characters per token” or “0.75 words per token” are useful napkin math for English prose. They fail more often on code, URLs, tables, IDs, and many non-English scripts. Treat heuristics as planning shortcuts, then tokenize the real prompt.
The tokens vs words guide covers conversion formulas and failure modes. The Tokens ↔ words tool applies the English heuristic interactively. Neither replaces model-specific tokenization for shipping budgets.
Chat templates and hidden tokens
API requests are more than the user string you see in a text box. System prompts, tool definitions, multimodal parts, and chat formatting can add tokens you did not type into a demo field. Provider count APIs that accept the full request body are the safest way to measure that overhead.
When you paste only the user message into TokenCALC, you still get a useful core count, but remember production wrappers can add more. Size context windows with margin.
Exact vs Approx on TokenCALC
Exact means we use gpt-tokenizer with an OpenAI encoding such as o200k_base (and some cl100k_base paths). Approx means the provider tokenizer is not available in-browser, so we use a labeled heuristic instead of pretending the number is exact.
Read the Exact vs Approx guide for budgeting tips, and open Claude or Gemini bridge guides for provider-specific counting advice (including official count_tokens / countTokens APIs).
Practical workflow
Use this sequence when tokenization actually matters for money or fit.
- Paste the real prompt into the tokenizer with the target model selected
- Note Exact vs Approx and add margin for Approx or missing chat wrappers
- Check the context meter before long documents
- Send the same prompt through the cost calculator for spend
- Confirm critical Claude/Gemini counts with provider APIs before large commits
Common mistakes
Assuming one tokenizer rules all providers. Copying tiktoken counts into a Claude spreadsheet. Ignoring punctuation and whitespace. Forgetting that “same words” is not “same tokens.”
Tokenization quality also interacts with cost: a denser tokenizer can mean fewer tokens for the same text, which changes both context fit and price. Always measure on the model you will call.
Real-world scenarios
A legal team pastes contract clauses with numbered sections and defined terms. Tokenizers often attach spaces to word starts and split long compound terms. Two contracts with similar word counts can differ by hundreds of tokens depending on formatting and punctuation density.
Developers embedding code snippets in prompts see inflated counts because symbols, operators, and indentation become many small tokens. A fifty line function can cost more tokens than a fifty line email with similar character length.
RAG pipelines prepend retrieved chunks before the user question. Tokenization runs on the combined string, so chunk boundaries and metadata headers affect the final count. Reordering chunks can change token totals even when the information content is the same.
Step-by-step in TokenCALC
Start on the tokenizer page and pick your target model from the catalog. Paste representative production text, not lorem ipsum. Include system prompts and tool definitions if they are stable parts of every request.
Compare the same paste on a second model family to see vocabulary differences in action. Note Exact vs Approx badges and treat Approx results as planning numbers with margin.
If the context meter shows tight or overflow, trim or summarize before you open the cost calculator. Tokenization and fit checks should precede price optimization.
Related concepts
The what is a token guide defines the billing unit. Exact vs Approx explains labeling policy when browser tokenizers exist or do not. Tokens vs words covers heuristics when you lack the raw text.
Context windows shows how token totals compete with output reservation. Provider bridge guides describe official count APIs for Claude and Gemini after your Approx estimate.
Expert notes
Pre tokenization normalization (Unicode normalization, stripping zero width characters, collapsing whitespace) can shift counts slightly. Most teams tokenize the string they actually send rather than an idealized cleaned version, because that matches billing.
Multimodal inputs add non text tokens for images or audio when models support them. TokenCALC focuses on text tokenization today. Check provider docs for modality specific counting rules on those workloads.
Open weight models hosted on third party GPUs still use the model family tokenizer even when the host brand differs. Pick the model row that matches the weights, not only the hosting company name.
Warnings
Never ship production limits based solely on English word heuristics. Never assume tiktoken output applies to non OpenAI APIs. Never compare two models on token integers without comparing the same UTF-8 string.
Chat templates applied server side can add special tokens you did not paste. Undercounting hidden wrapper tokens is a common cause of surprise context overflows in staging.
Frequently asked questions
What is tokenization?
Tokenization splits text into the token units a language model reads and generates. It happens before the model processes your prompt and determines billing units.
Is tokenization the same for every LLM?
No. Each major provider or model family typically uses its own vocabulary and rules. Counts on identical text can differ across families.
What is BPE?
Byte-pair encoding is a common subword method that builds tokens from frequent character sequences. Many modern LLM tokenizers are BPE variants or closely related algorithms.
Why does my token count change when I switch models?
Different vocabularies split the same text differently, so counts change even when the string does not. This is expected, not a bug in your prompt.
Should I use word count instead?
Only for rough planning. APIs bill tokens. Convert words to tokens only as a temporary estimate, then tokenize real text before you commit.
Does punctuation affect tokenization?
Yes. Punctuation and whitespace are often merged with adjacent pieces or split into their own tokens, which changes totals compared to a naive word split.
Next steps
Use the calculator links above for Exact or Approx counts on your own prompts, then browse related guides and model pages to compare pricing assumptions.