Word count is the wrong unit, and that is how a prompt quietly gets cut off or a bill comes in bigger than planned.
The model never sees a word. It sees a list of ID numbers, and the length of that list is what you pay for and what has to fit.
Those numbers stand for tokens: chunks of text somewhere between a single letter and a whole word. Your text is chopped into them, each chunk is swapped for its ID number from a fixed list the model was built with, and that list is what the math runs on.
The chopping rule is BPE (Byte Pair Encoding). Start from single characters, then repeatedly glue together whichever pair shows up most often in the training text, until that list is full. Frequent things end up as one chunk. Rare things stay in pieces.
OpenAI's own example: " the" is one token, while " tokenization" comes out as " token" plus "ization". The detail worth noticing is that the leading space belongs to the token, not the word after it. Typos, code identifiers like getUserById, and languages that were rare in training all shatter into more pieces than their length suggests.
So the conversion is a guess, and it moves per model, and even per version: OpenAI puts English at roughly 4 characters per token, Anthropic's glossary puts Claude at roughly 3.5, and Anthropic says its newer Claude tokenizers pack noticeably more tokens into the same text than that number assumes. I treat those as back-of-envelope only, never as a promise that something fits.
When it has to be right, count tokens with the model's own tokenizer, the tool that does the chopping. Word counts are for humans.
Quick check before you scroll: Why can't you just count words and assume that's roughly the token count for cost or context-window planning?
Full breakdown + the answer: frankduah.me/learnings/2026-09-14-tokens-vs-words-vs-characters-what-a-model-sees
New here? I post a bite-size AI / ML concept like this every day - follow me for the daily drop, and it compounds fast. Why I do it: https://lnkd.in/gK8knHDH
#TokensVs #AI #LLM #AIAgents #MachineLearning
The answer
Because tokenization is sub-word, not word-based - the ratio (roughly 0.75 words per token in English) shifts a lot for code, rare words, typos, or non-English text, so a word count is only a rough proxy, not the real unit the model bills and reasons in.