What a token actually is

Most explanations of tokens stop at "roughly four characters, or about three-quarters of a word." That rule of thumb is accurate enough to budget with and useless for understanding why your particular text costs what it does — why a paragraph of German costs half again what the English costs, why a UUID is more expensive than the sentence containing it, and why a model that can write a sonnet cannot count the letters in strawberry.

All of that follows from one thing: tokenization is a compression algorithm, and it was fit to somebody else's data.

The algorithm, briefly#

The dominant method is byte pair encoding, and it is startlingly simple.

Start with a vocabulary of the 256 possible bytes. Scan a large corpus and find the most frequent adjacent pair of symbols. Merge that pair into a single new symbol and add it to the vocabulary. Repeat, tens of thousands of times.

Early merges produce common letter pairs — th, in, er. Later ones produce fragments, then whole common words, then common words with their leading space attached. Stop after 100,000 merges and you have a 100,000-entry vocabulary; stop after 200,000 and you have a larger one. That vocabulary is the tokenizer. Encoding new text means greedily applying those learned merges.

Two properties fall straight out of this, and between them they explain nearly everything.

Frequency determines efficiency. A sequence that appeared often in the corpus got merged early and is one token. A sequence that appeared rarely never got merged and is billed as its fragments. The tokenizer is not measuring meaning or length. It is measuring how ordinary your text was.

There is no failure case. Because the vocabulary bottoms out at raw bytes, any input encodes successfully. Unusual text is never rejected — it is just expensive. That is why tokenization problems show up on the invoice rather than in the logs.

What this predicts#

Once you see the mechanism, the surprising counts stop being surprising.

Rare words fragment. getUserPreferences is one identifier to you and typically four or five tokens to a model, because that exact string was not frequent enough in the corpus to earn a merge — but get, User, and Preference were.

The space is part of the token. " the" with its leading space is a different, and more common, token than "the" without one. This is why doubled spaces, trailing whitespace, and text glued together without spaces all cost more than they look like they should.

Case matters. The, the, and THE are three distinct entries. A block of shouty capitals or a SCREAMING_SNAKE_CASE constant list tokenizes noticeably worse than the same words in lower case.

Numbers split into arbitrary chunks. Long digit strings are broken into groups of one to three digits, chosen by corpus frequency rather than by place value. 1234567 might split as 123|456|7. The model never sees the number — it sees fragments whose boundaries do not respect arithmetic. This, not a lack of intelligence, is most of why models are unreliable at multi-digit arithmetic.

Non-English text pays a byte tax. Most tokenizers were fit on corpora dominated by English. A character outside the Latin range takes two to four bytes in UTF-8, and if its surrounding sequences were not frequent enough to merge, you pay per byte. The same meaning, expressed in a less-represented language, can cost two to three times as many tokens — a real and rarely-discussed cost asymmetry in multilingual products.

Structure costs. Deeply indented JSON, XML, and pretty-printed data spend a substantial share of their tokens on braces, quotes, and whitespace. Minifying a payload before sending it is one of the few edits that reduces cost with zero effect on meaning.

Random strings are the worst case. UUIDs, hashes, base64 blobs, and API keys have no learnable structure by construction. They approach one token per two or three characters. A log excerpt full of request IDs can cost twice what its length suggests.

Why the model cannot spell#

The strawberry question — how many Rs are in it — is a good illustration because it looks like a reasoning failure and is not.

The model never receives the letters. It receives something like str|aw|berry: three symbols, each an opaque vocabulary index. Asking it to count characters is asking it to recover information that was discarded before inference began. It can often answer correctly anyway, by having memorised facts about spelling from text that discussed spelling — which is a different and much less reliable mechanism than looking.

The practical lesson is narrow but useful: character-level operations — counting, reversing, checking an exact prefix — are the wrong job for a language model. Do them in code.

What the ratios actually are#

For planning, the numbers behind the rule of thumb are worth knowing, because the spread is wider than "about four characters" suggests. Measured across a mixed corpus for this tool, both current GPT encodings land near 4.1 characters per token overall — but that average conceals the range:

• Clean English prose: about 5.0 characters per token. The best case.
• Mixed instruction text: around 4.1.
• Code: roughly 3.3, driven by identifiers, punctuation and indentation.

So the same 10,000 characters is about 2,000 tokens as prose and about 3,000 as TypeScript — a 50% difference in cost between two files of identical size. On GPT-4o at $2.50 per million input tokens that is small in isolation and material at volume.

Why the count differs between providers#

Every provider trains its own tokenizer, on its own corpus, to its own vocabulary size. Nothing standardises them.

Larger vocabularies generally compress better — more sequences got merged, so more text fits in one token — which means the same paragraph legitimately yields different counts on different models. Counts also change within a provider when a new generation ships with a retrained tokenizer, which is why a count you measured last year is not necessarily the count you will be billed for today.

This is the reason a comparison tool has to run each model's real tokenizer rather than divide by four. It is also why estimates for models whose tokenizer is not public are estimates, and should be labelled as such.

What to do with this#

Three things follow directly.

Tokenize the text you actually send — including the system prompt, the tool definitions, and the retrieved documents — rather than the user's message alone. The expensive parts of a request are usually the parts nobody typed.

Expect your code and non-English content to cost more per character than your prose, and check them separately rather than applying one blended ratio.

And treat any character-level requirement as a code problem. The tokenizer threw those characters away several steps before the model saw them.