Why code and JSON cost more tokens than prose

English prose is the best case for a tokenizer. Common words become single tokens, and a page of ordinary writing lands close to the familiar rule of thumb — somewhere near four characters per token. Then you paste in a stack trace and the count doubles for the same screen of text.

That is not a bug in the counter. It is what tokenizers do.

Tokenizers are trained on a corpus#

A tokenizer learns its vocabulary from data. Sequences that appeared often during training earn a slot as a single token; everything else gets assembled from smaller fragments. Since that training data is overwhelmingly natural language from the web, natural language is what compresses well.

Anything unlike that corpus fragments. And code is very unlike it.

Where the tokens actually go#

Identifiers split at every boundary. getUserPreferences reads as one idea and is typically four or five tokens: get, User, Prefer, ences. A file full of descriptive camelCase names — which is to say, good code — is a file full of multi-token identifiers. snake_case fares no better, because the underscore is usually its own token.

Indentation is charged by the level. Four spaces of leading whitespace on a deeply nested block is not free, and it repeats on every line. A JSON document indented to six levels can spend a substantial fraction of its tokens on structure that carries no information the model needs.

Punctuation density adds up. Prose has a comma every dozen words. JSON has {, }, ", :, and , constantly, and each is generally its own token. A key-value pair like "userId": "abc123" can cost seven or eight tokens to convey what a person would read as two words.

Non-English text is penalised the same way. Languages underrepresented in the training corpus can take two to three times as many tokens for identical meaning. Character-based scripts are worse again — a Japanese sentence may spend a token per character where English spends one per word.

What to do about it#

Most of this overhead is structural rather than essential, which means you can remove it without removing information.

• Minify JSON before sending it. Stripping insignificant whitespace from a nested payload is one line of code and frequently cuts 20–30% of its tokens. The model does not need the indentation; you needed it to read the file.
• Send a schema and a sample, not the whole dataset. Twenty representative rows plus a description of the columns usually teaches the model as much as two thousand rows, at a fraction of the cost.
• Trim code to the relevant span. A whole file for a question about one function spends most of its budget on context the answer never touches. Include the function, its immediate callers, and the types it depends on.
• Drop comments and license headers from pasted code. They are for humans reading the repository, and a 30-line license block at the top of every file is 30 lines of pure cost on every request.
• Prefer compact formats where you control them. CSV or TSV is markedly cheaper than the equivalent JSON, because the field names appear once in a header row instead of on every record.

Measure it rather than assume it#

The savings from each of these varies enormously by input. Minifying an already-compact payload gains almost nothing; minifying deeply nested configuration can be dramatic.

The direct way to find out is to paste the before and after into this calculator and read the difference. GPT counts run in your browser using the real tokenizer, so the number is exact rather than a heuristic, and nothing you paste leaves the page — which matters when the text you are measuring is a production payload or proprietary source.