Why a long conversation costs more than you think

The first surprise on a chat product's invoice is rarely the price of a model. It is that a conversation costs several times what the same number of messages costs when they are sent independently.

The cause is a single property of the API: it is stateless. The model does not remember the previous turn. To continue a conversation you resend the whole conversation, every time. So the tenth message in a thread does not cost what the first one cost — it costs roughly ten times as much, because it carries nine messages of history in front of it.

That turns a cost you probably budgeted as linear into one that grows with the square of the length.

The arithmetic#

Take a realistic support assistant: an 800-token system prompt, user messages averaging 100 tokens, assistant replies averaging 400.

Turn one sends 900 input tokens — the system prompt plus the first question — and returns 400. Turn two sends those 900, plus the 400-token answer, plus the new 100-token question: 1,400. Turn three sends 1,900. Each turn adds the previous exchange permanently to every turn that follows.

Sum that across a 50-turn conversation and the total input is 657,500 tokens for 5,000 tokens of actual questions. The output total is 20,000. On GPT-4o at $2.50 per million in and $10 per million out, the conversation costs about $1.84.

Now price it the way most estimates are built — one turn's cost multiplied by the number of turns. A turn is 900 in and 400 out, which is $0.00625, so fifty turns is $0.31. The real bill is close to six times that estimate.

The gap widens as the thread does. Strip out the system prompt and the pattern is exactly the triangular numbers: a 50-turn conversation is not 50 units of history, it is 1 + 2 + 3 + … + 50 = 1,275 units. Double the length and you roughly quadruple the cost.

The last turn is the expensive one#

Averages hide where the money goes. In that same conversation, turn one sends 900 input tokens and turn fifty sends 25,400 — a 28-fold difference between two requests that look identical in your logs and cost wildly different amounts.

This matters for two reasons beyond the total.

The first is latency. Input tokens are processed before the first output token appears, so a long thread gets slower in exactly the way that feels like a broken product, and it does so gradually enough that nobody files a bug.

The second is that per-request averages become useless for capacity planning. If your monitoring reports mean tokens per request, a population of short conversations will mask a small number of very long ones that dominate the bill. Track the distribution, not the mean.

"Just use a bigger context window" makes it worse#

The instinct when a conversation gets long is to reach for a model with more room. GPT-4o holds 128,000 tokens; Claude Sonnet 5 holds 1,000,000.

A bigger window does not reduce what you resend. It raises the ceiling on how much you are allowed to resend, which removes the only thing that was capping the growth.

In the example above, the conversation hits GPT-4o's window at roughly turn 250 — and by then a single turn is sending about 125,000 input tokens. On a million-token window you can keep going to roughly turn 2,000, at which point one turn costs around $3 at Sonnet 5's $3 per million. The window did not solve the cost problem. It postponed the error message that would have told you about it.

Some models add a further step: past a threshold, the provider bills the entire request at an elevated rate. A conversation that quietly crosses that line does not get gradually more expensive, it doubles.

Three ways to bend the curve#

None of these are exotic. The reason threads get expensive is usually that nobody chose between them.

A sliding window. Keep the system prompt and the last N exchanges; drop the rest. Cost becomes flat per turn instead of growing, which is the whole point. The failure mode is obvious and worth naming: the assistant forgets what it was told on turn three. This is right for task-shaped conversations — a checkout flow, a form-filling assistant — and wrong for anything where early context stays relevant.

A running summary. Periodically compress the older half of the history into a paragraph and carry that forward instead of the transcript. Cost grows very slowly, and unlike a sliding window, information from early turns survives in some form. The trade is that summarisation is itself a model call, and that the summary is lossy in ways you cannot predict — the detail it drops is occasionally the one that mattered.

Retrieval over the transcript. Store every turn, and at each new turn fetch only the handful of previous exchanges relevant to what was just asked. This is the most accurate and the most machinery. It pays for itself in long-lived threads — an assistant a user returns to for weeks — and is over-engineering for a conversation that ends in ten minutes.

There is a fourth lever that is not a substitute for these but composes with all of them: prompt caching. It does not reduce the number of tokens you resend, it reduces the price of the ones that were identical last time. Because the growing part of a conversation is a stable prefix with a new turn appended, it is close to the ideal shape for a cache — provided the requests arrive close enough together that the cache is still warm.

What to do this week#

Two measurements will tell you whether any of this applies to you.

First, find the 95th-percentile conversation length in your product — not the average. If the tail is short, this whole problem is theoretical for you and you should spend the effort elsewhere.

Second, take a real transcript from that tail, paste the full history into the calculator as it would be sent on the final turn, and look at the number. That is the cost of one request at the end of a long thread. Multiply out the triangle if you want the total, but the single figure is usually enough to settle the argument about whether to build the summariser.

The general rule holds regardless: in a stateless API, you are not billed for what the user typed. You are billed for everything they have ever typed, once per turn.