Context budgets for multi-turn conversations

Turn three is fine. Turn thirty is not.

Conversation history is the only component of the budget that grows on its own. Everything else is either fixed or re-selected per request. History accumulates, and without a policy it eventually consumes the space that retrieval and the answer needed.

The characteristic symptom is a system that works well in testing — where nobody has a forty-turn conversation — and degrades in production in a way that looks random until you plot quality against turn number.

The growth problem

Each turn adds the user’s message, the assistant’s response, and on agentic systems any tool calls and their results. That last category is usually the largest.

All numbers hypothetical. Suppose an average turn adds 600 tokens of conversation and your non-history budget is 9,000 tokens:

Turn	History	Remaining for retrieval + output
5	3,000	6,000
10	6,000	3,000
15	9,000	0
20	12,000	overflow

Notice what happens before the overflow. At turn 10 the retrieval budget has halved, so the system is answering with half as many passages — silently, with no error, and probably worse. The quality degradation starts long before the failure, and only the failure is visible.

On an agent loop with tool calls returning JSON, an average turn can be several times larger, and this table compresses into a handful of turns.

Policies, worst to best

Unbounded. Keep everything until it breaks. Common as an accident, never as a decision.

Sliding window. Keep the last N turns. Simple, predictable, and it has one serious flaw: it drops the first turn, which frequently contains the task framing, the document under discussion, or the constraints everything since has assumed. A conversation that loses its opening keeps the small talk and discards the point.

Sliding window plus pinned head. Always keep the first turn (or first few), plus the last N. Fixes the main flaw for one extra line of code. This is the minimum acceptable policy and where you should start.

Token-budgeted window. Keep the most recent turns that fit a token allowance, rather than a fixed count. Better, because turns vary enormously in size — one tool call returning a large payload can exceed ten conversational turns. Counting turns treats those as equivalent; counting tokens doesn’t.

Summarised middle. Keep the head, keep the last N turns whole, replace everything between with a generated summary. The best general-purpose policy for long conversations, and the one worth building.

Summarising the middle

The technique that actually solves this, with real costs.

The mechanics:

When history exceeds its allowance, take the turns between the pinned head and the recent window.
Generate a compact summary of them.
Replace those turns with the summary, marked clearly as a summary.
Continue. When it exceeds again, extend the summary with the newly-aged turns.

What it costs:

An extra model call, adding latency and cost when it fires. Mitigate by summarising asynchronously after a turn completes rather than in the request path, so the user never waits for it.

Lossy compression. The summary is model-generated and will lose detail. Some of that detail may have mattered.

Compounding error, if you’re careless. Summarising a summary repeatedly degrades badly — each pass distorts a little, and the distortions accumulate. Where you can, summarise from the original turns rather than from the previous summary. That requires retaining the originals outside the prompt, which is cheap: keep the full transcript in your own storage and treat the prompt as a rendered view of it. This one design choice makes most of the difficulty go away.

Structure beats prose. A summary as free-flowing narrative loses specifics. A structured one retains them:

## Established
- Working on the Q3 reconciliation report
- Output format: CSV, comma-separated, header row
- Excluding the Berlin office per user instruction

## Decided
- Use the revised exchange rates, not the booking-date rates

## Open
- Whether to include pending transactions

That’s compact, and it preserves precisely the things whose loss would break the conversation.

Extract state instead of remembering it

The most effective change, and it’s architectural rather than a tuning parameter.

Do not keep task state in the transcript.

If the conversation established a file, an account, a format, a set of constraints — extract those into a structured record maintained outside the message list, and render that record into the prompt as a compact block.

[system prompt]
[tools]

## Session state
file: 2026-q3-recon.csv
format: CSV, header row, comma-separated
excluded: Berlin office
rates: revised (not booking-date)

[recent turns]
[instructions]
[query]

Why this changes everything: once state lives outside the transcript, history becomes safe to cut. The reason people fear truncating history is that it’s load-bearing. Remove that dependency and the whole problem softens — you can be aggressive with history, and the things that actually matter are preserved exactly rather than approximately.

It also makes state inspectable, testable, and correctable, none of which is true of state that exists only as an implication of a message sequence.

The cost is that something has to maintain the record — either a tool the model calls to update it, or an extraction step after each turn. That’s real work, and it’s the right work.

Agent loops overflow fastest

Worth separating out, because the arithmetic is different.

An agent taking actions generates tool calls and tool results, and results are frequently large: search results, file contents, API responses, database rows. A single tool call returning 4,000 tokens of JSON, from which the model needed one field, is common.

Left in the transcript, these dominate. Three fixes, in order of value:

Truncate tool results at the source. Cap what a tool can return, and make the cap part of the tool contract. A search tool returning the top 5 results with 200 tokens each beats one returning 50 results in full. This is a tool design decision and it’s the highest-leverage one available.

Replace old tool results with their outcome. Once the model has used a result, the raw payload is rarely needed again. Replace it with a one-line record: [searched "reconciliation policy" → found policy doc, section 4.2]. Enormous savings, low risk.

Keep only the most recent tool interaction in full. Older ones get the outcome treatment above.

Together these routinely reclaim the majority of an agent conversation’s tokens, and they cost almost no quality — because the model has already extracted what it needed.

Measuring

Instrument by turn number, or none of this is visible:

History tokens by turn. Should plateau under a working policy. If it grows linearly, your policy isn’t firing.
Retrieval tokens by turn. The one to watch. Declining with turn number means history is winning the contest, which is the silent degradation.
Summarisation frequency and latency.
Answer quality by turn number. The metric that matters. If quality at turn 20 is materially worse than at turn 5, your policy is losing something it shouldn’t.

That last one requires an evaluation set with genuinely long conversations, which most teams don’t have — and that’s exactly why this class of problem reaches production. If you build one thing here, build a long-conversation test case.

Recommended defaults

Pin the first turn. Always.
Budget history in tokens, not turns.
Keep the last few turns whole.
Summarise the middle, from originals, structured, asynchronously.
Extract task state out of the transcript into a structured block.
Cap tool results at the tool, and collapse old ones to outcomes.
Instrument everything by turn number.

Steps 5 and 6 do most of the work. The rest is refinement — and once history is bounded, the cut order rarely has to fire at all.