What to cut first when context overflows

The budget doesn’t balance. System prompt, tools, history and retrieved passages add up to more than the window minus your output reservation, and something has to go.

Most systems answer this by accident — whatever the assembly code happens to truncate first. That’s a policy, just an unexamined one, and it’s usually the wrong one.

The default cut order

Start here and deviate deliberately:

1. Old conversation history — oldest turns first
2. Low-ranked retrieved passages — lowest score first
3. Tool definitions not relevant to this request
4. Few-shot examples in the system prompt
5. Recent conversation history
6. The output reservation — never, if you can avoid it

The controversial part is (1) above (2), so that’s worth defending.

Why history usually loses to retrieval

The instinct is the other way round. History feels like state you must preserve; retrieval feels like something you fetched and can fetch again.

But consider what each is for.

Retrieved passages were selected for this specific question. A retriever ran, scored candidates against the current query, and returned the best. Every one of them is there because something judged it relevant to what’s being asked right now.

Conversation history is there because it happened. Turn six of a twenty-turn conversation is in the prompt because of chronology, not because anything assessed its relevance to turn twenty. Often it is genuinely irrelevant — a tangent, a clarification long since resolved, a question that got answered.

So retrieval is selected content and history is accumulated content. When space is short, drop the accumulated one.

The exceptions, which are real:

• The user refers back explicitly. “Like I said earlier” or “use the same format as before” means history is carrying the actual question. Detecting this reliably is hard; a conservative approach is to keep more history when the current turn contains referential language.
• Task state lives in history. If the conversation established constraints — a chosen file, a selected account, a format decision — and that state exists nowhere else, cutting it breaks the task. The real fix is to stop keeping state in the transcript. Extract it into a structured summary that persists outside the message list, and then history becomes safe to cut. Systems that do this handle long conversations far better than systems that don’t.
• The retriever is unreliable. If retrieval frequently returns junk, its selected-ness advantage is theoretical. Fix the retrieval rather than compensating with history.

Cutting history well

Not all truncation is equal.

Drop oldest first. The default, and reasonable. Recency correlates with relevance in most conversations.

Always keep the first turn. Frequently overlooked and frequently important — the opening message often contains the task framing, the constraints, or the document under discussion. A sliding window that discards it drops the point of the conversation while carefully preserving small talk.

Cut in whole turns, never mid-message. A truncated message is worse than an absent one: the model sees a fragment and may treat it as complete. If a single turn is too large to keep whole, summarise it rather than slicing it.

Summarise instead of dropping, past a threshold. Replace the oldest N turns with a compact summary. This is the highest-value technique available for long conversations, and it has a cost worth naming: the summary is generated by a model, so it can lose or distort detail, and errors accumulate if you repeatedly summarise summaries. Mitigations: summarise from the original turns rather than from the previous summary where you can, and keep the summary structured — decisions, constraints, open questions — rather than free prose.

Drop tool-call noise aggressively. Intermediate tool calls and their raw results are often the bulk of an agentic conversation and the least useful part to keep. A tool call that returned 3,000 tokens of JSON, from which the model extracted one fact, can usually be replaced by that fact. This is frequently the single largest saving available and it’s low-risk.

Cutting retrieval well

Drop by score, lowest first. If your retriever produces scores, use them. The eighth-ranked passage is there because you asked for eight, not because it earned its place.

Consider that fewer may be better anyway. Adding passages past some point degrades answer quality — the relevant one competes with mediocre ones for attention. So the cut you’re forced into may be an improvement. This is worth measuring rather than assuming: vary passage count against a fixed evaluation set and see where quality peaks. Many systems are past their peak and don’t know it.

Trim within passages before dropping whole ones, if your chunks are large. A 1,000-token chunk of which 200 tokens are relevant can sometimes be trimmed. This costs a processing step and risks removing context the passage needed to make sense, so it’s a second-line option.

Deduplicate first. Near-duplicate passages are common when a corpus has overlapping documents, and they’re pure waste — the same content occupying two slots. Deduplicating before the cut is free space.

Cutting fixed costs

If you’re regularly cutting into system prompt and tools, the fixed costs are too large and the fix is structural rather than per-request. Still, two runtime options:

Conditional tool loading. Include only tools plausibly relevant to the current request. Needs a routing step and risks omitting a tool the model needed, so it wants a fallback path — but on tool-heavy applications the saving is large and paid on every request.

Conditional few-shot examples. Include examples matching the detected task type rather than all of them. Same trade-off, smaller stakes.

Neither should be a runtime scramble. If they fire often, revisit the budget instead.

Never cut the output reservation

Borrowing from the output allowance to fit more input is always a mistake, and it’s the most tempting one because it makes the immediate error go away.

What you get is a truncated answer. The model starts responding, hits the ceiling, and stops — often mid-sentence, often after the preamble and before the actual answer. From the user’s perspective this is worse than an error, because it looks like a response.

If input genuinely doesn’t fit with a proper output reservation, cut input. That’s what the order above is for.

Degrade visibly

Whatever the policy, the system should know when it fired.

Log on every cut: what was dropped, how much, and which rule triggered. Emit a metric. Then watch two things:

Cut frequency. If cutting happens on 40% of requests, it isn’t an overflow handler any more — it’s your normal operating mode, and it’s operating without anyone having designed it. Rebudget.

What gets dropped. If the same component always loses, either that’s correct and you should reduce it at the source, or your priority order is wrong.

The failure mode this prevents: silent degradation. A system that quietly drops half its retrieved passages under load looks fine on every dashboard and answers noticeably worse, and nobody connects the two because nothing recorded it.

The policy, written down

1. Reserve output. Non-negotiable.
2. Fixed costs (system + tools) must fit. If not → configuration error, fail loudly.
3. Fit retrieval to its share, dropping lowest-scored first, after deduplication.
4. Fit history to what remains:
     - keep the first turn
     - keep the most recent N turns whole
     - summarise the middle
     - drop raw tool-call payloads first
5. Log every cut with component, amount, and rule.
6. Alert if cut frequency exceeds your threshold.

Write it down and put it in one place in the code. The alternative isn’t the absence of a policy — it’s a policy nobody chose, distributed across four functions.