Token Budget

When compaction collides with your cache

Compaction fires on turn nine. It reclaims 4,000 tokens of history, and the next request costs more than the one before it — because rewriting the middle of the transcript invalidated a cached prefix that had been serving every turn since turn one.

Both mechanisms are correct in isolation. Together they conflict, and the resolution is a threshold you choose deliberately rather than a default that fires whenever history feels large.

Why they conflict

Prompt caching rewards an unchanging prefix. A conversation is naturally cache-friendly: turn n’s prompt is turn n−1’s prompt with new content appended, so the prefix grows monotonically and each turn reuses everything before it.

Compaction breaks exactly that property. Replacing turns 2–8 with a summary means everything from turn 2 onward is different, so the reusable portion collapses back to whatever sits above the summary — usually just the system prompt and tools.

Turn History tokens Prefix reusable from previous turn
7 4,200 system + tools + turns 1–6
8 4,900 system + tools + turns 1–7
9 (compaction fires) 1,100 system + tools only
10 1,700 system + tools + summary + turn 9

Hypothetical numbers. The token saving at turn 9 is real: 3,800 tokens of history reclaimed. So is the loss: the prompt is now largely uncached, and it takes several turns of appending before the prefix is worth much again.

Cost the decision instead of guessing

The comparison is between one expensive turn and the turns of savings that follow.

compaction_cost  = uncached_tokens_at_next_turn × (input_price − cached_price)
                 + summarisation_call_cost
                 + cache_write_cost

compaction_saving = tokens_reclaimed
                  × (input_price − cached_price)      # per subsequent turn
                  × turns_remaining_in_conversation

Substitute your provider’s unit prices — the comparison depends on the ratio between cached and uncached input rates, not on any absolute figure.

The term that decides it is turns_remaining_in_conversation, and you do not know it. Which gives the rule that matters:

Compaction pays off in long conversations and loses in short ones. A conversation that ends two turns after compaction fired paid for nothing. One that continues for twenty turns amortises it easily.

Choose the threshold, and add hysteresis

Three parameters, and most systems set only the first.

A high-water mark. History tokens, not turn count — turns vary enormously in size. Set it where history genuinely threatens the other components, not where it merely looks large. A useful anchor: fire when history exceeds its allocated share of the budget, not before.

A low-water mark. Compact down to well below the trigger, not just under it. Compacting to just-under-threshold means the next two turns push you over again and you compact repeatedly, paying the cache and summarisation cost each time. Reclaim generously: down to perhaps half the high-water mark.

if history_tokens > HIGH_WATER:
    compact until history_tokens <= LOW_WATER      # LOW_WATER ≈ HIGH_WATER / 2

A minimum interval. No more than one compaction every N turns regardless of size, so a burst of large tool results cannot trigger three compactions in three turns.

Together these turn compaction from a per-turn hazard into a rare event, which is what the cache arithmetic wants.

Placement makes the loss smaller

If compaction is going to rewrite part of the prompt, control which part.

Put the summary as low as you can. Content above the rewrite point survives. If the summary sits immediately above the recent turns and below everything static, then the system prompt, tool block and any static reference block are still cached — see designing a prompt prefix that stays stable.

Keep the summary append-only if your design allows it. Rather than regenerating one summary each time, emit summaries as successive blocks: summary_of_turns_2_8, then later summary_of_turns_9_15. Each is written once and never edited, so the prefix keeps growing monotonically and compaction stops invalidating anything. This costs some redundancy and a little more structure, and it is the single most effective fix available here.

Never rewrite the pinned head. The first turn should be byte-identical for the life of the conversation. It is both the most useful history to keep and the cheapest to keep cached.

Extract state into a block you overwrite as rarely as possible. Session state is more compact than the turns it replaces, but if you rewrite it every turn it is volatile content sitting high in the prompt. Update it on change, not on schedule.

Do it out of the request path

Whatever the threshold, the user should not wait for a summarisation call.

Compact after a turn completes, asynchronously, so the work lands before the next request rather than inside it. Then the latency cost disappears and only the token arithmetic remains.

This also makes the append-only approach easier: you have time to summarise from the original turns rather than from the previous summary, which is what stops summary quality degrading over a long conversation. The mechanics are in context budgets for multi-turn conversations.

What it costs

Detail loss, at a moment you chose. Compaction is lossy compression of the conversation. Firing it less often — which the cache arithmetic wants — means it fires later, with more turns to compress at once, which is a larger single loss. Structured summaries limit the damage; they do not eliminate it.

A bimodal cost distribution. Most turns cheap, occasional turns expensive. Fine for cost, awkward for latency percentiles if you compact synchronously. Another reason not to.

Complexity in the assembly path. High and low water marks, an interval, append-only summary blocks and a state extraction step is more machinery than a sliding window. It is justified when conversations are long and volume is high, and over-engineered when they are not. If your conversations rarely exceed a handful of turns, do not build this.

What to measure

  • Compactions per conversation. Should be a small number. Several per conversation means the water marks are too close together.
  • Cache hit rate by turn number. The graph that shows the collision. Expect a dip at compaction turns and a recovery after; if it never recovers, something above the summary is being rewritten too.
  • Tokens reclaimed per compaction, against tokens re-paid uncached on the following turn. This is the actual trade, per event, and logging both makes it arguable rather than theoretical.
  • Turns after compaction, distributed. If the median is one or two, compaction is firing too early in the conversation lifecycle and you should raise the high-water mark.
  • Answer quality by turn number, before and after any threshold change — the check that the token savings did not come out of the conversation.
cachinghistory