Size the budget for the tail, not the average

Your mean prompt is comfortably inside the limit. Overflow happens anyway, on a small percentage of requests, and each time someone trims something and the alert clears.

A budget built on averages is a budget that fails on the requests that differ from average — which is the definition of the requests that fail. Size it at the tail instead, and the overflow handler becomes an exception rather than a routine.

Averages hide the whole problem

Log per-component token counts and look at the distribution, not the mean. Hypothetical numbers:

Component	Median	p95	p99	Max observed
System prompt	900	900	900	900
Tool definitions	2,400	2,400	2,400	2,400
Conversation history	1,800	6,900	11,400	18,200
Retrieved passages	3,600	5,100	9,800	24,600
Output reservation	2,000	2,000	2,000	2,000
Total	10,700	17,300	26,500	—

Two things this table says that a mean cannot.

The fixed components have no tail. Their p99 equals their median. They are not your problem, however much attention they get.

The variable components have enormous tails. History p99 is over six times its median; the passage maximum is nearly seven times its median. That spread is where every overflow lives, and it is invisible in an average.

Note also that you cannot add the p99 column and call it a p99 request. Component tails are not independent — a long conversation and a broad question often arrive together — so the honest number is the p99 of the total, measured directly.

Where the tail comes from

Five causes, all specific and all fixable:

One oversized retrieved item. A source that is far larger than typical — an appendix, a merged document, a transcript. Unremarkable as a source; ruinous when it arrives in the window as one oversized item and takes the whole passage share with it.

A long conversation. History grows monotonically, so the tail of your history distribution is just your longest sessions. Every system has some.

An agent turn with a large tool result. A query that returned many rows, a file that was bigger than expected. Typically the single largest tail contributor on agentic systems.

Content that fragments badly. Base64 blobs, minified code, dense tables, long identifiers, non-Latin text. The same character count can be far more tokens, so a prose-calibrated estimate is worst exactly where the request is largest.

A user pasting something. The most reliable source of outliers in any application with a text box.

Fix the tail at its source, then budget for what remains

In order of effectiveness:

Cap what any single item can contribute. A hard per-item token ceiling at assembly time, applied before fitting. One item should never be permitted to consume the whole passage share — that is a guaranteed bad allocation regardless of how relevant the item is.

per_item_cap = passage_budget / min_items_wanted

Cap tool results at the tool. Make the ceiling part of the tool’s contract rather than a cleanup step afterwards. A search tool that returns a bounded number of bounded results cannot produce a tail.

Cap the input the user can supply, and say so in the interface. An explicit limit is better than a silent truncation, and much better than an error the user cannot interpret.

Bound history in tokens, not turns. Turn counts treat a 200-token exchange and a 4,000-token tool result as equivalent. See context budgets for multi-turn conversations.

Then set floors and shares against the p99, not the median, so the allocation is one that survives a tail request rather than one that only works on typical traffic. The mechanism is in fixed shares or priority order.

Test with a tail request, not a happy path

The commonest testing gap on this whole site: every fixture is average-sized.

Build these cases and keep them:

A p99-sized request, assembled from real logged component sizes.
A single oversized item, larger than the entire passage share on its own.
A long conversation — longer than your longest real session.
A badly fragmenting payload: a large base64 string, a dense table, a non-Latin document.
A fallback-model version of the p99 request. If your fallback has a smaller window, this is the case that fails during an incident, which is the worst time to discover it.

Each should produce a deliberate, logged, correct outcome — a clean cut, or an explicit refusal. Not an exception, and not a truncated answer.

What tail-sizing costs

Unused space most of the time. Reserving for p99 means the median request leaves window on the table. That is the trade, and it is usually the right one: unused window costs nothing on a per-token bill, while an overflow costs a failed or degraded answer.

Where it genuinely costs something is when you have set static shares that starve a component on typical requests to protect against rare ones. The fix is the floors-then-surplus mechanism rather than rigid shares — unclaimed space gets returned, so the median request still uses the window fully.

More machinery. Per-item caps, tool contracts, and distribution logging is real work. Skip it if you never overflow. If you cut on more than a few percent of requests, it is cheaper than the incidents.

What to measure

Total prompt tokens at p50, p95, p99 and max. The headline. If you track one thing from this post, track the p99, because that is the number your limit has to accommodate.
Per-component distributions, so the tail has an address.
Largest single item per request, distributed. Reveals the oversized-item problem immediately.
Cut frequency and cut size. Frequency tells you whether overflow is exceptional; size tells you how much information a cut request lost.
Overflow-to-refusal ratio. Of requests that could not fit, how many degraded gracefully versus how many failed. Both are acceptable outcomes; a truncated answer is not.
The gap between your count and the provider’s, at the tail specifically. Calibration error is worst on unusual content, so the p99 request is where your estimate is least trustworthy — which is why the safety margin exists, per counting tokens before you spend them.

The reframing

Overflow is not a runtime accident to be handled; it is a budget that was sized against the wrong statistic. Move the target from the mean to the p99, cap the contributors that create the tail, and keep a tail-sized fixture in your test suite. The cut order should be the thing that almost never runs.