Fixed shares or priority order?
History is 60% of the prompt on turn twenty and retrieval has been squeezed to two passages. Nobody decided that. The assembly code appends history first and fits passages into whatever remains, and that implementation detail became your allocation policy.
There are two coherent ways to divide a window between components. Most systems have neither, and the choice between them is worth making explicitly.
Priority order: fill in sequence
Rank the components, then give each one everything it wants until the space runs out.
remaining = available_after_output_reservation − fixed_costs
for component in [passages, recent_history, older_history]:
take = min(component.wants, remaining)
remaining -= takeHypothetical numbers, retrieval prioritised over history, 9,000 tokens available after fixed costs and the output reservation:
| Component | Wants | Gets | Note |
|---|---|---|---|
| Retrieved passages | 6,000 | 6,000 | full allocation |
| Recent history (4 turns) | 2,400 | 2,400 | full allocation |
| Older history | 5,000 | 600 | truncated |
What it is good at: the thing you said matters most always gets what it needs. Simple to implement, easy to explain, degrades in a predictable direction.
Where it fails: the top-priority component can starve everything below it. If retrieval asks for 9,000 tokens on a broad question, history gets nothing at all — including the turn where the user said “in euros, not dollars.” Priority order has no notion of a floor.
Fixed shares: give each a percentage
Assign each component a share of the available space and fit it into that share, regardless of what the others want.
history_budget = available × 0.30
passage_budget = available × 0.55
static_budget = available × 0.15| Component | Share | Tokens | Wants | Result |
|---|---|---|---|---|
| Retrieved passages | 55% | 4,950 | 6,000 | trimmed |
| History | 30% | 2,700 | 7,400 | trimmed |
| Static reference block | 15% | 1,350 | 1,350 | fits |
What it is good at: nothing starves. Every component has a guaranteed floor, so a conversation always keeps some history and a query always gets some passages. Costs are predictable, which matters if you are forecasting spend.
Where it fails: it wastes space. A short conversation cannot use its 30%, and under fixed shares that space is simply unused while retrieval is being trimmed. Rigid shares are a budget that ignores demand.
What to actually build: floors, then priority
The useful policy is a hybrid, and it is barely more code than either half.
1. Reserve output. Non-negotiable.
2. Fixed costs must fit, or fail loudly — that is a configuration bug.
3. Give every component its FLOOR (a minimum share).
4. Distribute what remains in PRIORITY ORDER.
5. Return unclaimed floor tokens to the pool before step 4 completes.Worked through, 9,000 available:
| Component | Floor | Wants | Floor grant | Surplus grant | Final |
|---|---|---|---|---|---|
| Retrieved passages | 2,000 | 6,000 | 2,000 | 4,000 | 6,000 |
| Recent history | 1,200 | 2,400 | 1,200 | 1,200 | 2,400 |
| Older history | 0 | 5,000 | 0 | 600 | 600 |
| Total | 3,200 | 9,000 |
Step 5 is what makes it better than fixed shares: a two-turn conversation cannot use its history floor, so those tokens go back to the pool and retrieval gets them. Demand-responsive, with guarantees.
Choosing the floors is the judgement call. Ask what each component’s minimum useful amount is: one passage is often worth more than zero; the current turn plus the pinned first turn is usually the irreducible history. Anything below the floor should be treated as a failed request rather than a degraded one — a “no room” error is more honest than an answer built on one truncated passage.
One profile per request type
The second half of this, and the part that most improves real systems: a single budget for a whole application is wrong, because different requests need different shapes.
Hypothetical shares, same window, three routes:
| Request type | Passages | History | Output reservation |
|---|---|---|---|
| Single-shot document question | 70% | 5% | 25% |
| Multi-turn conversational | 40% | 40% | 20% |
| Summarisation of a supplied text | 80% | 0% | 20% |
| Agent step with tool results | 25% | 55% | 20% |
The agent row is the interesting one: most of its window goes to prior tool results and actions, and a policy tuned for document Q&A will cut exactly the wrong thing there.
Implementation is a lookup, not a rewrite: classify the request — by endpoint, route, task flag, or a cheap classifier — and select a named profile. Keep the profiles in one place, in data rather than in branching code, so they can be listed, reviewed and diffed.
Two cautions. Profiles proliferate if you let them; four or five is a system, twenty is a maze. And a misclassified request gets the wrong budget, so make the default profile the safest one rather than the most aggressive.
What it costs
A configuration surface that can be wrong. Floors and shares are now numbers someone has to maintain. Mitigate by asserting in a test that floors plus fixed costs plus the output reservation fit within the smallest limit you support — otherwise the failure appears in production as an unfittable request.
Non-uniform behaviour. Two similar-looking requests can be allocated differently because they classified differently. Log the chosen profile with every request or this becomes very hard to debug.
More things to tune. Which is only worth it above a certain scale. If your window is comfortable and nothing ever gets cut, write down the priority order and stop there.
What to measure
- Realised share per component, not configured share. Mean and p95. The gap between intended and actual allocation is where the surprises are.
- Floor-breach rate. How often a component could not get its minimum. Should be near zero; if it isn’t, your floors or your fixed costs are wrong.
- Surplus distribution. Who receives the unclaimed tokens, and how often. If one component always takes the surplus, the priority order is doing the real work and the floors are decoration.
- Profile selection counts, and quality per profile. A profile that never fires should be deleted; a profile with worse quality is a tuning target.
- Cut frequency per profile. From the cut order: if cutting is routine rather than exceptional for a profile, that profile is misbudgeted rather than under pressure.
Whichever mechanism you choose, the requirement is that it is one function, in one place, with the numbers visible. The alternative is not the absence of a policy — it is a policy nobody chose, distributed across the code that happens to assemble the prompt. Start from the budget and write the shares down.