Reserving room for the answer
The answer stopped mid-sentence. Not an error — a 200, a response, a user staring at half a paragraph that ended after the preamble and before the part they asked for.
You had room for everything except the thing you were generating. On most models input and output draw on the same window, so the answer is a line item in the budget and it has to be reserved before anything else is allocated.
The reservation goes first
The order of operations is the whole technique:
reserved = max_output_tokens + safety_margin
available = documented_limit − reserved
fixed = system_prompt + tool_definitions
remaining = available − fixed
history = fit(history, remaining × history_share)
passages = fit(passages, remaining − history)Retrieval is the residual. Most systems invert this — fill the window with passages, then discover what is left for the response — and the symptom is the truncated answer above.
Hypothetical numbers. A budget with the reservation taken first:
| Component | Tokens | Share |
|---|---|---|
| System prompt | 800 | 6% |
| Tool definitions | 1,600 | 12% |
| Conversation history | 2,600 | 20% |
| Retrieved passages | 5,400 | 41% |
| Output reservation | 2,400 | 18% |
| Safety margin | 400 | 3% |
| Total | 13,200 | 100% |
The two bottom rows are the ones that get borrowed against under pressure, and they are the two that should never be.
Sizing the reservation from data
Guessing produces a number that is simultaneously too small for your longest answers and too large for everything else. Measure instead.
Log completion token counts for real traffic, then look at the distribution rather than the mean:
| Statistic | Output tokens | What it tells you |
|---|---|---|
| Median | 240 | typical answer |
| p90 | 700 | routine long answer |
| p99 | 1,900 | the long tail |
| Max observed | 3,100 | the reservation floor |
Reserve at roughly the p99, not the median. The median sizes a budget that truncates one answer in ten, which is a defect rate nobody would accept if it were reported as one.
Then decide deliberately what happens above the reservation. Silent truncation is the wrong answer. Better options: detect the stop reason and continue generation in a second request, or shorten the required format so the answer fits, or return an explicit “response too long” state that your interface can handle.
Check the stop reason on every response. Providers report whether generation ended because the model finished or because it hit a cap. Systems that ignore that field cannot distinguish a complete answer from a severed one, and neither can their dashboards.
The unset cap is not “no cap”
If you do not set a maximum output length, one of two things is true, and both are worth knowing about.
Either the provider applies a default cap — which you have now delegated to a value you did not choose and that can change between model versions — or generation is bounded only by the remaining window, meaning your output allowance is whatever your input happened to leave. That makes response capacity a function of how much history accumulated, which is the opposite of a budget.
Set it explicitly. A number you chose can be measured, tuned, and alerted on.
There is a related trap on models that emit intermediate reasoning tokens before their visible answer: those tokens are generated, and generally counted, even though the user never sees them. A reservation sized only for the visible answer under-reserves. That case has its own post — budgeting for tokens you never see.
The safety margin is not padding
The margin covers the gap between your count and the provider’s, which is never zero. Message role markers, delimiters, serialised tool schemas and tool-call structure all add tokens that summing your message strings will not see.
A few percent is prudent, and it should shrink as your calibration improves rather than staying a round number forever. Deriving it is covered in counting tokens before you spend them.
Where the margin earns its place is the tail: the request carrying an unusually fragmenting payload, where your estimate is furthest from the truth. That is exactly the request you cannot afford to have fail.
Why borrowing from the reservation is always wrong
It is the most tempting cut available, because it makes the immediate error disappear. A rejected request becomes an accepted one. Something got returned.
What you shipped is a worse failure than the error. A rejection is visible, logged, alerting, and obviously broken. A truncated answer looks like a response: it renders, it reads fluently for two sentences, it appears in your success metrics, and the only person who knows it failed is the user.
Worse, truncation lands preferentially on the answers that were hardest — the ones needing the most explanation, the multi-part questions, the tables. Your quality regression is concentrated in the cases you most wanted to get right.
If the input does not fit alongside a proper reservation, cut input. That is what the cut order exists for.
Output is a cost decision too
No prices here, because they change, but two structural facts:
cost_per_request = (input_tokens × input_unit_price)
+ (output_tokens × output_unit_price)Output is typically priced higher per token than input, which means a verbose response format can cost more than a substantial prompt. And output is generated serially, so it usually dominates latency in a way input does not.
Both point the same way: a shorter required answer is often the cheapest quality-neutral optimisation available, and it frees window space at the same time. Which is a formatting decision as much as a budgeting one — see the output format is a budget decision.
What to measure
- Stop reason distribution. The share of responses that ended at the cap. This is your truncation rate and it should be near zero. If you log nothing else from this post, log this.
- Output token distribution, with p90/p99, per request type. Reservations should be per type; a summarisation endpoint and a yes/no classifier do not need the same allowance.
- Reservation utilisation. Actual output over reserved output. Consistently low means you are holding space that retrieval could use; occasionally at 100% means you are truncating.
- Continuation rate, if you implemented continuation. A rising rate means the reservation or the format needs revisiting.
The rule
Reserve first, from measured p99, per request type, explicitly capped, with the stop reason checked on every response — and never lend the reservation to input, however much easier that makes the error go away.