Rebudgeting when you change model
The swap looked like a configuration change: new model identifier, same prompt, same retrieval, ship it. A week later the truncation rate is up, the bill moved in a direction nobody predicted, and answer quality is different in a way that is hard to name.
A context budget is model-specific. Almost every number in it is a property of the model you derived it against, and a model change invalidates the numbers rather than carrying them forward.
What changes underneath you
Six things, and only the first is obvious.
The documented limit. Check it, and check whether it is a single total or separate input and output limits — that difference changes the arithmetic of your reservation entirely.
The tokenizer. Different model families use different vocabularies, so the same text is a different number of tokens. Your counting code must be pinned per model, and a count derived for one tells you little about another. This is the error that silently invalidates every table you have.
The maximum output length. Often a separate cap from the window, often different across models, sometimes different from the default your client library sends. A reservation tuned to one model’s cap may be unachievable or wastefully large on another.
Whether there is hidden output. If the new model produces an internal reasoning pass and the old one did not, your output reservation is now too small in a way that only shows up on hard requests. See budgeting for tokens you never see.
Positional behaviour. How reliably a model uses material in the middle of a long input varies by model, and newer models generally handle it better than older ones — but the magnitude is model- and task-specific and there is no transferable number. A layout tuned for one model is a hypothesis about the next one.
Caching semantics. Whether prefix caching applies, what the minimum cacheable length is, how entries expire, and the ratio between cached and uncached input pricing. All of that feeds the trade-offs in designing a prompt prefix that stays stable, and none of it necessarily carries across.
Re-derive the table, do not scale it
The temptation is to multiply everything by a ratio. Don’t — the components do not scale together, because a tokenizer change affects prose, code and structured text differently.
Instead, re-count each component against the new tokenizer and rebuild the table. Hypothetical numbers, the same application on two models:
| Component | Model A | Model B | Change |
|---|---|---|---|
| System prompt | 900 | 960 | +7% |
| Tool definitions | 2,400 | 2,760 | +15% |
| Conversation history (10 turns) | 3,200 | 3,300 | +3% |
| Retrieved passages (8) | 4,000 | 4,600 | +15% |
| Output reservation | 2,000 | 2,000 | set by you |
| Total | 12,500 | 13,620 | +9% |
Note the shape: the tool block and the passages moved most, because both contain structured text and punctuation that fragments differently. The prose-heavy system prompt barely moved. A flat 9% multiplier would have under-counted the two components that matter most and over-counted the one that doesn’t.
Also re-derive your per-message framing overhead, which is provider- and version-specific: assemble a representative request, sum your own counts, and compare against the input token count the API reports. The method is in counting tokens before you spend them.
The checklist
Work through it before the swap, not after.
1. Read the new model's documented limit. Note whether input and
output share it or are capped separately.
2. Pin the new tokenizer alongside the model version.
3. Re-count every component. Rebuild the budget table.
4. Re-derive per-message framing overhead against reported counts.
5. Re-measure the output distribution on real traffic; reset the
reservation at p99, including any hidden reasoning output.
6. Re-run the fixed-cost assertions: system + tools + floors +
reservation must fit, with margin.
7. Re-run the layout experiment: passage order, instruction
placement, history position. Do not assume the old winner wins.
8. Re-run the few-shot ablation. Examples are the most
model-dependent part of a prompt.
9. Re-check caching: hit rate, minimum cacheable size, price ratio.
10. Widen the safety margin for the first week, then tighten it
once your counts are calibrated against reality.Steps 7 and 8 are the ones that get skipped because they need an evaluation set. They are also where the quality surprises come from. If you have no evaluation set, a model change is the moment building one becomes unavoidable.
Supporting several models at once
If you route between models, the budget is per model and there is no single answer.
Keep limits, caps and tokenizers in one table keyed by model. Not scattered across call sites. The assembly path takes the model as an argument and looks the numbers up.
Assemble per model, or assemble for the smallest. Two strategies. Fitting the prompt separately for each model uses each window fully but means the same question yields different prompts. Building one prompt that fits your smallest window is simpler, reproducible, and wastes space on the larger models. Pick deliberately; the second is a reasonable default for fallback paths, the first for routed production traffic.
Guard the fallback. A fallback model with a smaller window is a request that fits your primary and fails your backup — at the worst possible moment, because the fallback fires during an incident. Test the fallback path against a p99-sized request, not an average one.
What it costs
Rebudgeting is a day of work with no visible feature at the end, which is why it does not happen. The cost of skipping it is not a crash; it is a system that is slightly wrong in several places at once — over-reserved here, truncating there, cutting passages on requests that used to fit — and no single symptom points at the model change.
There is also a real cost to over-caution. Widening every margin “because the model is new” leaves window unused and passages cut. Widen for a week, measure, then tighten with data.
What to measure through the transition
- Your count versus the provider’s reported input tokens, per request. The single most valuable metric here: a drift means the tokenizer or framing assumption is wrong.
- Truncation rate, from the stop reason. Should not move. If it does, the reservation is wrong.
- Cut frequency by component, before and after. A jump means the new counts broke your shares — see fixed shares or priority order.
- Realised share per component. The proportions are what you tuned against; if they moved, your tuning no longer applies.
- Quality on a fixed evaluation set, held constant across the swap. Everything else is inference.
Run the old and new model side by side on shadow traffic if you can afford it. It converts all of the above from a post-deploy investigation into a pre-deploy comparison, and it is the only way to distinguish “the model is different” from “our budget for the model is wrong.”