About Token Budget
Token Budget covers one thing: deciding what gets into the context window, and what doesn’t.
Not how to split documents. Not how to score retrieval candidates. Not whether to build retrieval at all. The allocation problem — you have a finite window, more things want in than will fit, and you have to choose.
Why this is a whole site
Because the interesting failure isn’t the one that errors.
When you exceed a model’s limit, you get a clear message and you fix it. That’s a bad afternoon. The expensive failures are quieter:
- You add more retrieved passages, and answers get worse, because the useful one is now buried among nine mediocre ones.
- Your bill triples, and nobody can say which part of the prompt caused it, because nobody measured the split.
- The model ignores an instruction that’s definitely in the prompt, because of where in the prompt it is.
- Conversation history slowly crowds out retrieval over a long session, and quality degrades in a way that looks random.
Every one of those is an allocation problem. None of them raises an error. All of them are fixable once you can see where the budget is going, which is why nearly every post here has a table in it.
What we think
Reserve output space first. On most models input and output share the window. A budget that allocates the whole thing to input either fails or truncates the answer mid-sentence. This is the single most common real-world budgeting mistake and it’s entirely avoidable.
More context is not more better. Past some point — model- and task-dependent — additional retrieved passages dilute rather than help. Precision in what you include beats volume, and the only way to find your point is to measure.
Position matters, and you can’t fully fix it with wording. Where something sits in the prompt affects whether the model uses it. The phenomenon is widely documented, its magnitude varies by model and task, and newer models handle it differently — so the practical response is structural (put critical material where it gets attention) rather than rhetorical.
Cut history before you cut retrieval. Not always, but as a default. Retrieved passages were selected because they’re relevant to this question. Turn six of a conversation usually isn’t. Most people default the other way round.
On numbers
We don’t publish context window sizes or per-token prices.
Both change with every model release. A post that names a specific limit is wrong within months, and a reader who trusts it builds against a number that no longer holds. What you get instead is the formula — which quantities to add up, what to reserve, what to divide by — with clearly labelled hypothetical values in the worked examples. Substitute yours and the arithmetic holds.
Same discipline on position effects, which is the most tempting place on a site like this to invent a statistic. You will not find “accuracy drops 23% in the middle of the context” here, because that number would depend on the model, the task, the prompt structure, and the year. The phenomenon gets described qualitatively and honestly, including the part where it’s getting better.
And a standing caveat that applies to everything: tokens are not words or characters. Every rule of thumb about ratios is approximate and model-specific. The only reliable count comes from running the model’s own tokenizer, which is cheap and which most people skip.
Where the boundary is
This site assumes you already have candidate content and a window to fit it in. How documents get split into chunks in the first place, and how candidates get scored and selected, are genuinely different problems with different readers, and they aren’t covered here.