Token Budget

The context window is a budget

A request fails with a context length error. You look at the retrieved passages, trim two of them, and it goes through. Problem solved, in the sense that the symptom stopped.

What you didn’t find out is where the rest of the window went — and on most systems the retrieved passages aren’t the largest line item.

The five claimants

Every request’s window is divided among five things. Writing them down is most of the work.

1. The system prompt. Instructions, persona, output format, constraints, few-shot examples. Fixed per application, so it’s easy to forget it’s there. It’s also the line item that grows without anyone noticing — every edge case someone patches adds a sentence, and nobody ever removes one.

2. Tool definitions. If the model can call tools, their schemas are in the prompt. Every tool, every parameter, every description, on every request, whether or not it gets used. This is routinely the most underestimated item on the list. A dozen tools with thorough JSON Schema descriptions is substantial, and it’s paid on turn one and every turn after.

3. Conversation history. Grows without bound if you let it. On turn two it’s negligible; on turn forty it can dominate everything else.

4. Retrieved content. The passages, documents, or search results you fetched for this query. The part people think of as “the context,” and frequently not the biggest part.

5. The output. The answer the model has to write. On most models input and output share the window, so space for the response has to be reserved from the same total. Skip this and you get either a rejected request or an answer that stops mid-sentence.

That last one is the row people leave off the table, and it’s the one that turns a working system into a mysteriously truncated one.

Write the table

Here’s the exercise. For one real request, count each component and put it in a table.

All numbers below are hypothetical placeholders. Substitute your own — that’s the entire point.

Component Tokens Share
System prompt 900 6%
Tool definitions 2,400 15%
Conversation history 3,200 20%
Retrieved passages (8 × 500) 4,000 25%
Output reservation 2,000 13%
Used 12,500 78%
Headroom 3,500 22%

Three things usually become obvious the first time someone does this:

  • The system prompt and tool definitions together are a large fixed cost — here 21%, paid on every single request forever. They’re also the cheapest to optimise, because you do it once.
  • History and retrieval are the variable costs, and they’re the ones that grow into each other.
  • The headroom is smaller than it felt. 22% sounds comfortable until history grows for another ten turns.

If you have never made this table for your own application, make it before reading further. It reliably surprises people, and it changes which optimisation is worth doing.

Fixed versus variable

The most useful split, because the two need completely different treatment.

Fixed costs — system prompt, tool definitions — are paid identically on every request. Optimising them is a one-time job with a permanent payoff. Worth doing carefully:

  • Delete instructions covering cases that no longer occur. There are always some.
  • Trim tool descriptions to what the model needs to choose correctly, not to what a human reader would want. These are often written like documentation.
  • Cut tools the model never calls. Every unused tool is pure overhead on every request.
  • Load tools conditionally by task, if your framework allows it. A request that can’t possibly need the database tools shouldn’t carry their schemas.
  • Consider whether few-shot examples still earn their space. They’re often added early with a weaker model and never revisited.

Variable costs — history and retrieval — need a policy, not a one-time edit. They grow, and they need rules governing what happens when they collide. That’s what to cut first and multi-turn budgets.

The output reservation

Worth its own section because it’s the most common error.

Reserve output space explicitly, as a line item, before allocating anything else:

available_for_input = documented_limit
                    − max_output_tokens
                    − safety_margin

The safety margin covers counting error. Your count and the provider’s will not agree perfectly — message formatting, role markers, and tool-call structure all add tokens you didn’t count. A few percent is prudent.

Then, and only then:

retrieval_budget = available_for_input
                 − system_prompt
                 − tool_definitions
                 − history_allowance

Note the order. Retrieval is the residual — what’s left after the fixed costs and the reservations. Most systems do it backwards: fill the window with retrieved passages, then discover there’s no room for the answer.

Two limits, not one

An important distinction, and the reason a working system can quietly degrade.

The hard limit is where the model rejects the request. It’s visible, it errors, you fix it.

The soft limit is where quality starts declining even though everything fits. Adding more retrieved passages past some point makes answers worse rather than better — the relevant passage is now competing with nine mediocre ones, and the model has more opportunity to be distracted.

The soft limit is lower than the hard limit, it’s specific to your model and task, and nothing tells you when you’ve crossed it. No error, no warning. Answers just get slightly worse while your bill gets larger.

This is why “we have a big window now, let’s just include more” is a mistake dressed as an optimisation. The window growing raises the hard limit. It does not obviously raise the soft one.

Finding your soft limit is empirical: hold everything else constant, vary the number of retrieved passages, and evaluate answer quality at each setting. The curve usually rises, flattens, and then declines. You want the flat part, at its left edge — the fewest passages that get you the quality.

Cost, as a formula

No prices here, since they change. The structure doesn’t:

cost_per_request = (input_tokens  × input_unit_price)
                 + (output_tokens × output_unit_price)

Two observations that follow from the formula rather than from any specific price:

Output is typically priced higher than input per token. So a verbose response format is often more expensive than a large prompt, which is counterintuitive when you’ve been staring at your retrieval budget.

Fixed costs multiply by request volume. A system prompt reduced by 500 tokens saves 500 tokens times every request you will ever serve. At scale that dwarfs any per-request retrieval tuning, and it’s a one-afternoon job.

If prompt caching is available for your model, the arithmetic shifts again — stable prefixes become much cheaper to repeat, which changes what’s worth trimming versus what’s worth keeping byte-identical.

The procedure

  1. Count all five components for one real request. Make the table.
  2. Reserve output space and a safety margin first.
  3. Optimise the fixed costs once — system prompt, tool definitions. Best return available.
  4. Set an explicit allowance for history rather than letting it grow.
  5. Give retrieval the residual, and find your soft limit empirically rather than filling it.
  6. Re-measure after any prompt change. They accumulate.

Then the two harder questions: counting accurately, and what to do when the budget doesn’t balance — what to cut first.

budgetingtokens