Budgeting for tokens you never see

The answer was 200 tokens. The provider billed you for far more output than that, and a request that should have fit comfortably was rejected on the limit.

Nothing is wrong with your counting of the visible parts. There are tokens in a request’s total that never appear in anything you or the user reads, and if your budget only models what is visible, it is modelling the wrong request.

The invisible line items

Four categories, in rough order of how often they surprise people.

Intermediate reasoning tokens. Some models generate an internal reasoning pass before their visible answer. Those tokens are generated, generally counted against output, and generally billed — while the answer you display may be a small fraction of the total. Whether a given model does this, whether the trace is returned to you, and how it is counted are things to check in that model’s documentation rather than assume; the behaviour differs by model and by API version.

Discarded and retried generations. A response that failed validation and was re-requested cost you the full first attempt. Same for a malformed tool call the model then corrected, and for any “regenerate” path. The user sees one answer; you paid for two or three.

Tool-call round trips inside one logical turn. An agent that searches, reads, searches again and then answers has issued several requests, each carrying the full prompt. The user sees one exchange.

Request framing. Role markers, message delimiters and the serialised form of tool schemas. Small per message, and it scales with message count rather than message length, so it grows fastest exactly where you have the least room.

What the request actually costs

Model the logical turn, not the API call. Hypothetical numbers:

Line item	Tokens	Visible to the user?
Prompt, as counted from your strings	9,400	partly
Request framing overhead	300	no
Reasoning pass (model-dependent)	1,100	no
Visible answer	240	yes
Second search round trip (full prompt again)	9,700	no
Reasoning pass, second call	600	no
Discarded generation after a schema failure	180	no
Total for one user-visible answer	21,520	240 tokens of it

The user got 240 tokens. The turn consumed roughly ninety times that. Ratios of that order are ordinary on agentic systems and they are why “our prompt is only 9,000 tokens” is not an answer to “why is this expensive.”

Notice also that the prompt is paid once per round trip, not once per turn. That is the strongest argument in this post for a small fixed cost: your system prompt and tool block are multiplied by round trips, not by turns.

Reserving for what you cannot see

The reservation rule from reserving room for the answer needs one extra term:

reserved = expected_hidden_output      # reasoning, if the model produces it
         + max_visible_output
         + safety_margin

The problem is that expected_hidden_output is variable and not fully under your control. Practical approach:

Find out whether your model has a controllable budget for it. Several do — a parameter or effort setting that bounds the reasoning pass. If yours does, set it, because an unbounded hidden component in a budget is not a budget.

Otherwise, measure it. Compare provider-reported output tokens against the token count of the visible answer, over real traffic. The difference is your hidden output, and its p99 is what you reserve against.

Reserve per request type. A classification request and an analytical one will not have the same hidden component, and averaging them over-reserves for one and under-reserves for the other.

Round trips change what is worth trimming

Once you count per logical turn, the ranking of optimisations shifts:

turn_cost ≈ round_trips × (fixed_prompt + variable_prompt)
          + total_output_including_hidden

Because fixed_prompt is multiplied by round trips, a 500-token reduction in the system prompt or tool block is worth 500 × round trips per turn, not 500. On a four-hop agent that is a 2,000-token saving per turn from a one-time edit. See tool definitions are a line item and auditing a system prompt nobody trimmed.

The second consequence: reducing round trips is a budget optimisation, not just a latency one. Fewer, better-specified tool calls beat more exploratory ones on both axes.

Do reasoning traces belong in history?

An allocation question with a genuine trade-off, and one to resolve deliberately rather than by default.

If your API returns reasoning content and you append it to the transcript, it is now paid again on every subsequent request in the conversation, compounding. The default should be drop it from history and keep the visible answer, for the same reason you collapse old tool payloads to their outcomes: the model has already used it.

Two caveats. Some APIs require returning specific reasoning artefacts on subsequent calls for multi-step interactions to work correctly — check what your provider documents, because dropping something required breaks correctness rather than just quality. And on a genuinely multi-step deliberation, keeping the most recent reasoning while dropping older ones is a reasonable middle.

The general principle from multi-turn budgets applies: keep what the next turn needs, not everything the last turn produced.

What to measure

• Provider-reported output tokens minus visible answer tokens, per request. This is the hidden output, and almost nobody logs it. Watch its p99, not its mean.
• Round trips per user-visible answer. Distribution, not average — the tail is where the cost lives.
• Retry and validation-failure rate. Each one is a fully paid request that produced nothing.
• Total tokens per resolved user request, as your headline number. This is the figure that maps to the bill; per-call token counts do not.
• Tokens per useful token: total turn tokens over visible answer tokens. A crude efficiency ratio, but it moves when something regresses and it is easy to alert on.

The correction

Budget per logical turn, reserve for the hidden output at p99, bound the reasoning pass if your model lets you, drop reasoning traces from history unless the API requires them, and remember that every token of fixed prompt is multiplied by round trips. Then re-derive the whole allocation from the budget using turn totals rather than single-call totals — the proportions usually change enough to redirect what you optimise next.