How much of your window is earning its place

You know what your prompt costs. You do not know what any part of it is buying, and those are different questions.

A budget is only meaningful if you can say which line items are earning their space. Otherwise you are optimising by intuition, and intuition consistently over-values the components that were hardest to build.

Utilisation, not just size

Add a second column to the budget table: what evidence exists that this component changed the answer. Hypothetical numbers:

Component	Tokens	Share	Evidence it was used
System prompt	900	7%	ablation: quality drops sharply
Tool definitions	2,400	19%	3 of 11 tools called this month
Few-shot examples	860	7%	ablation: no measurable change
Conversation history	3,000	23%	referenced in 12% of answers
Retrieved passages	4,500	35%	2.1 of 8 cited on average
Output reservation	1,900	15%	21% of reservation used at median

Read that table as a spending review. Roughly a third of this window has weak evidence of doing anything: eight unused tool schemas, a stale example block, six uncited passages per request, and an output reservation four times larger than the median answer.

None of those rows is proof of waste — an uncited passage may still have influenced the answer, and the reservation must cover the tail, not the median. But each is a hypothesis worth testing, and having them in one table is what turns “the prompt is too big” into a list of experiments.

Ablation is the only honest test

For each component, the question is causal: does removing it change the answer? Nothing about a token count answers that.

For each component C:
  1. Fix an evaluation set and a scoring method. Hold both constant.
  2. Baseline: full prompt. Record quality.
  3. Remove C entirely. Re-run. Record.
  4. Halve C. Re-run. Record.
  5. Read the three points:
       flat from full → removed   → C is not earning its space
       flat from full → halved    → C is oversized, not useless
       drops on removal only      → C matters at reduced size
       drops on both              → C is load-bearing at full size

Step 4 is the one that pays. Most components are not binary; the interesting finding is usually “this is worth having at half the size,” which is a reallocation rather than a deletion.

Two requirements, or the results mislead you:

Change one component at a time. Removing examples and passages together tells you nothing about either.

Score more than correctness. Format compliance, refusal behaviour on unanswerable questions, and citation validity each fail differently. An ablation scored only on correctness will happily delete the thing that was keeping your output parseable.

If you have no evaluation set, this post is blocked and building one is the prerequisite. A fixed set of questions with known-good answers, including the awkward cases, is enough to start — it does not need to be large to rank six components against each other.

Cheap proxies you can log continuously

Ablation is periodic. These run in production and tell you where to point the next one.

Citation coverage. If answers cite passage identifiers, log how many of the included passages were cited. Persistently citing 2 of 8 is a strong signal that you are paying for eight and using two. The caveat is real: an uncited passage may have contributed, and citation behaviour is itself a function of your instructions. Treat it as a pointer, not a verdict.

Tool call frequency per definition. Direct, unambiguous, and almost nobody logs it. A tool never called in a month of traffic is a deletion candidate — see tool definitions are a line item.

Reservation utilisation. Actual output tokens over reserved output tokens. Consistently low means you are holding window that retrieval could use, though the reservation must still cover the tail per reserving room for the answer.

Position of the cited material. If citations cluster at the start and end of the passage section and rarely land in the middle, that is consistent with the position effects described in where in the prompt things should go — but read it as a prompt to run a positional experiment, not as a measurement of the effect. Retrieval order, not position, may be doing all the work.

History reference rate. How often answers depend on earlier turns. Low rates mean your history allowance is generous; use it to justify a smaller one.

Overhead ratio. Section tokens over content tokens, per what formatting overhead costs you. Pure scaffolding is the easiest waste to remove because removing it cannot change the substance.

The efficiency number worth putting on a dashboard

One derived metric, crude but useful:

tokens_per_resolved_request = total_tokens_across_all_round_trips
                            / user_visible_answers_delivered

It captures retries, extra hops and hidden output, all of which per-call metrics miss, and it moves when something regresses. Track it as a trend with quality alongside it — falling tokens with falling quality is not an improvement, and the pairing is what stops this becoming a race to a small number.

What this measurement costs

Evaluation runs cost tokens. An ablation over six components at three settings each is eighteen runs of your evaluation set. Budget for it, and do it on a schedule — after model changes and major prompt edits — rather than continuously.

Proxies can mislead. Citation coverage understates influence. Reservation utilisation looks wasteful when it is correctly sized for the tail. Read every proxy as a hypothesis generator.

You will find results you dislike. Sometimes the expensive component nobody wants to remove is the one with no measurable effect. That is the point of measuring, and it is the only version of this work that saves anything.

The review, on a schedule

1. Log per-component tokens and the proxies above continuously.
2. Quarterly, and after any model change, run the ablation ladder on each component.
3. Rank components by tokens divided by measured contribution.
4. Cut or halve the worst-ranked one. One at a time.
5. Reassign the freed tokens explicitly — into retrieval, history, or a smaller total.
6. Re-run the table.

Step 5 is where most of these efforts quietly fail. Reclaimed space that nobody reassigns is not a saving, it is unused window. Take the result back to the budget and rewrite the shares.