Token Budget

Auditing a system prompt nobody trimmed

Every incident produced a sentence. Every edge case produced a paragraph. Nobody has ever deleted a line from the system prompt, because deleting one might reintroduce a bug from March.

So it is now the second-largest fixed cost in your budget, paid identically on every request, and no single person can tell you what half of it is for.

Audit by section, not by total

The total is not actionable. Break the prompt into labelled sections and count each one.

Hypothetical numbers. The shape is what matters:

Section Tokens Share Last touched
Role and framing 90 6% at launch
Output format spec 240 16% 2 months ago
Grounding and citation rules 160 11% 3 weeks ago
Refusal and safety instructions 180 12% 1 month ago
Edge-case patches (14 of them) 430 29% last week
Few-shot examples (3) 380 26% at launch
Total 1,480 100%

Two rows do most of the damage in most real prompts: the accumulated patches and the examples. Both were added under conditions that no longer hold — a weaker model, an older format, a bug since fixed — and neither has an owner.

The “last touched” column is the useful one. A patch added at launch and never revisited is either load-bearing or dead, and you cannot tell which by reading it.

Get the sections into the code

Before deleting anything, stop maintaining the prompt as one string. Assemble it from named parts:

system_prompt = join([
  ROLE,
  OUTPUT_FORMAT,
  GROUNDING_RULES,
  SAFETY,
  *ACTIVE_PATCHES,
  *ACTIVE_EXAMPLES,
])

Three things this buys immediately:

  • You can count each part and log the table above automatically.
  • You can toggle a part off for one experiment without editing prose.
  • A patch can carry a comment saying which failure it exists to prevent, which is the information the audit needs and the flat string destroys.

If your prompt is a single triple-quoted literal, this refactor is the highest-value hour available and it changes nothing about behaviour — verify that by asserting the assembled string is byte-identical to the old one, which also matters for caching.

Deleting safely

The reason nobody deletes is that nobody can prove a line is unnecessary. Make that provable.

Build the regression set out of the patches themselves. Each patch exists because something went wrong. Turn each one into a test case: the input that failed, and an assertion about the output. Fifteen patches gives you fifteen tests, and the prompt becomes editable.

If a patch’s originating failure is not recoverable — nobody remembers what it was for — that is itself a finding. Write the case you think it prevents, remove the patch, and see whether the case fails. An unexplained instruction that changes nothing measurable is dead weight.

Delete one section at a time, re-run, compare. Not a rewrite. A rewrite changes everything at once and you learn nothing about which part mattered.

Look for contradictions while you’re in there. Prompts that grew by accretion routinely contain two instructions that conflict, because the second was added by someone who did not read the first. Behaviour under contradiction is unpredictable, and resolving it usually removes tokens.

Consolidate duplicates. The same constraint stated three ways in three sections is common. Once, clearly, is shorter and more reliable.

The few-shot question

Examples are usually the largest single row and the least examined. They were added early, often with a weaker model, to demonstrate a format that is now specified explicitly two sections above.

The test is direct: remove them, run your evaluation set, compare. Three outcomes, all useful:

  • No quality change. Delete them. This happens more often than people expect, particularly for format compliance on current models, and particularly if you also have a structured-output mechanism doing the same job.
  • Quality drops. They are earning their space. Now ask whether three are needed or whether one does most of the work, and whether the ones you keep are the shortest adequate examples.
  • Quality drops only on a specific input class. Load examples conditionally for that class, and pay the tokens only when they matter.

The cost of getting this wrong is asymmetric in an easy direction: format regressions are visible and cheap to catch, so this is a safe experiment to run.

The saving, and its multiplier

tokens_saved_per_request = removed_system_tokens
total_saved              = removed_system_tokens × requests_served

Two consequences worth stating plainly. First, this is the cheapest optimisation you have, because it is done once and paid back on every request indefinitely. Second, it is also the one with the most headroom, because unlike retrieval nobody has ever tuned it.

There is a second-order effect too. Freeing fixed tokens does not just reduce cost — it hands the space to the variable components, so retrieval and history each get more room before the cut order has to fire.

Keeping it from regrowing

The audit is worthless if the prompt is back to its old size in four months, and it will be.

Set a token budget for the system prompt and enforce it in a test. A hard assertion in CI: the assembled prompt must be under N tokens. When someone needs to exceed it, they have to remove something or raise the limit deliberately. Either is fine; drifting past it silently is not.

Require a test with every patch. New instruction, new case in the regression set. This is the rule that makes the next audit possible.

Log the section table on deploy. Then growth is visible as a trend rather than discovered during an incident.

Re-run the whole audit on model change. Instructions written to work around one model’s habits are frequently unnecessary on its successor, and occasionally counterproductive. A model upgrade is the best available excuse to delete things, and rebudgeting on a model change is worth doing at the same time.

What to measure

  • Assembled system-prompt tokens, per deploy, as a trend.
  • Tokens per section, so growth has an address.
  • Pass rate on the patch-derived regression set, per edit.
  • Evaluation quality before and after each deletion — one deletion at a time.

The end state is not a short prompt. It is a prompt where every section has a test, an owner, and a reason, and where the count is a number someone chose rather than a number that happened. Where it sits against everything else competing for the window is the budget.

budgetingmeasurement