Token Budget

Designing a prompt prefix that stays stable

Two requests, the same application, the same system prompt. One was served from cache and one was not, and the difference was a timestamp somebody added to the top of the prompt six months ago.

Prompt caching turns the front of your prompt into an asset with a maintenance requirement: it pays off only while it stays byte-identical. That changes the allocation calculus, because suddenly some tokens are much cheaper to keep than others.

What caching actually rewards

Implementations differ and you should read your provider’s documentation for specifics, but the general shape is consistent: an identical leading portion of the prompt can be reused across requests at reduced cost and latency. Divergence ends the reuse — everything from the first differing token onward is processed normally.

Three consequences follow directly, and they are what the rest of this post is about:

  1. Order by volatility. Stable content first, volatile content last.
  2. Identical means identical. Not semantically equivalent. Byte-for-byte.
  3. The boundary is where your first change is. One early volatile token forfeits the whole prefix.

Sort the budget by how often it changes

Take your budget table and add a column for change frequency. Hypothetical numbers:

Component Tokens Changes Cacheable?
System prompt 900 on deploy yes
Tool definitions 2,400 on deploy yes
Static reference block (schema, glossary, taxonomy) 1,800 weekly yes
Retrieved passages 4,500 every request no
Conversation history 2,600 every turn partly
Instructions + query 300 every request no

That is a 5,100-token stable prefix out of a 12,500-token prompt — 41% of the input, reusable across every request, if nothing volatile is placed above it.

The interesting row is the third. Once a stable prefix is cheap to repeat, small always-needed reference material becomes affordable to keep resident rather than fetching it every time. That is an allocation decision caching unlocks, and it has its own post — what deserves a permanent seat in the window.

What silently invalidates it

Every item here is something a reasonable engineer adds without thinking.

A timestamp or date at the top of the system prompt. The single most common cause. If the model needs today’s date, put it after the cacheable section, not before it.

A user ID, session ID, tenant name or request ID in the system prompt. Same problem. Personalisation belongs below the shared prefix.

Non-deterministic serialisation. Tool definitions built from a dictionary whose iteration order varies, JSON with unstable key order, a set rendered to a list. Semantically identical, byte-different, cache missed. Sort explicitly and pin the order.

Whitespace drift. A templating engine that emits a trailing newline conditionally. Indentation that depends on how a string was constructed. Invisible in a diff, fatal to a prefix.

Locale- or environment-dependent formatting. Numbers, dates, or currency rendered per locale inside what you believed was a fixed string.

Conditional inclusion of anything early. A per-request feature flag that adds one line to the system prompt splits your traffic into two prefixes — which is survivable if there are two, and not if the flags combine into dozens.

Make it testable

Prefix stability is a property you can assert, and if you do not assert it, it will regress.

# in tests
prefix_a = build_prefix(request_fixture_1)
prefix_b = build_prefix(request_fixture_2)
assert prefix_a == prefix_b          # byte equality, two unrelated requests

assert hash(prefix_a) == GOLDEN_HASH  # changes only when someone means it

The first assertion catches per-request contamination. The second catches accidental edits: the golden hash changes on deploy when the prompt genuinely changed, and a diff nobody intended shows up as a failing test rather than as a cost graph three days later.

Also log the prefix hash with every request. Then “how many distinct prefixes is production actually producing?” is a query rather than an investigation — and the answer is frequently more than anyone expected.

How caching changes what is worth trimming

This is the part that reverses ordinary advice.

Without caching, every token in the fixed cost is paid at full price on every request, so trimming the system prompt and tool block is the best return available. With caching, those same tokens are the cheapest tokens in the prompt.

effective_input_cost = (cached_prefix_tokens × cached_unit_price)
                     + (uncached_tokens      × input_unit_price)

Substitute your provider’s numbers. The cached unit price is materially lower than the standard one, which produces four practical shifts:

Trimming a cached prefix has a lower payoff than it used to. Still worth doing for window space, but the cost argument weakens.

Stability is worth more than brevity. A prefix 300 tokens larger that never varies usually beats a smaller one that varies per request. If you must choose, choose stable.

Volatile tokens are the expensive ones. Retrieved passages, history and the query are paid at full rate every time. That is where per-token discipline now belongs, and it re-ranks your optimisation queue.

Cache writes are not free. Establishing a cache entry typically costs more than a plain request. A prefix that is written and rarely read is a loss — which makes hit rate the metric that decides whether any of this is working.

Two things that fight caching

Name them, because both are otherwise good ideas.

Conditional tool loading. Assembling a bespoke tool set per request maximises window efficiency and destroys prefix stability. Compromise: a small number of fixed bundles, so you have a handful of cacheable prefixes rather than one per request. Tool definitions are a line item covers the trade in detail.

History compaction. Rewriting the middle of a conversation changes the prompt above the point where you are still appending, so the reuse you had is gone on the next turn. The tension is real and it needs a policy — see when compaction collides with your cache.

What to measure

  • Cache hit rate, and cached versus uncached input tokens as reported by the provider. If your provider exposes these fields and you are not logging them, start there.
  • Distinct prefix hashes per hour. Should be a small number. A rising count means contamination.
  • Hit rate around deploys. A dip after a deploy is expected — the prefix changed for everyone. Know that so you do not diagnose it as a regression, and consider whether you want to deploy prompt changes during your quietest hour.
  • Time to first token, cached versus not. The latency benefit is often the bigger practical win and it is easy to forget to check.
  • Entry lifetime versus your traffic gaps. Cache entries expire; if your traffic is bursty enough that entries routinely expire between bursts, you are paying to write caches nobody reads.

The rule

Stable first, volatile last, byte-identical enforced by a test, prefix hash logged, hit rate on a dashboard. Then reallocate: the space and money caching frees belong to the volatile components, which means going back to the budget and re-deciding the shares with the new prices in hand. Where the sections physically sit is where in the prompt things should go.

cachingprompt-order