Token Budget

What formatting overhead costs you

Your passages total 4,000 tokens of source text. The prompt section containing them counts 5,200. The difference is the wrapper you put around each one, and you are paying it once per item on every request.

Overhead is not waste by definition — labels do real work — but it is a line item nobody has costed, and in a prompt assembled from many small items it is routinely 20–30% of the section.

Price the wrapper per item

Count one item’s wrapper, then multiply by item count. Hypothetical numbers, a passage rendered as JSON:

Wrapper element Tokens
Opening brace, indentation, newlines 6
"source_document_title": + value 14
"section_heading": + value 11
"retrieval_score": + value 9
"ingested_at": + timestamp 13
"content": key and quoting 5
Closing brace, comma 3
Wrapper per item 61

At 12 items that is 732 tokens of wrapper against 4,000 tokens of content — 15% before you have written a single instruction. Change the same passage to a compact delimited form:

[7] Employee Handbook / Leave policy
Full-time employees accrue 1.75 days per month...
Wrapper element Tokens
Identifier in brackets 4
Title and section, slash-separated 9
Newlines 2
Wrapper per item 15

12 items × 46 tokens saved = 552 tokens reclaimed, which is another passage of actual content. The formula:

section_tokens = Σ item_content
               + (wrapper_tokens × item_count)

reclaimable    = (old_wrapper − new_wrapper) × item_count

The multiplier is item count, so the smaller your items and the more of them you include, the more this matters. A prompt with 40 short items and a verbose wrapper can spend more on scaffolding than on substance.

What the wrapper is for, and what to keep

Do not strip it to nothing. Labels prevent a specific failure: unlabelled items concatenated together blur into one document, so the model cannot tell where one ends, cannot cite, and will happily merge two contradictory sources into one confused claim.

Keep, in order of value:

A boundary. Something unambiguous between items. A blank line plus a bracketed identifier is enough; you do not need a fenced block per item.

A short identifier. So the answer can cite [7] and you can resolve it back to a source. Four tokens, and it is what makes verification possible at all.

A source name. Enough for the model to weigh two sources against each other. The document title, not its full path.

Drop, unless you can name what reads them:

Timestamps in ISO form on every item. Long, and rarely used by the model. If recency matters, a year or a short date is a fraction of the tokens. If it does not matter, remove it.

Scores. They rarely change the answer and can bias the model toward the top item in a way you did not intend. Order already conveys rank.

Internal identifiers. Long random-looking strings are among the worst content for a tokenizer — near random text barely compresses, so a UUID costs far more than its length suggests. Use a per-request index [1]…[n] in the prompt and keep the mapping to real IDs in your own code.

Paths, URLs and full file locations. Same problem, same fix.

Repeated boilerplate that came along with the text. Headers, footers, navigation, cookie notices, confidentiality blocks. This is content-shaped overhead and it is often the single largest offender — the same 40 tokens of footer appearing in eleven items is 440 tokens of nothing. Strip it at assembly if it survived earlier processing.

Format choice for the section as a whole

Three shapes, in order of token efficiency:

Delimited plain text with a short label line. Cheapest, and models handle it well. Default choice.

Lightweight markup — a heading per item. Slightly more, and worth it if the model must reproduce structure.

JSON or XML per item. Most expensive. Justified when the section is machine-generated and machine-verified, or when your provider’s guidance for the specific model recommends a particular structure — some models are documented as responding better to certain delimiters, which is worth following over generic advice.

One caution: consistency matters more than compactness. A section where three items are labelled one way and nine another is worse than a slightly verbose but uniform one.

The overhead you cannot see

Two more sources, both easy to forget in the arithmetic:

Per-message framing. Role markers and delimiters the API adds around each message. It scales with message count, not length, so a conversation of many short turns carries more of it than one of few long turns — and it is invisible if you count by summing message strings. Calibrate it against the provider’s reported input count, per counting tokens before you spend them.

Whitespace from your templating. Indentation from a nicely formatted template, blank lines from conditional blocks that rendered nothing. Free to fix and it shows up immediately in the count.

What it costs to trim

Citation reliability. Remove identifiers and the model cannot cite; remove source names and it cannot distinguish. Both are worse failures than a 500-token saving is a win, so keep them.

Debuggability. A verbose wrapper is easier to read in a logged prompt. Fix this in logging, not in the prompt: log the compact prompt plus a side table expanding the identifiers.

A change you cannot attribute. Reformatting the passage section changes the prompt, so answer quality can move for reasons unrelated to tokens. Change the format alone, hold retrieval and instructions still, and compare against a fixed evaluation set — otherwise you will spend a day attributing a quality shift to the wrong cause.

What to measure

  • Overhead ratio for the section: section tokens divided by content tokens. One number, easy to alert on, and it makes the waste arguable.
  • Wrapper tokens per item, logged at assembly, so a change in the template shows up as a change in a metric.
  • Items included per request, before and after. The point of the saving is more content, not a smaller prompt — if item count did not rise, you banked nothing.
  • Citation rate and citation validity. Your guard against over-trimming the labels.
  • Boilerplate detection hits. Count how much repeated text you stripped per request; a rising number means something upstream changed and is worth knowing about.

Then reassign what you reclaimed. Overhead reduction is only a win if the freed tokens go somewhere you chose — back to the budget and its shares.

tokensbudgeting