Token Budget

The output format is a budget decision

You asked for JSON with descriptive field names and a nested structure, because it was pleasant to consume. The answer’s payload is 60 tokens of actual information wrapped in 400 tokens of scaffolding, and you are paying for that on every request at the output rate.

Response format looks like an interface decision. It is an allocation decision: it sets the size of the output reservation, which sets how much window is left for everything else.

Price the format before you choose it

Take one representative answer and count it in each candidate format. Hypothetical numbers, same information in all four rows:

Format Output tokens Schema tokens in the prompt
Verbose nested JSON, descriptive keys 460 320
Flat JSON, short keys 180 140
Delimited lines (field: value) 110 60
CSV row against a documented header 70 40

Two columns, because a structured format is paid at both ends: the schema or format instructions sit in every prompt, and the response pays for the syntax again.

format_cost_per_request = schema_tokens_in_prompt
                        + (response_syntax_tokens × responses)

The prompt side is a fixed cost multiplied by request volume. The response side is paid at the output unit price, which is typically the higher of the two. That combination is why format is worth an hour of attention that it almost never receives.

Where the tokens actually go in structured output

Specific and fixable causes, in the order they usually rank:

Long field names, repeated per record. customer_account_identifier costs several tokens every time it appears. In a 50-record response that name is paid 50 times. Short keys with a documented meaning cost once, in the schema.

Nesting. Every level adds braces, indentation and the enclosing key. Flat structures with compound key names are frequently smaller than the nested equivalent, and easier for the model to produce correctly.

Pretty-printing. Indentation and newlines are tokens. If a machine consumes the output, ask for it compact.

Fields that are always the same. A status: "success" on every response, a version field, an echo of the request. Move constants to your own code; the model should not be generating values you already know.

Echoing the input. Asking the model to repeat the question, restate the passages it used, or return the record it was given. Expensive, and it is information you already hold — return an identifier instead.

Prose alongside structure. A reasoning or explanation field on every record, added to aid debugging and never removed. Sometimes worth it; usually it should be a flag you can turn on.

The cheapest format is often not JSON

JSON is the default because tooling expects it, not because it is compact. Alternatives worth pricing:

Delimited lines. One record per line, fields separated by a character that cannot appear in the data. Minimal syntax, and models are reliable at it. Your parser is ten lines.

A documented header, then rows. The field names live in the prompt once. Rows carry values only. For list-shaped answers this is usually the cheapest correct option by a wide margin.

Enum or identifier only. For classification, the answer is a label. Not a sentence containing the label, not an object containing the label — the label. It is remarkable how often a prompt asks for a paragraph and then regexes one word out of it.

Plain prose. For genuinely narrative answers, structure adds tokens without adding meaning. Do not wrap an essay in JSON to feel organised.

There is a real cost on the other side of this, and it is not token cost: fragile parsing. A provider-enforced structured-output mode gives you guarantees that a hand-rolled delimited format does not. If your provider offers constrained decoding, the reliability is often worth the syntax. Price the options, then decide — but decide, rather than defaulting to the most verbose one available.

Length instructions, and whether they hold

The other half of output cost is not syntax but volume: the model writing four paragraphs where one would do.

What works reasonably well:

  • A structural constraint rather than a numeric one. “One sentence per finding, at most five findings” is followed more reliably than “about 100 words.” Models are not good at counting their own output.
  • A format that has no room for padding. A CSV row cannot contain a preamble. This is the most reliable length control available, and it is a side benefit of the compact formats above.
  • Explicitly forbidding the preamble. “Do not restate the question. Do not summarise your approach.” Removes a predictable fixed cost from every answer.

What does not work: assuming a stated word limit will be respected precisely. Treat length instructions as a strong nudge, verify against your measured output distribution, and keep the cap set — see reserving room for the answer.

What a leaner format costs

Name the trade-offs, because they are real:

Readability for humans. Short keys are worse in a debugging session. Mitigate by expanding them in your own logs, not in the wire format.

Parse reliability. Every token you remove from the format specification is a token the model was possibly using to get the shape right. Watch your parse-failure rate after any format change, because a retry costs a full request and instantly erases the saving — see budgeting for tokens you never see.

Room for the model to think. Removing an explanatory field can reduce answer quality on tasks where articulating a chain helped. If it does, that is information: the field was doing work, and you can decide whether to keep it or move the deliberation somewhere you are not billing for it repeatedly.

Schema churn. Compact formats are less self-describing, so a field change is more likely to break a consumer silently. Version the format.

What to measure

  • Output tokens per request, before and after the change, at median and p99.
  • Syntax overhead ratio: output tokens divided by tokens of actual information. Crude, effective, and it makes the waste visible in a single number.
  • Parse-failure and retry rate. The regression guard. Any format saving is void if this rises.
  • Schema tokens in the prompt, tracked with your other fixed costs.
  • Reservation utilisation. A leaner format should let you lower the output reservation, which hands window back to retrieval and history. Actually lower it — otherwise you have banked the saving in a reservation nobody uses.

That last step is the one people skip. Shrinking the answer only improves the allocation if the freed space gets reassigned, which means going back to the budget and rewriting the table.

budgetingtokens