Token Budget

Counting tokens before you spend them

You estimated the prompt at 6,000 tokens using the four-characters-per-token rule. The provider counted 7,400. The request was rejected, and the difference wasn’t rounding — it was three separate things your estimate had no way to see.

Counting properly is unglamorous and it’s the foundation for everything else here. A budget built on estimates is a budget that fails at the worst moment, which is under unusual input.

Why the rules of thumb drift

The common heuristics — roughly four characters per token, roughly ¾ of a word per token — are derived from ordinary English prose. They’re serviceable for a back-of-envelope figure and they mislead in specific, predictable ways.

Tokenizers split on subword units learned from a training corpus. Common English words are often a single token. Rare words fragment. And content that doesn’t look like English prose fragments a lot.

Where the estimate goes wrong, roughly in order of how badly:

Content Behaviour vs the estimate
Ordinary English prose About right
Code Worse — punctuation, indentation, and identifiers fragment
JSON Much worse — structural characters are frequently their own tokens
URLs, UUIDs, hashes Much worse — near-random strings barely compress
Non-Latin scripts Often substantially worse, varying by script and tokenizer
Repeated whitespace Varies; some tokenizers collapse runs, some don’t

The pattern: the estimate is calibrated on prose, and it degrades in proportion to how unlike prose your content is. A RAG system over technical documentation full of code blocks, or one returning JSON tool results, will consistently blow past a prose-calibrated estimate.

This matters most at the tails. Your average request may sit comfortably within an estimate while the one containing a large base64 blob doesn’t, and that’s the request that pages someone.

Count with the real tokenizer

The only reliable method is the tokenizer the model actually uses. Providers publish these, either as a library you run locally or as an endpoint you call.

Prefer local. Counting is a hot path — you want it during prompt assembly, potentially several times as you decide what fits, and a network round trip per count is unworkable. A local tokenizer makes counting cheap enough to do freely.

Two cautions:

Tokenizers are model-specific. Different families use different vocabularies, and counts do not transfer. A prompt counted for one model tells you little about another. If you support several models, count per model.

Tokenizers change between model versions. Not often, but it happens. Pin the tokenizer version alongside the model version, and re-verify when you upgrade.

What you didn’t count

Here’s the part that explains most of the remaining gap: the tokens in your text are not the tokens in your request.

Chat APIs wrap your content in structure. Depending on the provider, that includes role markers for each message, delimiters between messages, and formatting around the whole conversation. Tool definitions get serialized into a schema representation. Tool calls and their results are wrapped too.

None of this appears when you tokenize your message strings and add them up. It’s a real, per-message overhead, and it scales with the number of messages, not their length. A conversation of forty short turns carries substantially more of it than one of four long turns with the same character count.

Two ways to handle it:

Use the provider’s counting endpoint or library method for whole requests, if one exists. Some providers expose a way to count a fully-assembled request including formatting. This is the accurate answer and worth using where available.

Calibrate your own overhead. Assemble a representative request, count it yourself by summing message contents, then compare against what the API reports as input tokens for the same request. The difference is your formatting overhead. Divide by message count to get a per-message constant, add it to your model, and re-derive it when the provider or model changes.

The second approach is worth doing regardless, because it turns an unknown into a measured number.

Build a budget check

Estimating is not the goal; not overflowing is. That means a check in the assembly path.

assemble(system, tools, history, passages, max_output):
    limit     = model_documented_limit
    reserved  = max_output + safety_margin
    available = limit − reserved

    fixed = count(system) + count(tools) + per_message_overhead × baseline_messages
    if fixed > available:
        fail loudly — this is a configuration bug, not a runtime condition

    remaining = available − fixed
    kept_history  = fit(history,  remaining × history_share)
    remaining    -= count(kept_history)
    kept_passages = fit(passages, remaining)

    return build(system, tools, kept_history, kept_passages)

The important properties:

It fails before the API does. You control the failure, so you can log what was dropped, emit a metric, and degrade deliberately rather than getting a 400.

Fixed costs exceeding the budget is a distinct error. If the system prompt and tools alone don’t fit, no amount of trimming retrieval helps. That’s a deployment-time bug and should be loud — ideally caught by a test rather than in production.

History and retrieval are fitted separately, in priority order. Whichever you fit first wins the contested space. That ordering is a policy decision, covered in what to cut first.

The safety margin absorbs counting error. A few percent. Tighten it once your calibrated overhead is trustworthy.

Measure what you actually spend

The check prevents overflow. It doesn’t tell you whether the budget is well spent.

Log, per request:

  • Token count for each component separately — system, tools, history, passages.
  • How many passages were dropped by the fit, and how often that happens.
  • Actual input and output tokens as the provider reports them.
  • The delta between your count and the provider’s. This is your calibration error, and if it drifts, something changed.

That last one is the highest-value metric on the list and almost nobody records it. A sudden divergence means the tokenizer changed, the formatting changed, or something upstream started emitting content your model doesn’t handle the way you assumed.

Then aggregate: what share of your tokens goes to each component, in the mean and at p95. The mean tells you where your money goes. The p95 tells you which requests are about to fail.

Estimating when you genuinely can’t count

Sometimes you need a figure before content exists — capacity planning, or sizing a chunk budget.

Reasonable approach: derive your own ratio from your own corpus rather than using a generic one. Tokenize a representative sample, divide tokens by characters, and use that. It accounts for your actual mix of prose, code, and structured data.

Then treat it as a planning figure only, never as a runtime check. Runtime always counts for real.

Checklist

  • Use the model’s own tokenizer, locally, pinned to the model version.
  • Calibrate per-message formatting overhead against actual API-reported counts.
  • Reserve output space plus a safety margin before allocating anything.
  • Check the budget during assembly; fail loudly on fixed-cost overflow.
  • Log per-component counts and the count delta.
  • Derive corpus-specific ratios for planning; never for enforcement.

Once counting is trustworthy, the allocation decisions become tractable — starting with the budget itself.

tokensmeasurement