Tool definitions are a line item

Your bill per request went up and nothing about your retrieval changed. Or a request failed on the limit and you trimmed passages, again, because passages are the part you think of as “the context.”

Count the tool block. On tool-heavy applications it is routinely larger than the retrieved passages, and unlike passages it is paid on every request, in full, whether or not any tool is called.

Where it goes, per tool

The mistake is counting the tool block as one number. Count it per tool, because that is the unit you can act on.

Hypothetical numbers throughout. Substitute your own counts:

Tool	Tokens	Share of tool block
`search_documents`	210	9%
`run_report`	640	27%
`list_accounts`	180	8%
`update_record`	890	37%
`get_exchange_rate`	130	5%
`send_email`	200	8%
5 further tools	150	6%
Tool block	2,400	100%

Then place that block in the full request budget so the proportion is visible:

Component	Tokens	Share
System prompt	900	7%
Tool definitions	2,400	18%
Conversation history	3,200	25%
Retrieved passages	4,500	35%
Output reservation	2,000	15%
Total	13,000	100%

Two tools are 64% of the tool block here. That is the normal shape: a small number of schemas with many parameters, deep nesting, and descriptions written like documentation dominate everything else.

The arithmetic that makes it worth an afternoon

Fixed costs multiply by volume. That is the whole argument:

tokens_saved_total = tokens_removed_from_tool_block × requests_served

No prices here — substitute your own input unit price — but note the shape. Removing 600 tokens from two schemas saves 600 tokens on every request your application will ever serve, forever, for one afternoon of work. No amount of per-request passage tuning competes with that, and passage tuning has to be redone whenever the corpus changes.

What to trim

Write descriptions for selection, not comprehension. A tool description exists so the model can decide whether this is the right tool and what to put in the fields. It does not need rationale, history, links, or worked examples. Most oversized schemas are oversized because someone wrote them for a human reader.

Prefer enumerations to prose constraints. A parameter documented as “one of draft, sent, archived” is shorter and more reliable than a paragraph explaining which states are valid when.

Flatten deep objects. Nested schema objects cost structural tokens at every level, and the serialised form is what gets counted, not your source definition.

Delete parameters nobody sets. Optional parameters accumulate. Each one costs its name, type, and description on every request in exchange for a capability used twice last quarter.

Delete tools nobody calls. This needs data, not intuition — see the measurement section. An unused tool is pure overhead, and it also makes selection harder for the ones you keep.

Collapse overlapping tools. Three tools that differ only in a filter are usually one tool with a parameter, at roughly a third of the token cost.

Conditional loading, and its price

The larger move is to stop sending all tools on all requests: route the request, then include only the plausibly relevant schemas.

The saving is real and it is paid per request. Three costs, and the third is the one people miss:

A routing step. Something has to decide which tools to include, and it can be wrong. Budget for a fallback: if the model asks for something it can’t reach, re-run with the fuller set. That request now costs two round trips.

Worse selection when the set changes shape. Models are sensitive to what surrounds a choice. A tool that behaved well in a set of twelve can behave differently in a set of three.

Cache invalidation. If your provider supports prompt caching, the tool block usually sits in the stable prefix. Varying it per request means a different prefix per request, so you trade a cache hit for a smaller prompt — frequently a bad trade. If you do this, group tools into a small number of fixed bundles rather than assembling a bespoke set each time, so there are only a handful of distinct prefixes to cache. The interaction is covered in where in the prompt things should go.

What terser schemas cost

Trimming descriptions is not free, and the failure is quiet.

The model chooses tools and fills parameters from those descriptions. Cut past a certain point and you get more wrong-tool calls, more malformed arguments, and more retries — each of which costs a full request. A tool block reduced by 40% that raises the retry rate has made things worse on both axes, and only the token count will look better on a dashboard.

So treat it as an experiment rather than an edit. Trim, then compare tool-selection accuracy against a fixed set of test requests before and after. If accuracy holds, keep the cut; if not, put the sentence back — and note which sentence it was, because that tells you what the model was actually using.

What to measure

Four things, all cheap to log:

Tokens per tool definition. Emit it at startup as a table sorted descending. This is the audit, and having it in logs means nobody has to redo it by hand.

Call frequency per tool. Over a representative window. A tool at zero calls after a month of production traffic is a deletion candidate. A tool at 0.1% of calls is a conditional-loading candidate.

Tokens returned per call, per tool. A separate problem with the same cause: a tool that returns large payloads inflates history for the rest of the conversation. Cap it at the tool.

Wrong-tool and malformed-argument rate. Your regression guard for every trim above. Watch it after any schema edit, and after any model change.

The order to do this in

Count each tool definition as it will be serialised, and rank them.
Trim the top two or three. That is most of the available saving.
Verify tool-selection accuracy held, on fixed test requests.
Delete unused tools using call frequency, not opinion.
Cap tool return sizes, which is the same budget from the other end.
Only then consider conditional loading, in fixed bundles, with a fallback path.

Then re-count. The tool block grows the same way a system prompt does — one plausible addition at a time — so put the audit on a schedule rather than treating it as done. Where the block sits in the overall allocation is the budget; what happens when it stops fitting is the cut order.