Do your few-shot examples still earn their space?
Three examples were added early, with a weaker model, to get the output format right. The format is now also specified explicitly, and possibly enforced by a structured-output mode, and the examples have never been touched since.
They are a fixed cost paid on every request. Nobody has checked whether they are still doing anything.
Size the block first
Count the examples as a block and as individuals, because the answer is often “keep one, drop two.”
Hypothetical numbers:
| Example | Tokens | What it demonstrates |
|---|---|---|
| Example 1 — simple case | 120 | basic format |
| Example 2 — multi-part answer | 310 | format under complexity |
| Example 3 — no answer available | 90 | refusal behaviour |
| Example 4 — table output | 340 | table formatting |
| Block | 860 |
Placed in a full request budget:
| Component | Tokens | Share |
|---|---|---|
| System prompt (excluding examples) | 620 | 5% |
| Few-shot examples | 860 | 7% |
| Tool definitions | 1,800 | 14% |
| Conversation history | 2,700 | 21% |
| Retrieved passages | 5,000 | 39% |
| Output reservation | 1,900 | 15% |
| Total | 12,880 | 100% |
7% of every request, forever. And note what it competes with: 860 tokens is a passage or two of retrieval, on every single query.
The ablation
This is a cheap experiment and it is the whole post. Do not reason about whether examples help; measure it.
1. Fix an evaluation set with known-good answers. Include the hard cases.
2. Baseline: current prompt, all examples. Record quality and format compliance.
3. Remove ALL examples. Re-run. Record.
4. If quality held → delete them. Done.
5. If quality dropped → add back one example at a time,
re-running after each, and stop when quality recovers.
6. Keep the smallest set that recovers baseline quality.Step 5 is the part people skip, and it is where the saving usually is. The common result is that one example carries most of the benefit and the others are insurance nobody priced.
Two things the evaluation set must include or the experiment misleads you:
Format compliance as a separate metric from answer quality. Examples mostly teach shape. If you only score correctness, you will miss a regression in structure — and structure failures cause parse errors and retries, each of which costs a full request.
The awkward inputs. Empty results, contradictory sources, questions the corpus cannot answer. This is where examples demonstrating refusal or edge-case behaviour earn their space, and a happy-path evaluation set will tell you they are useless.
What each outcome means
No measurable difference. Delete the block. This is a genuinely common result on current models for format-only examples, especially if the format is also stated explicitly or enforced by the API. Take the 7% and hand it to retrieval.
Quality drops broadly. The examples are doing real work, probably conveying something the instructions state badly. Now ask two follow-ups: can the instruction be improved so a shorter example suffices, and is each individual example necessary? A 340-token table example might be replaceable by a 40-token schematic one that shows the shape without a full worked case.
Quality drops on one input class only. The best outcome, because it is actionable: load that example conditionally when you detect that class, and pay the tokens only when they matter. Costs a routing step and some prefix stability — see below.
Quality improves without them. It happens. Examples anchor behaviour, and an example that is subtly unlike your real inputs can bias every answer toward its own shape. Overly narrow examples produce overly narrow answers.
Cheaper substitutes
Before concluding that examples are required, price the alternatives that do the same job for fewer tokens:
A schematic instead of a worked example. Show the shape with placeholders rather than a full realistic case. Most of the instructional value, a fraction of the tokens.
A tighter format specification. Often the examples exist because the instruction is vague. “Return
one line per finding, severity|component|description, no header” may replace two examples outright.
Provider-enforced structured output. If the model can be constrained to a schema, examples teaching that schema are redundant by construction. Check whether your provider supports it before spending window on demonstrating syntax.
Truncated examples. An example does not have to include a full realistic input. A 300-token document in an example can usually become a 30-token stub, since what you are demonstrating is the response.
The caching wrinkle
Examples usually sit in the system prompt, inside the cacheable prefix, so if you use prompt caching they are among the cheapest tokens you pay for — the cost argument for deleting them weakens considerably.
But the window argument does not. Cached or not, those tokens occupy space that retrieval and history cannot use. If you are nowhere near your limit, keep them and stop worrying. If you are cutting passages on a meaningful share of requests, they are still competing.
And conditional loading of examples fragments the prefix, so it trades cache hits for window space. Group into a small number of fixed variants rather than assembling per request — the same conclusion as designing a prompt prefix that stays stable reaches for tools.
What to measure
- Example block tokens, tracked with your other fixed costs.
- Format compliance rate, separately from answer quality. Your regression guard for every deletion.
- Parse failure and retry rate. The expensive consequence of a format regression.
- Quality per input class, not aggregated. An aggregate score hides the class where the example was the only thing holding behaviour together.
- Re-run on model change. Examples are the most model-dependent part of a prompt. Advice tuned for one model’s habits is frequently unnecessary on its successor, which makes an upgrade the best time to re-run this ablation. See rebudgeting when you change model.
The default position
Assume examples are stale until an ablation says otherwise, keep the smallest set that recovers baseline quality, prefer schematics to worked cases, and re-run the test whenever the model changes. Then put the reclaimed tokens somewhere deliberately — the point of the audit in auditing a system prompt nobody trimmed is not a smaller number, it is a better allocation.