What deserves a permanent seat in the window
The same database schema is retrieved on 90% of requests. So is the glossary that explains what your internal terms mean, and the taxonomy of product categories. Each time, something searches for them, ranks them, and puts them in the window — where they were always going to end up.
Material needed on nearly every request is a candidate for a permanent line item: assembled into every prompt as a fixed block. This is an allocation decision about a small, known, always-needed set — not a question about how the rest of your content gets into the window.
The test
Four conditions. It needs all four.
1. It is needed on almost every request. Not “often useful.” If it is needed on 30% of requests, a resident block is paying for it on the other 70% forever.
2. It is small. A block of a few hundred to a couple of thousand tokens can be resident. Something that is a meaningful fraction of the window cannot, because it will crowd out the material selected for the specific question.
3. It changes rarely. Weekly is fine; per request is not. This matters for caching, below.
4. Fetching it is unreliable or wasteful. If a schema is only retrieved when the question happens to mention a table name, the answers where it was needed but not mentioned are silently wrong. Residency converts an intermittent failure into a fixed cost, which is a good trade.
Good candidates in practice: a schema or data dictionary, a glossary of internal abbreviations, a category taxonomy, output conventions, the current date and tenant context, a short list of hard constraints.
Bad candidates: full documents, policy manuals, changelogs, anything with an “and while we’re here” case attached to it. Also anything you cannot name a per-request consumer for.
Cost it as a fixed line item
Residency moves tokens from the variable side of the budget to the fixed side, which is the whole point and also the whole risk. Hypothetical numbers:
| Component | Before | After | Change |
|---|---|---|---|
| System prompt | 900 | 900 | — |
| Tool definitions | 2,000 | 2,000 | — |
| Resident reference block | 0 | 1,400 | +1,400 |
| Conversation history | 3,000 | 3,000 | — |
| Retrieved passages | 5,200 | 3,800 | −1,400 |
| Output reservation | 1,900 | 1,900 | — |
| Total | 13,000 | 13,000 | — |
At roughly 500 tokens per passage, that block cost you between two and three passages on every request, including the ones where the schema was irrelevant. Whether that is a good trade depends entirely on condition 1 above, and it is arithmetic rather than opinion:
resident_cost = block_tokens × all_requests
fetched_cost = block_tokens × requests_where_fetched
+ failure_cost × requests_where_needed_but_missedResidency wins when the miss rate is high enough that failure_cost dominates, or when the fetch was
going to happen almost every time anyway. Estimate the miss rate from logs, not from intuition — it is
usually higher than people expect, because the requests where the schema was needed but the question
did not mention it are exactly the requests nobody inspects.
Where it goes
Place a resident block immediately after the tool definitions and before anything that varies per request. Two reasons.
It belongs in the cacheable prefix. A block that changes weekly, sitting above the volatile content, is paid at the cached rate on almost every request — which materially changes the arithmetic above and is the strongest argument for residency when your provider supports prefix caching. See designing a prompt prefix that stays stable.
And it sits near the start of the input, which is one of the positions models use most reliably. That suits reference material the model must apply rather than merely quote.
One exception worth knowing: volatile fields do not belong in the resident block. The current date, the user’s identity, and the session context change per request or per user, so putting them inside the cached block destroys it. Render them separately, below the prefix. It is tempting to keep all “context about the situation” in one place; resist it, because that one place is the most expensive place to put a changing value.
Keep it from growing
A resident block is a fixed cost with no natural limit, and it will attract additions. Every one is plausible: one more abbreviation, one more edge-case rule, one more table.
Three controls:
A token ceiling, asserted in a test. The block must stay under N tokens. Exceeding it requires removing something or raising the ceiling on purpose.
An owner and a reason per entry. Same discipline as auditing a system prompt nobody trimmed: a line nobody can justify is a line that gets deleted at the next review.
A periodic ablation. Remove the block, run your evaluation set, and confirm quality drops. If it does not, the block is not earning its seat — the method is in how much of your window is earning its place.
Compact it, too. A schema written for humans is several times larger than one written for the model: drop comments, drop unused tables and columns, use the shortest unambiguous names, and prefer a compact line-per-entity form over pretty-printed structure. The same per-item arithmetic as what formatting overhead costs you applies, and a resident block is where it pays best because you pay it on every request.
Per-profile residency
Residency does not have to be all-or-nothing across your application. If you have request-type profiles, the resident set can differ by profile: the reporting route carries the schema, the support route carries the glossary, neither carries both. That satisfies condition 1 within each route where it failed across the whole application.
The cost is one cacheable prefix per profile, so keep the number small — the same conclusion fixed shares or priority order reaches about profiles generally.
What to measure
- Resident block tokens, tracked with your other fixed costs, as a trend. Growth is the failure mode.
- Requests where the block was demonstrably used — cited, or referenced in the answer. Your check on condition 1.
- Passages displaced. Block tokens divided by your average passage size, so the cost is expressed in the currency you gave up rather than in tokens.
- Quality with and without, on a schedule and after every model change.
- Cache hit rate on the prefix containing it. If it is not being cached, residency is more expensive than you modelled it.
- Miss rate before residency, captured before you change anything. Without that baseline you cannot tell later whether the block fixed something or just cost something.
The rule
A permanent seat goes to material that is small, stable, needed nearly always, and unreliable to fetch — placed in the cacheable prefix, capped by a test, and re-ablated on every model change. Everything else should compete per request against the shares in the budget, and lose when it deserves to.