Blog
-
Context budgets for multi-turn conversations
History grows every turn until it crowds out everything else. Window policies, structured state extraction, and why agent loops overflow fastest.
-
Where in the prompt things should go
Position affects whether the model uses what you put there. A default layout, why instructions get repeated at the end, and what prompt caching constrains.
-
What to cut first when context overflows
A cut order for when the budget doesn't balance: which component gives up space first, why history usually loses to retrieval, and how to degrade visibly.
-
Counting tokens before you spend them
Why character-count estimates drift, what message formatting adds that you didn't count, and how to build a budget check that fails before the API does.
-
The context window is a budget
Five things compete for every request's window, including the answer. How to write the allocation down, and why the output reservation is forgotten.
-
What deserves a permanent seat in the window
Some material is needed on nearly every request. The test for making it a resident line item instead of fetching it, and the cost of getting it wrong.
-
Giving each agent its own window
One window shared by a long task overflows. Splitting work across separate windows with a handoff is an allocation strategy, and it has real costs.
-
How much of your window is earning its place
Every component costs tokens on every request. How to find which ones change the answer, using ablation and citation coverage rather than intuition.
-
Size the budget for the tail, not the average
Average requests never overflow. The failures live at p99, where one oversized item or a long conversation breaks a budget built on means.
-
What formatting overhead costs you
Delimiters, labels and serialisation are paid per item in the prompt. How to price the per-passage wrapper and keep the parts that earn their tokens.
-
Rebudgeting when you change model
A budget is model-specific. Token counts, limits, output caps and position behaviour all shift on a swap. The checklist for porting an allocation.
-
Fixed shares or priority order?
Two ways to divide a window: give each component a percentage, or fill in priority order. Where each fails, and why request types need their own profiles.
-
Do your few-shot examples still earn their space?
Examples are often the largest block in a system prompt and the least examined. How to run the ablation, and what to do with each outcome.
-
When compaction collides with your cache
Compacting history rewrites the prompt above what you are appending, so it discards your cached prefix. How to choose thresholds that pay for that.
-
Designing a prompt prefix that stays stable
Prompt caching rewards byte-identical prefixes. What to put in yours, what silently invalidates it, and how caching changes what is worth trimming.
-
The output format is a budget decision
Verbose response formats spend the window twice — once on the schema, once on the answer. How to price a format and pick a cheaper one.
-
Budgeting for tokens you never see
Reasoning traces, retries and discarded drafts consume window and billing without appearing in the answer. How to find them and reserve for them.
-
Reserving room for the answer
Input and output share the window on most models. How to size the output reservation, what an unset cap means, and why truncation looks like success.
-
Auditing a system prompt nobody trimmed
System prompts grow one patch at a time and nothing is ever removed. How to audit yours by section, delete safely, and keep it from regrowing.
-
Tool definitions are a line item
The tool block is paid on every request whether or not a tool is called. How to audit it per tool, what to trim, and what terser schemas cost you.