Tagged “caching”
-
Where in the prompt things should go
Position affects whether the model uses what you put there. A default layout, why instructions get repeated at the end, and what prompt caching constrains.
-
What deserves a permanent seat in the window
Some material is needed on nearly every request. The test for making it a resident line item instead of fetching it, and the cost of getting it wrong.
-
When compaction collides with your cache
Compacting history rewrites the prompt above what you are appending, so it discards your cached prefix. How to choose thresholds that pay for that.
-
Designing a prompt prefix that stays stable
Prompt caching rewards byte-identical prefixes. What to put in yours, what silently invalidates it, and how caching changes what is worth trimming.