How the summarizing of the conversation happens in an LLM?
back
1 comments
Instead of prompting "Summarize: $SESSION" you submit "$SESSION Summarize what has been done so far".
This way you have a prefix cache hit on the session and don't pay any more than for a regular prompt.
This seems to be a very intuitive take, but people often miss on the technicalities.
The prompt is assembled as tool schemas, then system instructions, then message history, and caching works on prefixes of that. So the cost of an edit is not its size, it is the size of everything behind it. Edit a tool definition and you have invalidated the system prompt and the whole conversation with it. Edit the tail and you pay for the tail.
I measured this by accident in a multi agent ablation where the only variable was whether the supervisor sent a fixed tool array or a per-task subset to the worker. Runs paying cache creation, out of 120 each: fixed 0 cache creation, per-task subset 58 cache creation. Under a prompt load that was $0.0382 per run against $0.0230, and on a clean context it reversed because there was no prefix worth caching.
Which is the argument for what Pi does here. A pointer is a tail edit. Rewriting is a head edit.