The summarization call replays the conversation prefix for KV-cache reuse
摘要调用回放对话前缀以复用 KV Cache
Automatic compaction fires mid-conversation, right after the loop has warmed the provider's KV cache with the last routed request (`system` + `tools` + derived history). The default summarizer then issued a *separate* auxiliary request whose prefix shared nothing with that warm request: a bespoke summarizer `system` prompt followed by the older history flattened to a single rendered transcript string. A provider cach
English
Problem
Automatic compaction fires mid-conversation, right after the loop has warmed the provider's KV cache with the last routed request (system + tools + derived history). The default summarizer then issued a separate auxiliary request whose prefix shared nothing with that warm request: a bespoke summarizer system prompt followed by the older history flattened to a single rendered transcript string. A provider caches on the request's leading token sequence, so a first token that differs — a different system prompt — invalidates the entire cached prefix. Every compaction therefore paid full prompt-processing cost for the whole replayed history twice: once for the conversation request that tripped pressure, and again for the summarization call, defeating the cache exactly when the conversation is largest.
Decision
The summarization directive moves from the front of the request (a fresh system prompt) to the end of the conversation (the final user message). The auxiliary call now reproduces the last routed request's prefix verbatim and appends one trailing instruction, so it is a genuine prefix-extension of the warm request and the provider reuses the cached tokens.
SummarizationInput carries the replayed prefix, not a rendered string
summarize() (and the internal summarizeWithLlm) take a SummarizationInput — { system?, tools?, messages } — instead of a flat transcript string. region.ts builds it from session.requestHeader() (the durable system and tools) plus the shadowed region mapped through session.deriveEventMessage, which yields byte-identical Message objects to what deriveMessages() folded into the routed request. summarizeWithLlm forwards system and tools onto GenerateOptions and sends [...input.messages, { role: 'user', content: COMPACTION_INSTRUCTION }]. tools ride along even though the summarizer never calls one: dropping them would shorten the token sequence and break alignment with the cached request.
The instruction is a trailing user message
COMPACTION_INSTRUCTION opens "You are now acting as a compaction engine…" and directs the model to condense the conversation ABOVE. It keeps the prior checkpoint's structured headings and adds two rules the front-loaded system prompt did not need in its new position: do not mention the summarization request, and output only the checkpoint text without calling a tool. The shadowed region always ends on a tool-pairing-balanced boundary, so appending a user message after it is a valid message ordering for OpenAI-compatible and DeepSeek adapters.
Cache reuse is best-effort, correctness is not
Auto-compaction always anchors at the surface head, so the shadowed region is the head of the routed request and the replayed prefix matches it exactly — the guaranteed-hit case. Manual mid-range compactRegion still replays the true prefix and stays correct, but forgoes reuse because its shadowed region is not the request head. A configured summarizationProvider/summarizationModel that differs from the conversation's route also forgoes reuse; that is the deployment's explicit trade-off, not a defect. Target resolution (configured override → latest routed header → agent options, else throw) is unchanged.
Alternatives considered
- Keep the summarizer system prompt but reuse the rest — rejected: the system slot is the very first token region a provider caches on, so a distinct summarizer system prompt invalidates the whole prefix regardless of what follows. Only moving the directive off the front recovers the cache.
- Send only the shadowed region without the
system/toolshead — rejected: a differently-headed sequence still diverges from the cached request at the first token, so it caches no better while losing the framing the summary needs. - Omit
toolsfrom the summarization request (the model never calls one) — rejected: tool schemas are part of the cached token sequence; omitting them misaligns every following token and defeats reuse. - A dedicated
assistant/chunk-emitting summarization sub-session for snapshot replay — rejected: the durablecompaction/summaryevent records the successful local call's position and complete output, while its explicit call marker prevents replay from treating template or remote output as a local stream.
Consequences
dsh-compaction-basicownsSummarizationInput; the protectedsummarize(input, agent, signal?)hook signature changed (acceptable pre-release), andregion.tsgainedbuildSummarizationInputfoldingderiveEventMessageover the shadowed seqs behind the header prefix.- Dead render surface removed. The old flattening path (
renderTranscript/renderContentBlocksand its spec indsh-compaction) had no remaining consumer and was deleted with its export. - README model experience for
dsh-compaction-basicnow documents the auxiliary request as the replayed prefix plus a trailing compaction-instruction message, and its KV-cache effect as reuse of the warm conversation prefix. - The framed checkpoint output is unchanged, so the landed
user/messageand every conversation-request snapshot are unaffected; only the auxiliary request's shape changed.
Testing
- Unit:
compaction-basic.spec.tsasserts the auxiliary call forwardssystem/tools/leading messages and appends the compaction instruction as the final message, and thatcompactRegionreplays the latest routed header prefix. Existing content assertions read the summarizer input through the replayed messages rather than a transcript string. - Loop:
compact-loop-repro.spec.tsclassifies the summarization request by the compaction instruction in its trailing user message, and the overflow-recovery tests continue to pin conversation-vs-summary request counts across the real loop. - Snapshot: keyless replay reconstructs one canonical successful stream from a marked
compaction/summary; the compaction-seam note owns the durable marker contract.
中文
问题
自动压缩（compaction）在对话中途触发，恰好在循环用最后一个已路由请求（system + tools + 派生历史）预热了提供方的 KV Cache 之后。随后默认摘要器发出一个独立的辅助请求，其前缀与那个已预热请求没有任何共享部分：一个专门的摘要器 system 提示词，后接被拍平成单个渲染后 transcript（文本记录）字符串的较早历史。提供方基于请求起始的 token 序列做缓存，因此第一个 token 只要不同（即一个不同的系统提示词），整个已缓存前缀就会失效。于是每次压缩都要为整段回放的历史付出两次完整的提示词处理成本：一次用于触发压力的对话请求，另一次用于摘要调用，恰好在对话最大时让缓存失去作用。
决策
摘要指令从请求的前端（一个全新的 system 提示词）移到对话的末尾（最后一条 user 消息）。辅助调用现在逐字复现最后一个已路由请求的前缀，并追加一条尾部指令，因此它是已预热请求的真正前缀扩展，提供方会复用已缓存的 token。
SummarizationInput 携带回放的前缀，而非渲染后的字符串
summarize()（以及内部的 summarizeWithLlm）接受一个 SummarizationInput（{ system?, tools?, messages }）而不是一个扁平的 transcript 字符串。region.ts 用 session.requestHeader()（持久的 system 和 tools）加上经 session.deriveEventMessage 映射的被遮蔽区域来构建它，后者产出与 deriveMessages() 折叠进已路由请求的内容字节级一致的 Message 对象。summarizeWithLlm 把 system 和 tools 转发到 GenerateOptions，并发送 [...input.messages, { role: 'user', content: COMPACTION_INSTRUCTION }]。tools 会一同带上，即便摘要器从不调用任何工具：丢弃它们会缩短 token 序列，破坏与已缓存请求的对齐。
指令是一条尾部 user 消息
COMPACTION_INSTRUCTION 以 "You are now acting as a compaction engine…" 开头，指示模型浓缩上方的对话。它保留先前检查点的结构化标题，并在其新位置上新增了两条前置系统提示词此前不需要的规则：不要提及摘要请求，以及只输出检查点文本而不调用任何工具。被遮蔽区域总是结束在工具配对平衡的边界上，因此在其后追加一条 user 消息，对 OpenAI 兼容适配器和 DeepSeek 适配器而言是合法的消息排序。
缓存复用是尽力而为，正确性则有保证
自动压缩总是锚定在表层头部，因此被遮蔽区域就是已路由请求的头部，回放的前缀与之完全匹配，这就是保证命中的情形。手动的中段 compactRegion 仍然回放真实的前缀并保持正确，但会放弃复用，因为它的被遮蔽区域不是请求头部。配置的 summarizationProvider/summarizationModel 若与对话的路由不同，也会放弃复用；这是部署方明确的权衡，而非缺陷。目标解析（配置的覆盖值 → 最新的已路由 header → agent（智能体）选项，否则抛出）保持不变。
考虑过的替代方案
- 保留摘要器系统提示词但复用其余部分——否决：system 槽位正是提供方最先做缓存的 token 区域，因此一个不同的摘要器系统提示词无论后面跟着什么都会使整个前缀失效。只有把指令移离前端才能恢复缓存。
- 只发送被遮蔽区域而不带
system/tools头部——否决：头部不同的序列在第一个 token 处仍然与已缓存请求分叉，因此缓存效果并不更好，反而丢失了摘要所需的框架。 - 从摘要请求中省略
tools（模型从不调用任何工具）——否决：工具 schema 是已缓存 token 序列的一部分；省略它们会让后续每个 token 失去对齐，破坏复用。 - 为快照回放专门建立一个发出
assistant/chunk的摘要子会话——否决：持久的compaction/summary事件会记录成功本地调用的位置和完整输出，而显式调用标记可防止回放把模板或远程输出当作本地流。
后果
dsh-compaction-basic拥有SummarizationInput；受保护的summarize(input, agent, signal?)钩子签名发生变化（发布前可接受），并且region.ts新增了buildSummarizationInput，它在 header 前缀之后对被遮蔽的 seq 折叠deriveEventMessage。- 移除无用的渲染表面。 旧的拍平路径（
renderTranscript/renderContentBlocks及其在dsh-compaction中的 spec）已无消费方，连同其导出一并删除。 - README 的 Model Experience 现在把
dsh-compaction-basic的辅助请求记述为回放的前缀加上一条尾部压缩指令消息，并把其 KV Cache 效果记述为复用已预热的对话前缀。 - 带框架的检查点输出未改变，因此落地的
user/message和每个对话请求快照都不受影响；只有辅助请求的形状发生了变化。
测试
- 单元：
compaction-basic.spec.ts断言辅助调用转发system/tools/前导消息，并把压缩指令作为最后一条消息追加，且compactRegion回放最新的已路由 header 前缀。现有的内容断言通过回放的消息而非 transcript 字符串来读取摘要器输入。 - 循环：
compact-loop-repro.spec.ts依据摘要请求尾部 user 消息中的压缩指令对其分类，溢出恢复测试则继续在真实循环中固定对话请求与摘要请求的数量。 - 快照： 无密钥回放会从带标记的
compaction/summary重建一条规范成功流；compaction-seam Agent Note 负责持久标记约定。