Web multimodal image input and durable attachments
Web 多模态图片输入与持久附件
Before this change, the Web composer accepted only text: `InputBar` received a string draft, `ConversationController.send()` created text content, and the host forwarded that content to the agent. Users could not paste an image, inspect it before sending, submit an image-only prompt, or recover sent images from history. This is not only a composer gap. Core needs a durable image content block, providers need explicit
English
Problem
Before this change, the Web composer accepted only text: InputBar received a string draft, ConversationController.send() created text content, and the host forwarded that content to the agent. Users could not paste an image, inspect it before sending, submit an image-only prompt, or recover sent images from history.
This is not only a composer gap. Core needs a durable image content block, providers need explicit modality handling, and the session log must reconstruct everything visible to a model. The previous image-block removal rejected a partial design that could silently lose or flatten images. A browser object URL, local path, provider URL, or base64 payload cannot be canonical session content.
The Web client architecture keeps components pure and per-session composer state in ctx.conversation; the GUI layering and RPC protocol makes durable events the source of truth for both live rendering and history replay. Image intake, persistence, provider conversion, and rendering therefore need one explicit lifecycle.
Peer products converge on an attachment rail above the editor, but their storage choices differ. Codex-style paths such as /var/folders/.../codex-clipboard-*.png are reasonable intake staging locations, not durable message identities: the operating system may delete them, another host cannot read them, and a resumed session cannot rely on them.
Decision
Pasted or dropped raster images are the Web composer's first consumer of a durable attachment capability. Unsent files remain temporary client-owned draft state. Every rich-content intake adapter decodes its wire blocks, proves route capability, and delegates the complete image batch to the attachment service before appending its message event. A provider adapter that produces structured image output must durably commit the output before appending its assistant block. Canonical user and assistant content contains only role-neutral ImageBlock references.
Version one supports PNG, JPEG, WebP, and GIF paste and drag-and-drop, image-only or mixed prompts, historical user and assistant image rendering, and original-image preview on a single click (display and interaction specifics superseded in part by the attachment-display alignment note). File picking, generic files, PDF, audio, video, image copying, and a custom context menu remain separate follow-ups.
Product behavior
- Pasting or dropping one or more supported images adds ordered thumbnails above the textarea without inserting placeholder text. Dragging files over the composer highlights the drop target.
- The same resident
InputBarrenders the rail in both blank-session Hero and active-session layouts. The rail is hidden when empty and scrolls horizontally instead of widening the composer. - Each 64-by-64-pixel thumbnail carries a hover-revealed remove control inside the card and opens its original draft image on a single click; overflow pages with edge arrows instead of a visible scrollbar.
- A prompt may contain text and images or images only. Pure text paste remains native browser behavior; mixed clipboard content inserts its text normally while adding its files to the rail, and file-only paste prevents default browser handling. File drops on the composer always prevent browser navigation and report unsupported files locally.
- A failed send restores the complete text and image draft without clobbering text or images added while the request was in flight. Removal, successful send, session-scope disposal, rendered-history disposal, and application disposal revoke the object URLs they own.
- Historical user and assistant images use one
MessageImagecontrol. Inline images preserve intrinsic aspect ratio, do not upscale, and stay within a 240-by-240-pixel box. - Clicking a message image opens the stored original in a viewport-bounded modal. Escape, the close control, and backdrop activation close it and restore focus.
- Version one does not override the browser context menu and provides no explicit image-copy action.
Storage lifecycle and ownership
The persistence boundary is message acceptance, not paste:
| State | Allowed representation | Durability and ordering |
|---|---|---|
| Unsent user draft | Browser File plus object URL; a native client may use an OS temporary file such as /var/... | Temporary and client-owned. It may disappear on reload or process exit and never appears in a session event. |
| Accepted user image | Immutable object below DSH_HOME plus ImageAttachmentRef | The host commits every image before agent.send() or agent.steer() can append the owning user event. |
| Structured model image output | Immutable object below DSH_HOME plus ImageAttachmentRef | The provider adapter commits the bytes before it emits a completed image block or assistant message event. Temporary URLs, paths, and base64 are forbidden in the event. |
Each session's InputMachine state keeps the ordered runtime-only attachment identifiers alongside the live draft. The framework-owned chat store receives only the draft's plain-text persistence mirror, while ConversationController owns the corresponding browser-only File and object-URL registry:
import type { Branded } from '@deepseek-ai/dsh-brand'
type DraftAttachmentId = Branded<'DraftAttachmentId'>
interface ChatStoreState {
selection: object | null
draft: string
view: string | null
}
interface InputState {
draft: string
imageIds: readonly DraftAttachmentId[]
}
interface ComposerAttachment {
kind: 'image'
id: DraftAttachmentId
file: File
previewUrl: string
}
This split uses the session provide channel's input hook and actions as the single subscription path for live composer state while keeping non-serializable browser objects out of persisted JSON. Only the plain-text draft mirror uses localStorage; attachment identifiers, browser File objects, and object URLs remain scoped to the live session input shell. Unsent images therefore do not survive reload or session-scope disposal. A Workspace switch moves a mixed text-and-image draft only when the destination shell accepts the complete image batch; refusal leaves both parts with the source. A native client may stage input in an OS temporary directory, but it must treat that path exactly like the browser object URL: delete it when no longer needed and copy the bytes into the durable store before message acceptance.
The local attachment backend resolves an explicit dshHome, then $DSH_HOME, then ~/.dsh. It stores content-addressed objects below $DSH_HOME/attachments/v1/objects/<prefix>/<sha256> with owner-only directory and file permissions. On each process's first save for one home, it creates that home and synchronizes every ancestor entry to the filesystem root; existence is not treated as durability because another process may still be between mkdir and parent fsync. A temporary file is then written, synchronized, atomically published, and made durable with directory syncs on the publication path (POSIX; Windows relies on filesystem metadata journaling) before the service returns a reference. The content digest is encoded in the opaque sha256:<digest> identifier. Admission prepares a provider-independent master by applying orientation, removing metadata, converting to 8-bit sRGB/sRGBA, and preserving aspect ratio under independent dimension and byte limits. Reads verify the digest, byte length, and logged metadata. Route-specific deterministic request versions are cached separately; the full policy is recorded in Unified image masters, request versions, and provider files.
The store performs no automatic deletion in version one. Sent user images and model-generated images remain reachable for history, resume, and fork. Reference-aware garbage collection needs a separate design because an age-only rule can delete data still referenced by a durable session. Deployment byte and pixel limits are admission policy on writes; reads verify the digest and recorded metadata without reapplying current admission limits, so lowering policy does not invalidate older history.
Durable content and prompt wire
The attachment seam exposes immutable image write and verified read operations. The canonical metadata is deliberately narrower than a generic file record:
import type { Branded } from '@deepseek-ai/dsh-brand'
type AttachmentId = Branded<'AttachmentId'>
interface ImageAttachmentRef {
attachmentId: AttachmentId
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
bytes: number
width: number
height: number
name?: string
}
interface ImageBlock {
type: 'image'
attachment: ImageAttachmentRef
}
ImageBlock joins the merge-extensible core ContentBlockMap and is valid in either user or assistant content. It never carries base64, an object URL, a filesystem path, or a provider-owned locator. This keeps the session event plus immutable object store sufficient to reconstruct the exact model-visible image. The LLM vocabulary therefore has a type-only dependency on the attachment seam; provider runtime dependencies remain adapter-specific.
The browser cannot mint a durable reference, so session.prompt accepts a narrow intake union rather than canonical ContentBlock[]:
export {}
type PromptInputPart =
| { type: 'text'; text: string }
| {
type: 'image'
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
data: string
name?: string
}
Base64 crosses a wire boundary once and is discarded after persistence. Each front door validates canonical base64 and declared MIME fields, then calls AttachmentStore.saveImages() with the whole decoded batch. The service owns image count, aggregate bytes, individual bytes, fully decoded raster/MIME agreement, intrinsic dimensions, decoded-pixel count, and master preparation. It prepares and verifies every batch member once before publishing any member, so one malformed image cannot create partial references and large images are not decoded and encoded again at commit. Storage commits then run in submission order. If a later storage I/O operation fails, the caller appends no model-visible event and receives no partial references, but an earlier immutable content-addressed object may remain unreferenced under the existing storage rule. Only after every image succeeds does the front door call the agent with normalized text and durable image blocks in wire order. A failure exposes no attachment path or raw bytes.
session.attachment is a read-only, session-scoped endpoint. The host serves bytes only when a durable event in that session references the requested attachment identifier. The client deduplicates loads by session and attachment identifier while that session is rendered, revokes resolved URLs on rendered-session disposal, and rejects invalidated late loads before allocating an object URL so an unmounted session or disposed service cannot repopulate the cache.
Model capabilities and provider behavior
Model catalog entries gain optional merge-extensible input modality declarations. A missing declaration means unknown; a present list without image is an explicit negative capability.
The host is the authoritative preflight point. It resolves the session's latest routed provider and model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects a new image prompt before writing an attachment or event, and the client restores the draft. Image-bearing prompt admission and model selection share one per-agent serial chain (ordering decision), including steering that does not enter the queued UI mirror. This gives a prompt and concurrent selection a deterministic order. Selection itself may target a text-only model after images enter durable history; the shared LLM runtime replaces retained image blocks with deterministic text placeholders for that request. session.updateQueue edits accept text content only, so a queue edit cannot inject an image past admission. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. The browser rejects unsupported declared image media types before allocating preview URLs, but it does not snapshot deployment limits or model capability. The host validates the complete batch against current byte, count, aggregate, media, dimension, pixel, and routed-model policy before writing an attachment or event; its rejection appears through the composer's transient toast.
Pi-AI and the direct DeepSeek adapter resolve ctx.attachments at request time, recursively convert each retained image reference including references nested inside tool results, and emit native image content only for models that declare image input. Both adapters request the same deterministic route-specific version from the durable normalized attachment. Pi-AI carries it inline under a base64-aware request budget. The built-in DeepSeek route advertises deepseek-v4-flash-vision-exp, uploads every retained version through Files API, and sends file_id blocks with indexed reuse, expiry, bounded stale-id retry, quota cleanup, and explicit deletion. DeepSeek text models, custom models without an image declaration, and unlisted pass-through ids remain text-only. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. No adapter may flatten or silently skip a retained image; unsupported roles and models fail with typed UNSUPPORTED_CONTENT.
Core supports structured assistant image blocks, but no current production provider route is certified for image output. Any future output-capable adapter must retrieve provider bytes under bounded size and time policy, validate them through the same attachment service, persist them, and only then publish the atomic ImageBlock. A URL in assistant Markdown remains text and is never downloaded automatically.
Provider-neutral token estimation does not guess visual pricing from image dimensions; provider-reported usage remains authoritative. ACP advertises image prompts only when its configured exact route and attachment deployment can accept them, persists inline input before publishing the user event, and re-reads committed assistant image references for native ACP image updates. MCP keeps canonical raw blocks for programmatic callers while projecting admitted images to durable core blocks; Code Mode carries any settled image-bearing sub-result through the outer result as logged source-attributed context.
Compaction replays the selected conversation prefix, including image references, into the configured summarization route. A visual-capable route uses the same deterministic request versions as ordinary turns. A text-only route receives the same deterministic attachment placeholders as any other LLM request. The synthesized checkpoint remains text-only, and compaction-basic rejects image summary output with UNSUPPORTED_CONTENT.
History rendering and original preview
History folding preserves ImageBlock in both user and assistant messages. User images align to the trailing edge above their text; assistant images remain in their original content-block position in the leading narration flow. MessageImage derives a stable inline box from recorded dimensions, resolves bytes through the session-authorized loader, uses object-fit: contain, and turns a missing or corrupt object into a retryable error control.
Composer thumbnails and each MessageImage own ephemeral original-preview state and invoke the same pure ImageLightbox. The modal uses the already resolved original object URL, constrains only display size, focuses its close control, and restores the previous focus target when closed.
Limits and trust boundaries
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Source intake defaults are 32 MiB per image, 20 images and 100 MiB aggregate image bytes per message, 100 million decoded pixels per image, and 16384px on either side. The provider-independent master defaults to a 2048px long edge and 4 MiB safety cap. Provider request pixel and encoded-byte limits are separate route policies. These deployment-varying limits are validated backend configuration and enforced before persistence or request transmission. The client connection carrier has an independent configurable maxRequestBodyBytes cap, 160 MiB by default, and fails load if it cannot hold the aggregate source limit after base64 and envelope expansion. A body without a declared length is rejected when it crosses the cap rather than drained to its end.
Malformed base64, unsupported or mismatched media, truncated image payloads, excess bytes, excess image count, excess pixels, excess per-side dimensions, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser.
Package and surface changes
| Surface | Responsibility |
|---|---|
packages/attachment/attachment | Opaque attachment and request-version identifiers, image references, policies, failures, batch admission, derived reads, and crops through ctx.attachments. |
packages/attachment/attachment-local | Private content-addressed masters, deterministic request cache, complete raster decoding, integrity verification, and configuration. |
packages/llm/llm | Role-neutral ImageBlock, input-modality metadata, exact adapter generations, and text-only request projection. |
packages/llm/llm-pi-ai | Resolve durable images to deterministic inline request versions. |
packages/llm/llm-deepseek | Resolve official vision input to deterministic request versions and Files API ids. |
packages/compaction/compaction-basic | Preserve images in summary input and reject non-text checkpoint output explicitly. |
packages/host/apiproxy and packages/bundle/base | Narrow upload wire, shared batch admission, limits and routed-model preflight, persist-before-event ordering, session-authorized reads, and default profile composition. |
packages/client/connection and packages/client/runtime | Bounded request buffering, wire types, fixture images, prompt uploads, attachment reads, and durable-reference folding. |
packages/client/ui-conversation | Per-session draft images, attachment rail, user and assistant image controls, and original preview. |
packages/acp/acp | Conditional native image capability, atomic inline-image admission, and verified assistant-image delivery. |
packages/mcp/mcp-client | Lossless canonical MCP results plus capability-gated durable image projection and explicit diagnostics for unsupported rich blocks. |
packages/core/tools | Generic Code Mode forwarding of settled image-bearing sub-results after the outer result. |
The attachment packages form the interface/implementation side of one capability seam. Composer behavior stays in the conversation object layer, provider conversion stays in adapters, and no change is required in agent-loop.
Implementation
The implemented capability includes shared prepare-once batch admission, provider-independent masters, deterministic request versions, DeepSeek Files reuse, stable crop handles, role-neutral image blocks, Pi-AI and DeepSeek input conversion, durable Web/ACP/MCP ordering, Web upload/read protocol, conditional ACP image support, lossless MCP results with durable image projection, Code Mode rich-result forwarding, bounded Web requests, draft and historical image UI, compaction handling, and keyless assembled coverage.
No compatibility shim is required for the pre-release prompt wire; all call sites and fixtures change with the introducing slice.
Alternatives considered
Keep every intake image in /var or another temporary directory
Temporary storage is appropriate before send, including for a native client that receives clipboard files through the operating system. It is not appropriate after acceptance: cleanup is outside the harness's control, paths are host-specific, and resume or fork can outlive the file. The proposal permits temporary staging but copies accepted bytes into DSH_HOME before the event.
Persist immediately on paste or drop
Immediate persistence makes drafts reload-resistant but creates durable objects before a session or message owns them, which requires quota, orphan lifetime, and cleanup policy. Version one keeps the unsent draft temporary and makes send acceptance the durability boundary.
Inline base64 in messages and session logs
This duplicates binary data across RPC, events, history pages, forks, compaction, and browser storage, and invites token accounting to treat encoding text as model text. One immutable object plus small references keeps the durable representation bounded.
Use browser object URLs, local paths, or provider URLs as canonical content
Object URLs expire with the document, local paths are not portable, and provider URLs may expire, track viewers, or expose credentials. They remain temporary transport or preview details only.
Use one generic AttachmentBlock for images, files, audio, and video
Composer presentation can use a generic attachment rail, but provider semantics are modality-specific. Images are native multimodal input; PDFs may be provider files or extracted text; video may be native, sampled, or unsupported. A specific ImageBlock forces every consumer to handle or reject the modality explicitly.
Rely on UI capability checks or silently filter images
UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalogued model paths. Silent filtering changes user intent. Provider enforcement remains mandatory, while UI checks are optional earlier feedback.
Add a generic RichContent service above the core content vocabulary
Rejected because the core already has the role-neutral ContentBlock vocabulary and attachment references. A second generic service would duplicate ordering, capability, logging, and lifetime semantics while still requiring each wire adapter to parse its own protocol. Narrow image adapters around the existing core preserve ownership and leave audio/resources to earn their own lifecycle contracts.
Normalize MCP results into core content as the canonical tool value
Rejected because Code Mode and programmatic callers need the complete MCP JSON blocks and optional structuredContent; replacing that value with a Native projection would make the bridge lossy. MCP retains the protocol value and prepares a separate model projection, with final post-execute policy remaining authoritative.
Perform attachment reads and writes inside synchronous output renderers
Rejected because tool renderers are pure, synchronous, and replayable. MCP prepares image projection during async execution and installs it only at the registry's finalization boundary; ACP performs async admission and output conversion in its transport lifecycle. Code Mode forwarding observes the already settled final content instead of giving individual image tools private parent-token behavior.
Testing
- Storage tests cover content-addressed deduplication, private permissions, admission failures, corruption/missing-object failures, and reading history after deployment limits are lowered.
- Host and protocol tests cover persist-before-event ordering, absence of base64 in logs, session-scoped authorization, capability rejection, upload limits, bounded HTTP request bodies, image-admission/model-selection ordering, text-only queue edits, and text-only request projection.
- Client unit tests cover paste and drop, mixed clipboard text, image-only send, draft restoration, ordering, draft/session-scope/application object-URL cleanup, and a deferred historical read that completes after disposal; the keyless assembled built-client lane (
apps/web/tests/image-display.snapshot.ts,DSH_EXAMPLE_MODE=lib pnpm run test:snapshot) covers the historical user and assistant galleries over the authorized attachment route, the original-size lightbox, and the composer paste rail. - Adapter and compaction tests cover deterministic Pi-AI request versions, DeepSeek Files upload and reuse, stale-id recovery, text-only projection, recursively nested tool-result images, shared summary request versions, and explicit image-output rejection.
- Attachment, MCP, ACP, and Code Mode tests cover all-member validation before writes, mixed text/image ordering, no inline base64 in durable events, exact route-capability gates, explicit unsupported-content diagnostics, post-execute replacement/block precedence, cancellation during admission, verified assistant-image delivery, and generic nested-image forwarding. A keyless assembled ACP snapshot sends a real inline PNG and pins only its durable reference in the session log.
- Credentialed real-API tests cover the configured Anthropic route and the built-in
deepseek-officialFiles path. The DeepSeek test does not use a custom provider entry. - The current production adapter set has no certified image-output route; output-provider certification remains outside version one.
Consequences
- Durable storage grows without garbage collection. Version one chooses replay safety over premature deletion.
- A missing or corrupt object makes exact model reconstruction fail. Failing loud preserves integrity but may prevent that session from continuing until repaired.
- JSON-RPC base64 adds upload memory and roughly one-third encoding overhead. Version-one limits bound it; larger media needs streaming or a binary transport.
- Unsent images do not survive reload. Durable drafts need quota and orphan cleanup rather than reusing message storage implicitly.
- Original preview decodes more pixels than the inline control displays. Pixel limits, one clicked preview, and object-URL disposal bound but do not eliminate transient browser memory.
- Capability metadata may be missing or stale. Host preflight improves feedback, while adapter enforcement remains authoritative.
- A future output provider may require authenticated retrieval before an assistant image can complete, adding latency and a new failure point. Persist-before-event ordering favors replay integrity.
- File picking, generic files/PDF, audio/video, durable draft staging, image copying, custom context menus, output-provider certification, and reference-aware garbage collection remain independent designs.
中文
问题
在此变更之前,Web 输入区仅接受文本:InputBar 接收字符串草稿,ConversationController.send() 创建文本内容,宿主再把该内容转发给 agent(智能体)。用户无法粘贴图片、在发送前查看图片、提交仅含图片的提示词,也无法从历史记录中恢复已发送图片。
这不只是输入区功能缺失。核心层需要持久图片内容块,提供方需要明确处理模态,会话日志则必须重建模型可见的全部内容。此前移除图片块的决策否决了可能静默丢失图片或将其展平的不完整设计。浏览器对象 URL、本地路径、提供方 URL 或 base64 数据都不能成为规范会话内容。
Web 客户端架构要求组件保持纯粹,并将每个会话的输入区状态放在 ctx.conversation 中;GUI 分层与 RPC 协议则要求持久事件成为实时渲染与历史回放的共同真源。因此,图片接收、持久化、提供方转换和渲染需要遵循同一个明确的生命周期。
同类产品普遍在编辑器上方设置附件栏,但存储方案各不相同。诸如 /var/folders/.../codex-clipboard-*.png 的 Codex 式路径适合作为接收输入时的暂存位置,却不能作为持久消息身份:操作系统可能删除文件,另一台宿主无法读取文件,恢复后的会话也不能依赖文件仍然存在。
决策
粘贴或拖放的光栅图片是 Web 输入区对持久附件能力的首个应用场景。未发送文件仍是由客户端持有的临时草稿状态。每个丰富内容接入适配器都会解码自身协议块、证明路由能力,并在追加消息事件前把完整图片批次委托给附件服务。生成结构化图片输出的提供方适配器在追加相应助手块前,也必须持久提交输出。规范用户内容与助手内容只包含角色无关的 ImageBlock 引用。
第一版支持粘贴和拖放 PNG、JPEG、WebP 与 GIF,支持仅图片或混合提示词,支持渲染历史用户图片与助手图片,并支持单击预览原图(展示与交互细节部分由附件展示对齐 Note取代)。文件选择、通用文件、PDF、音频、视频、图片复制和自定义上下文菜单仍分别作为后续工作。
产品行为
- 粘贴或拖放一张或多张受支持的图片后,文本框上方会按顺序显示缩略图,但不会插入占位文本。文件拖入输入区时会高亮放置目标。
- 同一个常驻
InputBar会在空白会话 Hero 和活跃会话布局中渲染附件栏。附件栏为空时隐藏,通过横向滚动避免撑宽输入区。 - 每个缩略图为 64 × 64 像素,移除按钮位于卡片内部、悬停时显示;单击打开草稿原图,溢出用两端箭头翻页而非可见滚动条。
- 提示词可同时包含文本与图片,也可仅包含图片。粘贴纯文本时保持浏览器原生行为;粘贴混合的剪贴板内容时,文本会正常插入,文件则同时添加到附件栏;仅粘贴文件时才阻止浏览器的默认处理。在输入区放置文件时总会阻止浏览器导航,并在本地报告不受支持的文件。
- 发送失败时恢复完整的文本与图片草稿,但不会覆盖请求飞行期间新增的文本或图片。移除、发送成功、会话 scope 释放、已渲染历史记录释放和应用释放都会撤销各自持有的对象 URL。
- 历史用户图片与助手图片共用一个
MessageImage控件。行内图片保持固有宽高比、不放大,并限制在 240 × 240 像素的边界框内。 - 单击消息图片会在不超出视口的模态框中打开存储的原图。按 Escape、激活关闭控件或激活背景区域都会关闭模态框并恢复焦点。
- 第一版不覆盖浏览器上下文菜单,也不提供明确的图片复制操作。
存储生命周期与归属
持久化边界是消息被接受,而不是图片被粘贴:
| 状态 | 允许的表示 | 持久性与顺序 |
|---|---|---|
| 未发送的用户草稿 | 浏览器 File 加对象 URL;原生客户端可以使用 /var/... 等操作系统临时文件 | 临时且由客户端持有。它可能在重载或进程退出后消失,绝不出现在会话事件中。 |
| 已接受的用户图片 | DSH_HOME 下的不可变对象加 ImageAttachmentRef | 在 agent.send() 或 agent.steer() 能够追加所属用户事件前,宿主提交每张图片。 |
| 结构化模型图片输出 | DSH_HOME 下的不可变对象加 ImageAttachmentRef | 提供方适配器在发出已完成的图片块或助手消息事件前提交字节。事件中禁止出现临时 URL、路径和 base64。 |
每个会话的 InputMachine 状态在实时草稿旁保存仅限运行时的有序附件标识符。框架持有的 chat store 只接收草稿的纯文本持久化镜像,ConversationController 则持有相应的浏览器专用 File 与对象 URL 注册表:
import type { Branded } from '@deepseek-ai/dsh-brand'
type DraftAttachmentId = Branded<'DraftAttachmentId'>
interface ChatStoreState {
selection: object | null
draft: string
view: string | null
}
interface InputState {
draft: string
imageIds: readonly DraftAttachmentId[]
}
interface ComposerAttachment {
kind: 'image'
id: DraftAttachmentId
file: File
previewUrl: string
}
这一拆分把会话 provide 通道的输入 hook 与 actions 用作实时输入区状态的唯一订阅路径,同时避免把不可序列化的浏览器对象写进持久 JSON。只有纯文本草稿镜像使用 localStorage;附件标识符、浏览器 File 对象和对象 URL 都限定在实时会话输入外壳的 scope 内。未发送图片因此无法跨重载或会话 scope 释放保留。切换 Workspace 时,只有目标外壳接受完整图片批次,图文混合草稿才会移动;拒绝时,文本和图片都留在来源外壳。原生客户端可以在操作系统临时目录中暂存输入,但必须像对待浏览器对象 URL 一样对待该路径:不再需要时删除,并在消息被接受前把字节复制进持久存储。
本地附件后端依次解析显式 dshHome、$DSH_HOME 和 ~/.dsh。它把内容寻址对象存储在 $DSH_HOME/attachments/v1/objects/<prefix>/<sha256> 下,并为目录和文件设置仅所有者可访问的权限。每个进程首次为某个 home 保存对象时,都会创建该 home,并逐级同步每个祖先目录项直至文件系统根目录;不能把存在视为持久性,因为另一个进程可能仍处于 mkdir 与父目录 fsync 之间。随后,服务写入并同步临时文件,再以原子方式发布,并对发布路径执行目录同步使其持久(POSIX;Windows 依赖文件系统元数据日志),之后才返回引用。内容摘要编码在不透明的 sha256:<digest> 标识符中。准入会应用方向、删除元数据、转换为 8-bit sRGB/sRGBA,并在独立尺寸和字节上限内保持宽高比,生成与提供方无关的主版本。读取会校验摘要、字节长度和已记录元数据。路由专用的确定性请求版本单独缓存,完整策略见统一图片主版本、请求版本和提供方文件。
第一版不对存储执行自动删除。已发送的用户图片和模型生成图片会一直保留,以供历史记录、恢复和 fork 使用。按引用感知的垃圾回收需要单独设计,因为仅按时间清理可能删除仍被持久会话引用的数据。部署的字节和像素限制是写入时的准入策略;读取时会校验摘要和已记录的元数据,但不重新应用当前准入限制,因此收紧策略不会导致旧历史记录失效。
持久内容与提示词协议
附件 seam公开不可变图片写入和经过校验的读取操作。规范元数据刻意比通用文件记录更窄:
import type { Branded } from '@deepseek-ai/dsh-brand'
type AttachmentId = Branded<'AttachmentId'>
interface ImageAttachmentRef {
attachmentId: AttachmentId
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
bytes: number
width: number
height: number
name?: string
}
interface ImageBlock {
type: 'image'
attachment: ImageAttachmentRef
}
ImageBlock 加入可合并扩展的核心 ContentBlockMap,在用户内容和助手内容中都有效。它绝不携带 base64、对象 URL、文件系统路径或提供方持有的定位符。因此,会话事件与不可变对象存储足以共同重建模型可见的确切图片。LLM 词汇由此仅在类型层面依赖附件 seam;提供方运行时依赖仍由各适配器持有。
浏览器无法生成持久引用,因此 session.prompt 接受范围狭窄的接收联合类型,而不是规范 ContentBlock[]:
export {}
type PromptInputPart =
| { type: 'text'; text: string }
| {
type: 'image'
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
data: string
name?: string
}
Base64 只跨越一次协议边界,并在持久化后丢弃。每个入口都会校验规范 base64 与声明的 MIME 字段,再用完整解码批次调用 AttachmentStore.saveImages()。服务负责图片数量、总字节数、单张图片字节数、声明 MIME 与完整解码后的光栅图片是否一致、固有尺寸、解码像素数和主版本准备。它会在发布任何成员之前只准备并验证每个批次成员一次,因此一张畸形图片不会产生部分引用,大图也不会在提交时重复解码和编码。随后按顺序提交存储。如果后续存储 I/O 操作失败,调用方不会追加模型可见事件,也不会收到部分引用,但先前的不可变内容寻址对象可能按现有存储规则保持无引用状态。只有每张图片都成功后,入口才会用规范化文本和按协议顺序排列的持久图片块调用 agent。失败时不公开任何附件路径或原始字节。
session.attachment 是只读且限定于会话作用域的端点。只有该会话中的持久事件引用了所请求的附件标识符,宿主才提供字节。会话处于渲染状态时,客户端会按会话和附件标识符对加载操作去重;已渲染会话释放时会撤销已解析的 URL,并在分配对象 URL 前拒绝已失效的延迟加载,以免已卸载的会话或已释放的服务重新写入缓存。
模型能力与提供方行为
模型目录项增加可选且可合并扩展的输入模态声明。缺少声明表示未知;声明存在但不含 image,则明确表示不支持图片。
宿主是权威的前置检查点。它会解析会话最新路由到的提供方和模型,并在缺失时依次回退到 agent 选项和宿主默认值;如果模型明确排除图片输入,宿主会在写入附件或事件前拒绝新的图片提示词,客户端则恢复草稿。包含图片的提示词准入与模型选择共用一条逐 agent 串行链(顺序决策),也包括不进入排队 UI 镜像的 steering。这会为提示词和并发选择提供确定顺序。图片进入持久历史后仍可选择纯文本模型;共享 LLM 运行时会在该请求中把保留的图片块替换为确定的文本占位符。session.updateQueue 只接受文本内容,因此队列编辑无法绕过准入注入图片。能力未知时继续进入适配器强制检查,使未收录的模型标识符仍然可用。浏览器会在分配预览 URL 前拒绝声明不支持的图片媒体类型,但不会为部署限制或模型能力保留快照。宿主会根据当前的单张字节数、图片数量、总字节数、媒体类型、尺寸、像素数和路由模型策略校验整个批次,再写入附件或事件;拒绝会通过 composer 的短时 toast 显示。
Pi-AI 与直接 DeepSeek 适配器都会在请求时解析 ctx.attachments,递归转换每个保留的图片引用,包括嵌套在工具结果中的引用,并且仅为声明支持图片输入的模型生成提供方原生图片内容。两个适配器都从持久主版本请求同一个确定性路由版本。Pi-AI 在考虑 base64 扩张的请求预算内内联携带它。内置 DeepSeek 路由公布 deepseek-v4-flash-vision-exp,把每个保留的版本上传到 Files API,并通过索引复用、过期处理、有界陈旧 ID 重试、配额清理和显式删除发送 file_id 块。DeepSeek 纯文本模型、未声明图片能力的自定义模型和未列出的透传 ID 保持纯文本。在请求时解析服务,可避免 Cordis 加载顺序将可选附件服务的可用性固化。适配器不得展平或静默跳过保留图片;不支持的角色与模型会以类型化的 UNSUPPORTED_CONTENT 失败。
核心层支持结构化助手图片块,但当前没有任何生产提供方路径通过图片输出认证。未来任何支持输出的适配器都必须在有界的大小和时间策略下获取提供方字节,通过同一个附件服务校验并持久化字节,之后才能以原子方式发布 ImageBlock。助手 Markdown 中的 URL 仍是文本,绝不自动下载。
提供方无关的 token 估算不会根据图片尺寸猜测视觉定价;提供方返回的用量仍是权威值。只有配置的确切路由与附件部署可以接受图片时,ACP(Agent Client Protocol)才公布图片提示词能力;它会在发布用户事件前持久化内联输入,并重新读取已提交的助手图片引用来发送原生 ACP 图片更新。MCP 为程序化调用方保留规范原始块,同时把已准入图片投影为持久核心块;Code Mode 会把任何已经结算且含图片的子结果经外层结果转运为带来源归属且写入日志的上下文。
压缩会把选定的会话前缀和其中的图片引用回放到已配置的摘要生成路径。支持视觉的路径使用与普通轮次相同的确定性请求版本。纯文本路径接收与其他 LLM 请求相同的确定性附件占位符。合成的检查点仍仅包含文本,compaction-basic 会以 UNSUPPORTED_CONTENT 拒绝包含图片的摘要输出。
历史渲染与原图预览
历史记录折叠会在用户消息和助手消息中保留 ImageBlock。用户图片在文本上方靠尾端对齐;助手图片则保留在靠前端叙述流中的原内容块位置。MessageImage 根据记录的尺寸派生稳定的行内边界框,通过会话授权加载器解析字节,使用 object-fit: contain,并将对象缺失或损坏转换为可重试的错误控件。
输入区缩略图和每个 MessageImage 各自持有临时原图预览状态,并调用同一个纯 ImageLightbox。模态框使用已经解析的原始对象 URL,只限制显示尺寸;它会聚焦关闭控件,并在关闭时把焦点恢复到先前的目标。
限制与信任边界
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。源文件输入默认限制为每张图片 32 MiB、每条消息 20 张图片和 100 MiB 图片总字节数、每张图片一亿解码像素,以及任一边 16384px。与提供方无关的主版本默认长边 2048px,独立安全上限 4 MiB。提供方请求的像素和编码字节上限是单独的路由策略。这些随部署变化的限制属于经过校验的后端配置,并在持久化或请求发送前强制执行。客户端连接载体为每个 API 请求设置独立且可配置的 maxRequestBodyBytes 上限,默认 160 MiB;如果该上限无法容纳源文件总量限制经 base64 和请求封装膨胀后的大小,加载就会失败。未声明长度的请求体在越过上限时即被拒绝,而不是先读完再拒。
格式错误的 base64、不支持或不匹配的媒体、截断的图片数据、超出字节限制、超出图片数量、超出像素限制、超出单边尺寸限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段,控制字符会被移除,并且任何本地路径都不会写入日志或返回浏览器。
包与接口变更
| 接口 | 职责 |
|---|---|
packages/attachment/attachment | 不透明附件和请求版本标识符、图片引用、策略、错误,以及通过 ctx.attachments 提供的批量准入、派生读取和裁剪。 |
packages/attachment/attachment-local | 私有内容寻址主版本、确定性请求缓存、完整光栅解码、完整性校验和配置。 |
packages/llm/llm | 角色无关的 ImageBlock、输入模态元数据、精确适配器代次和纯文本请求投影。 |
packages/llm/llm-pi-ai | 把持久图片解析为确定性内联请求版本。 |
packages/llm/llm-deepseek | 把官方视觉输入解析为确定性请求版本和 Files API ID。 |
packages/compaction/compaction-basic | 在摘要输入中保留图片,并明确拒绝非文本检查点输出。 |
packages/host/apiproxy 和 packages/bundle/base | 范围狭窄的上传协议、共享批量准入、限制和路由模型前置检查、先持久化再追加事件的顺序、会话授权读取,以及默认 profile 组合。 |
packages/client/connection 和 packages/client/runtime | 有界请求缓冲、协议类型、fixture(测试前置数据)图片、提示词上传、附件读取和持久引用折叠。 |
packages/client/ui-conversation | 每个会话的草稿图片、附件栏、用户与助手图片控件和原图预览。 |
packages/acp/acp | 条件式原生图片能力、原子内联图片准入,以及经过校验的助手图片交付。 |
packages/mcp/mcp-client | 无损规范 MCP 结果、经能力门禁的持久图片投影,以及针对不受支持丰富块的明确诊断。 |
packages/core/tools | 在外层结果之后通用转发已经结算且含图片的 Code Mode 子结果。 |
附件包构成一个能力 seam 的接口与实现侧。输入区行为留在会话对象层,提供方转换留在适配器中,无需修改 agent-loop。
实现
已实现能力包括只准备一次的共享批量准入、与提供方无关的主版本、确定性请求版本、DeepSeek Files 复用、稳定裁剪句柄、角色无关图片块、Pi-AI 和 DeepSeek 输入转换、Web/ACP/MCP 持久化顺序、Web 上传与读取协议、条件式 ACP 图片支持、带持久图片投影的无损 MCP 结果、Code Mode 丰富结果转发、有界 Web 请求、草稿与历史图片 UI、压缩处理,以及组装后的无密钥覆盖。
预发布提示词协议不需要兼容包装层;引入相应切片时会同时修改所有调用点和 fixture。
曾考虑的替代方案
将每张粘贴图片保留在 /var 或其他临时目录中
临时存储适合在发送前使用,也适合通过操作系统接收剪贴板文件的原生客户端。但它不适合在消息被接受后继续使用:清理不在 harness 的控制范围内,路径因宿主而异,恢复或 fork 后的会话也可能比文件存在得更久。提案允许临时暂存,但会在追加事件前将已接受的字节复制进 DSH_HOME。
粘贴或拖放后立即持久化
立即持久化可以让草稿在重载后继续存在,但会在会话或消息持有对象前就创建持久对象,因此必须定义配额、遗留对象生命周期和清理策略。第一版保持未发送草稿为临时状态,并把发送被接受作为持久性边界。
在消息与会话日志中内联 base64
这种方式会在 RPC、事件、历史分页、fork、压缩和浏览器存储中复制二进制数据,还会诱使 token 计量把编码文本当成模型文本。单一不可变对象配合小型引用,可以让持久表示保持有界。
使用浏览器对象 URL、本地路径或提供方 URL 作为规范内容
对象 URL 会随文档失效,本地路径不可移植,提供方 URL 则可能过期、跟踪查看者或暴露凭据。它们只能作为临时传输或预览细节存在。
用一个通用 AttachmentBlock 表示图片、文件、音频和视频
输入区展示可以使用通用附件栏,但提供方语义取决于具体模态。图片是原生多模态输入;PDF 可能是提供方文件或提取后的文本;视频可能由模型原生支持、抽帧处理或不受支持。特定的 ImageBlock 会迫使每个消费方明确处理或拒绝该模态。
依赖 UI 能力检查或静默过滤图片
UI 状态可能陈旧,也无法保护直接 SDK、ACP、回放或未收录模型的路径。静默过滤会改变用户意图。提供方强制检查仍是必需项,UI 检查则是可选的提前反馈。
在核心内容词汇之上添加通用 RichContent 服务
不予采用,因为核心已经拥有角色无关的 ContentBlock 词汇与附件引用。第二套通用服务会重复顺序、能力、日志和生命周期语义,同时每个协议适配器仍需解析自身协议。围绕现有核心构建范围狭窄的图片适配器,可以保持归属清晰,并让音频/资源在确有需要时建立自己的生命周期契约。
把 MCP 结果规范化为核心内容,并将其作为规范工具值
不予采用,因为 Code Mode 和程序化调用方需要完整 MCP JSON 块及可选 structuredContent;用 Native 投影替换该值会让桥接有损。MCP 保留协议值,并另行准备模型投影;最终 post-execute 策略仍具有权威性。
在同步输出渲染器中执行附件读写
不予采用,因为工具渲染器必须纯净、同步且可回放。MCP 在异步执行期间准备图片投影,只在注册表最终化边界安装;ACP 在自己的传输生命周期中执行异步准入和输出转换。Code Mode 转发观察已经结算的最终内容,而不是让各图片工具各自处理私有父 token 行为。
测试
- 存储测试覆盖内容寻址去重、私有权限、准入失败、对象损坏或缺失时的失败,以及收紧部署限制后读取历史数据。
- 宿主与协议测试覆盖先持久化再追加事件的顺序、日志中不含 base64、会话作用域授权、能力拒绝、上传限制、大小受限的 HTTP 请求体、图片准入与模型选择的排序、仅文本的队列编辑,以及纯文本请求投影。
- 客户端单元测试覆盖粘贴与拖放、混合剪贴板文本、仅图片发送、草稿恢复、顺序、草稿、会话作用域和应用层级的对象 URL 清理,以及一项在释放后才完成的延迟历史读取;keyless 的组装后构建产物通道(
apps/web/tests/image-display.snapshot.ts,DSH_EXAMPLE_MODE=lib pnpm run test:snapshot)覆盖经授权附件路由渲染的历史用户与助手图片画廊、原图 lightbox,以及 composer 粘贴缩略图条。 - 适配器与压缩测试覆盖确定性 Pi-AI 请求版本、DeepSeek Files 上传与复用、陈旧 ID 恢复、纯文本投影、递归嵌套在工具结果中的图片、共享摘要请求版本,以及明确拒绝图片输出。
- 附件、MCP、ACP 与 Code Mode 测试覆盖写入前校验全部成员、图文混合顺序、持久事件不含内联 base64、确切路由能力门禁、明确的不支持内容诊断、post-execute 替换/阻止优先级、准入期间取消、经过校验的助手图片交付,以及通用嵌套图片转发。组装后的无密钥 ACP 快照发送真实内联 PNG,并在会话日志中只固定其持久引用。
- 需要凭据的实际 API 测试会覆盖配置的 Anthropic 路由和内置
deepseek-officialFiles 路径。DeepSeek 测试不使用自定义提供方条目。 - 当前生产适配器集合没有经过认证的图片输出路由;输出提供方认证仍不在第一版范围内。
后果
- 持久存储会在没有垃圾回收时持续增长。第一版选择回放安全,而不是过早删除。
- 对象缺失或损坏会让模型请求无法精确重建。明确失败可以保持完整性,但在修复前可能阻止该会话继续运行。
- JSON-RPC base64 会增加上传内存,并带来约三分之一的编码开销。第一版的限制可以约束开销;更大的媒体需要流式传输或二进制传输协议。
- 未发送图片无法跨重载保留。持久草稿需要配额和遗留对象清理,而不是隐式复用消息存储。
- 原图预览解码的像素多于行内控件显示的像素。像素限制、一次只打开一个预览和对象 URL 释放可以约束但无法消除浏览器瞬时内存占用。
- 能力元数据可能缺失或陈旧。宿主前置检查可以改善反馈,适配器强制检查仍是权威结果。
- 未来输出提供方可能需要经过身份认证的下载,助手图片才能完成,这会增加延迟与新的故障点。先持久化再追加事件的顺序优先保障回放完整性。
- 文件选择、通用文件与 PDF、音频与视频、持久草稿暂存、图片复制、自定义上下文菜单、输出提供方认证和按引用感知的垃圾回收仍是相互独立的设计。