2026-06-11proposedtesting

Mutation testing as the coverage counterweight

变异测试作为覆盖率的制衡手段

The per-file 100% coverage gate ([the quality-gates decision](../../implemented/process/2026-06-11-quality-gates.md)) proves every line *executes* under test — not that any assertion would notice if the line were wrong. Under agent-written tests, coverage pressure can produce execution-without-assertion. Mutation testing measures what coverage cannot: whether the suite *kills* deliberately injected bugs.

.agents/notes/proposed/testing/2026-06-11-mutation-testing.md

English

Problem

The per-file 100% coverage gate (the quality-gates decision) proves every line executes under test — not that any assertion would notice if the line were wrong. Under agent-written tests, coverage pressure can produce execution-without-assertion. Mutation testing measures what coverage cannot: whether the suite kills deliberately injected bugs.

Proposal

Stryker (@stryker-mutator/vitest-runner) over packages/*/src:

PR-scoped incremental runs (changed files only) as a CI job — fast enough to gate merges once tuned.
Nightly full runs with a tracked mutation score; start by recording, then set the threshold at the observed baseline and ratchet upward (same policy as coverage: thresholds only ever tighten).
Surviving mutants are work items: an agent picks a survivor, writes the killing test, repeats — a well-shaped autonomous loop.
Equivalent mutants (provably behavior-preserving) get annotated exclusions with reasons, mirroring the /* v8 ignore */ policy.

Plan

Add Stryker config scoped to one package (llm — smallest, most algorithmic) and measure runtime.
Expand to all packages; record baseline scores in the config.
Wire the nightly job; add the incremental PR job once runtime is acceptable.

Acceptance criteria

A Stryker config runs over packages/*/src with the vitest runner; a nightly job records the mutation score, and a ratcheting threshold fails the run when the score drops below the recorded baseline.
PR-scoped incremental runs gate merges once runtime is acceptable — or are explicitly kept nightly-only, with that outcome recorded here.
Equivalent mutants carry annotated exclusions with reasons, mirroring the /* v8 ignore */ policy.

Risks

Runtime: mutation testing is expensive; per-file 100% coverage helps (every mutant is at least reached). If PR-scoped runs stay too slow, keep them nightly-only and rely on the score ratchet.

中文

问题

逐文件 100% 覆盖率门禁（质量门禁决策）证明每一行代码在测试中都被执行了，但不能证明如果该行出错，任何断言会注意到。在 agent（智能体）编写测试的场景下，覆盖率压力可能产出「执行但不断言」的测试。变异测试衡量的正是覆盖率无法衡量的：测试套件是否能杀死被刻意注入的缺陷。

提案

使用 Stryker（@stryker-mutator/vitest-runner）对 packages/*/src 运行变异测试：

PR（Pull Request）范围的增量运行（仅变更文件），作为一个 CI job。调优后速度足以作为合并门禁。
每夜全量运行，跟踪变异分数；先记录基线，再将阈值设为观测到的基线并只升不降（与覆盖率策略一致：阈值只收紧）。
存活的变异体是待办项：agent 选取一个存活体、编写杀死它的测试、循环往复——一个适合自主执行的闭环。
等价变异体（可证明不改变行为的）加注释排除并附理由，与 /* v8 ignore */ 策略一致。

计划

添加 Stryker 配置，范围限定在一个包，即 llm（最小、最具算法性），并测量运行时间。
扩展到所有包；在配置中记录基线分数。
接入每夜 job；运行时间可接受后再添加 PR 范围的增量 job。

验收标准

Stryker 配置在 packages/*/src 上以 vitest runner 运行；每夜 job 记录变异分数并采用只升不降的阈值，分数低于已记录的基线时任务失败。
PR 范围的增量运行在运行时间可接受后作为合并门禁；否则明确决定只保留每夜运行，并将该结论记录于此。
等价变异体带有注释排除及理由，与 /* v8 ignore */ 策略一致。

风险

运行时间：变异测试开销大；逐文件 100% 覆盖率有所帮助（每个变异体至少会被执行到）。如果 PR 范围的运行始终过慢，则保持仅每夜运行，依靠变异分数阈值的只升不降机制。