Parallel pre-push gates
并行 pre-push 门禁
Aggregate jobs such as documentation synchronization hide long sequential chains whose members are read-only and independent. Duplicating their leaf inventory in workflow YAML gives future script changes multiple places to drift, while running package publication checks serially makes one gate consume time proportional to the package count.
English
Problem
Aggregate jobs such as documentation synchronization hide long sequential chains whose members are read-only and independent. Duplicating their leaf inventory in workflow YAML gives future script changes multiple places to drift, while running package publication checks serially makes one gate consume time proportional to the package count.
Decision
scripts/run-gates.ts owns the bounded scheduler used by CI, doc-sync, and the opt-in check:all command. It expands named modes into leaf gates, rejects empty or ambiguous dependency graphs before starting a child, respects artifact dependencies, buffers attributable output by default, reports exit and signal outcomes independently, and accepts DSH_GATE_CONCURRENCY when a caller needs a different worker bound. A needs edge requires the predecessor to pass and skips its dependent otherwise; an after edge waits for any terminal outcome and then permits the follower to run. A gate marked allowFailure still reports its result but does not fail the aggregate.
Long coordinator gates whose own subprocesses preserve useful attribution may opt into streamOutput. Their stdout and stderr reach the parent immediately without being buffered or printed again at completion. Partitioned coverage and parallel Web snapshots use this mode so a mid-run failure is visible without waiting for sibling work.
The Node 24 consumer job is one ten-gate mode rather than a shell-owned process pool. Its default worker count equals its gate count, while pull-request CI caps active gates at eight and dependencies control readiness. Build and source compatibility start immediately; after build, publint and built-package invariant validation run in parallel. Lint, both snapshot suites, documentation typechecking, NodeNext type checks, and built-bin smokes wait for the invariant validator to remove its temporary package views.
scripts/publint-all.ts discovers packages from packages/<group>/<pkg> and runs publint with a worker pool sized from availableParallelism(). DSH_PUBLINT_CONCURRENCY can cap or raise the worker count for local machines and CI runners with different resource profiles. Results are buffered per package and printed in deterministic package order, so parallel execution does not scramble each package's log block.
The per-gate package scripts remain the vocabulary for ad hoc local runs. hygiene invokes a scheduler mode containing the same thirteen checks with the local four-worker cap, while doc-sync owns its member list in the scheduler (doc-sync through the gate scheduler).
Verification
scripts/run-gates.spec.ts rejects invalid graphs before the executor runs, pins pass-required and settle-only ordering, pins the hygiene, consumer, and native Windows inventories and their failure semantics, exercises signal termination through a real child process, and proves that streamed output is immediate and unbuffered. scripts/publint-all.spec.ts rejects a missing public export before downstream artifact consumers run.
Alternatives considered
- Keep aggregate jobs serial — simpler execution but makes wall clock equal the sum of independent checks and repeats command-wrapper startup.
- Declare one CI job per leaf gate — exposes maximum workflow parallelism but repeats checkout, setup, and install overhead and duplicates the scheduler inventory in YAML.
- Background subcommands inside shell scripts — parallelizes work but loses per-gate timing, deterministic failure grouping, and straightforward signal handling.
- Inherit stdio for every gate — exposes progress immediately but interleaves ordinary independent gates and discards the scheduler's attributable output record. Streaming remains an explicit gate property.
- Declare one
publintjob per package — exposes maximum package parallelism but creates a hand-maintained package inventory that drifts when packages change. - Run
publintwith unbounded concurrency — minimizes elapsed time on small repositories only by gambling with process count, memory pressure, package tarball creation, and readable logs.
Consequences
Scheduler-backed commands take the slowest dependency chain instead of the sum of independent gates and report the gate that dominates. Invalid graphs fail before partial execution. The cost is a custom scheduler with an explicit mode inventory.
The consumer validation chain delays validated-artifact consumers and lint until the shared artifact view is known-good and transient staging is gone; those downstream gates can still overlap one another. publint needs the build but not the staged validation view, so it overlaps the validator instead of extending that chain.
Most gates retain deterministic output blocks. Selected long coordinators trade cross-gate ordering and buffered logs for immediate diagnostics, while their final status remains available to the aggregate summary.
publint-all.ts is asynchronous and buffers command output instead of inheriting stdio live. The payoff is package-level parallelism with stable output order and one environment variable for resource tuning.
中文
问题
文档同步等聚合任务隐藏了很长的串行链，其中各项检查只读且相互独立。在工作流 YAML 中重复这些叶子清单，会使未来脚本变更有多个位置可以发生漂移；而串行运行包发布检查，会使一道门禁的耗时与包数量成正比。
决策
scripts/run-gates.ts 拥有 CI、doc-sync 和按需启用的 check:all 命令所使用的有界调度器。它将具名模式展开为叶子门禁，在启动子进程前拒绝空的或有歧义的依赖图，遵守产物依赖，默认缓冲可归因的输出，分别报告进程退出与信号终止结果，并在调用方需要不同 worker 上限时接受 DSH_GATE_CONCURRENCY。needs 边要求前置门禁通过，否则跳过依赖方；after 边只等待前置门禁以任意结果结算，随后仍允许后继门禁运行。标记为 allowFailure 的门禁仍会报告结果，但不会使聚合流程失败。
自身子进程能够保留有效归因的长时间协调门禁可以选择 streamOutput。其 stdout 与 stderr 会立即到达父进程，不会被缓冲，也不会在结束时重复打印。分区覆盖率与并行 Web 快照使用该模式，使运行中途的失败无需等待兄弟工作结束就能显示。
Node 24 消费方任务采用单个包含 10 道门禁的模式，而非由 shell 管理的进程池。其默认 worker 数等于门禁数，拉取请求 CI 则把活动门禁限制为 8 道，并由依赖关系控制就绪状态。构建与源码兼容性立即启动；构建完成后，publint 与已构建包不变式验证并行运行。lint、两套快照、文档类型检查、NodeNext 类型检查和 built-bin 冒烟测试等待不变式验证器清除临时包视图。
scripts/publint-all.ts 从 packages/<group>/<pkg> 发现包，并以根据 availableParallelism() 确定大小的 worker 池运行 publint。DSH_PUBLINT_CONCURRENCY 可以针对资源配置不同的本地机器和 CI runner 限制或提高 worker 数量。结果按包缓冲，并按确定性的包顺序打印，因此并行执行不会打乱各包的日志块。
各门禁的包脚本仍是临时本地运行所用的命令入口。hygiene 调用包含相同十三道检查且限制为本地四个 worker 的调度器模式，而 doc-sync 的成员列表由调度器管理（通过门禁调度器运行 doc-sync）。
验证
scripts/run-gates.spec.ts 在执行器运行前拒绝无效图，锁定必须通过与只等结算两种顺序，锁定 hygiene、消费方与原生 Windows 清单及其失败语义，通过真实子进程验证信号终止，并证明流式输出会立即显示且不被缓冲。scripts/publint-all.spec.ts 在下游产物消费方运行前拒绝缺失的公开导出。
曾考虑的替代方案
- 保持聚合 job 串行：执行更简单，但墙钟时间等于各独立检查之和，并重复启动命令包装器。
- 每个叶子门禁声明一个 CI job：暴露最大工作流并行度，但会重复 checkout、设置和安装开销，并在 YAML 中复制调度器清单。
- 在 shell 脚本内后台运行子命令：可以并行处理，但会失去各门禁计时、确定性的失败分组和直接的信号处理。
- 让所有门禁继承 stdio：可以立即显示进度，但会交错普通独立门禁的输出，并丢失调度器可归因的输出记录。流式输出仍是显式的门禁属性。
- 每个包声明一个
publintjob：暴露最大包级并行度，但会创建手工维护的包清单，包发生变化时就会漂移。 - 以无界并发运行
publint：虽能最大限度缩短小型仓库的耗时，却会拿进程数量、内存压力、包 tarball 创建开销和日志可读性冒险。
后果
由调度器支持的命令耗时取决于最慢的依赖链，而非各独立门禁耗时之和，并会报告决定总耗时的门禁。无效图会直接失败，不会先执行其中一部分。代价是维护一个具有显式模式清单的定制调度器。
这条验证链会让使用已验证产物的下游消费方和 lint 延后启动，直至共享产物视图经确认有效且临时暂存已清除；这些下游门禁仍可彼此重叠运行。publint 需要构建，却不依赖暂存的验证视图，因此它会与验证器重叠，而不会延长这条依赖链。
大多数门禁仍保留确定性的输出块。少数长时间协调器用跨门禁输出顺序和缓冲日志换取即时诊断，而其最终状态仍可供聚合摘要使用。
publint-all.ts 采用异步执行并缓冲命令输出，而不是实时继承 stdio。换来的是具有稳定输出顺序的包级并行，以及用于资源调节的单一环境变量。