Select, execute and retry tests#

Inspect a catalog before choosing a subset. Selection uses resolved test IDs and inherited metadata; recognized definitions and needed inherited settings validate only when execution reaches a selected test. Unselectable files report uncertain coverage; invalid reached definitions are recorded/skipped while independent tests continue.

Choose a subset#

node tooling/testing/plan.mts inspect --catalog <project>/tests
node tooling/testing/plan.mts start --catalog <project>/tests --output <new-run-directory> --runtime <project>/testing-runtime.json --tag smoke

Without tag/ID selection, all tests run. Use qualified --ids CATALOG:SECTION/ID,... to name exact scenarios. The first-run guide selects one maintained fixed-save test, avoiding the deliberately broader sample smoke selection.

Keep start alive while the coordinator services requests. Tests start in filename traversal order. The run uses fresh agents and isolated browser contexts; fixture cleanup remains necessary even with isolation.

Retain partial work#

The checkpoint is the authoritative snapshot. Completed progress survives execution failure. Keep output and raw-evidence/ private until policy review produces a separate export. Do not treat missing screenshots as a passing attempt merely because an agent described the expected behavior.

Cancellation records a marker; the coordinator must stop/finish the worker and service cleanup requests. An interrupted coordinator can reconnect to pending work in the still-running process. Process-restart resumption of in-memory fixtures is not promised.

Retry explicitly#

node tooling/testing/plan.mts start --catalog <project>/tests --output <existing-run-directory> --runtime <project>/testing-runtime.json --retry --ids catalog:profile/profile-save-roundtrip

Earlier attempts remain. The resolved execution revision must match, and fixtures are prepared again. Default any-failure summary keeps earlier failures visible; use an intentionally selected policy when integrating modules directly. Changed definitions or legacy revisions require a new run/directory.

Review#

Use review and promotion to inspect all attempts and produce a bug report. Runner reference and canonical execution guidance contain command and checkpoint detail.

With several applications, the coordinator asks the human which application even when a default exists. Within that application include all configured catalogs unless explicitly narrowed. Case-insensitive tags support all/any/exclude. A no-match selection executes nothing and exposes alternatives and unselectable coverage gaps; changing the scope requires a new explicit selection.