# Markdown test execution This first workflow separates catalog definitions, retained test attempts, and bug reports. Skills coordinate the user's agent/browser tools; portable modules handle discovery semantics, hooks, validation, artifacts, and rendering. There is no mandatory T3 runtime. ## Start a run Use the repository's pinned Node/pnpm, install its lockfile, and build workspace modules. The runner is `tooling/testing/plan.mts`; `help` lists commands. An adopting project keeps its own catalog and configuration; the runner can remain in a separate SnagBadger checkout. ```sh node /tooling/testing/plan.mts inspect --catalog /tests node /tooling/testing/plan.mts start --catalog /tests --output --runtime /testing-runtime.json --tag smoke ``` Keep start running in a persistent session while the coordinating skill services bridge requests. The output directory must be new. A plan is a directory, not a separate manifest. Without a tag/ID selection, all tests run. Tags inherited from sections participate in selection. ## Test definition The [sample-site catalog](../../examples/test-plans/README.md) contains standalone tests and nested section files you can copy, inspect, and adapt. Discover `.snag.md` tests only. Supporting Markdown is ignored; optional `section.md` supplies shared settings/guidance, and directories form arbitrary-depth implicit sections. Bodies are free-form Markdown. Creation guidance recommends prerequisites, checkbox steps and expected outcomes; headings and steps are optional. Workers receive the original body, and only authored checkbox items become checklist steps. An explicit ID is optional and falls back to the filename without `.snag.md`. IDs are section-local. References use `catalog-id:section/local-id` (root: `catalog-id:local-id`). In filename order duplicates receive `-2`, `-3`, etc., skipping all occupied authored IDs; files are never rewritten to save suffixes. Identical local IDs in another section/catalog are distinct. Renames/new duplicates can change calculated references. Artifacts retain source path, body and resolved configuration; historical reports retain their recorded definitions. Existing catalogs must rename test files from `.md` to `.snag.md` deliberately; a regular `.md` file is now supporting documentation. Calculated references and execution revisions can change, so start a new run rather than forcing historical retry compatibility. This discovery change does not establish a general saved-artifact migration policy. ```markdown --- id: account-sign-in title: Sign in with a test account executor: browser tags: [smoke] components: [authentication] levels: [integration] --- ## Prerequisites A disposable staging account exists. ## Steps - [ ] Open the sign-in page. - [ ] Submit valid credentials. - [ ] Inspect the signed-in account. ## Expected outcomes The account page displays the expected test user. ``` Executors are browser, api, and cli. A section can supply the executor default. Explicit test values override it. Tags/components/levels are string lists merged as sets. Levels are project metadata, not directory depth. Unknown frontmatter is preserved without executable authority. Recognized fields validate only when execution reaches a selected test, together with the inherited settings it needs. Invalid reached definitions are recorded with file/reason and skipped; independent tests continue. Unreadable selection metadata is reported as unselectable coverage, rather than a pass. Overridden scalar settings need not validate; unrelated invalid branches are untouched. The `execution-v1-` revision fingerprints the resolved execution contract: body, authored checklist, extracted optional prerequisites/expectations, inherited instructions, executor, applicable evidence requirements, and declared run/section/test hooks. Titles, tags, ordering, and destinations do not affect compatibility; unrelated sections do not affect each other. This compact fingerprint is not a cryptographic integrity claim. Explicit retries require matching execution revisions; project hook implementations remain the project�s responsibility. Directories create sections even without section.md. Traversal follows filenames deterministically; numeric `order` has no precedence. Prefix filenames when start order matters. The current runner remains sequential; concurrent scheduling and lifecycle redesign belong to #154. ## Resolved lifecycle scope `resolveTestLifecycle({ sections }, test)` from `@snagbadger/agent-catalog` exposes root-to-leaf section ancestry and effective hook declarations with phase and section identity. It uses the already-resolved per-attempt inheritance in the test; run hooks belong only to the root. Missing ancestors, cycles, duplicate section identities, multiple roots and misplaced run hooks are rejected. The declaration list preserves nearest-section reference precedence; it is not an invocation schedule. Execution owns entry/exit transitions, reversed cleanup and failure policy. Catalog execution revisions, execution transitions and onboarding support review use this same scope, retaining existing execution revision identities. ## Optional section configuration ```markdown --- title: Accounts tags: [smoke] destination: { provider: jira, project: ACC } instructions: [Use disposable staging accounts.] evidence: browser: { screenshot: required, video: required } hooks: beforeEach: - { id: authenticate, function: authenticateUser } afterEach: - { id: remove-fixture, function: removeFixture } --- Use the staging environment. Keep test accounts independent. ``` The section body is inherited guidance. Instructions append root-to-leaf. Use `instructions: {replace: true, items: [New guidance]}` to replace inherited instructions. Tags match case-insensitively, accumulate, and support explicit removal with `tags: {remove: [smoke], items: [regression]}`. `components` and `levels` remain descriptive lists. Executor/retries use the nearest explicit value; evidence overrides individual settings. Root-to-parent guidance precedes one clearly emphasized “Test to execute” block; sibling bodies are excluded. Destination settings inherit as a provider/project pair and an explicit child destination replaces the pair. No credentials belong here. Evidence settings merge per executor and kind. Supported kinds are screenshot, video, diagnostic, recording, and file; requirements are required, optional, or off. Defaults require screenshot/video for browser tests and diagnostic output for API/CLI tests. Missing required proof makes the attempt incomplete; an observed behavioral failure remains recorded and can still create a bug. Optional missing evidence produces a warning. Hook diagnostics do not count as behavioral proof. ## Lifecycle hooks Run hooks live only in root section.md. Section hooks run once when that selected section is entered/exited. Test hooks inherit through the hierarchy and may be added in test frontmatter. - beforeRun / afterRun: shared environment setup and teardown. - beforeSection / afterSection: shared resources for that section. - beforeEach / afterEach: isolated test setup and cleanup. Setup proceeds root-to-leaf; cleanup reverses hook/section order and is attempted after setup/test errors. Cleanup functions must handle partially initialized resources and a null browser when context creation failed. Browser context creation precedes per-test hooks; video recording starts before them when supported. A once-per-section login does not authenticate each isolated context. Each hook declares a unique id and exactly one function or instruction. onFailure defaults to block; stop ends the run after cleanup. Inherited per-test hooks compose. Explicit overrides use: ```yaml hooks: beforeEach: disable: [authenticate] items: - { id: authenticate, instruction: Sign in through the UI. } afterEach: replace: true items: - { id: cleanup, function: customCleanup } ``` A function name is a registry reference, never inline executable YAML. The selected project module is trusted developer code. It receives runId, test, section, session, shared (a local Map for fixture state), and browser.perform(action) when a browser context exists. Shared values are not serialized into reports. ```js export function configureTesting({ redactReportSecrets }) { return { functions: { async authenticateUser(context) { if (!context.browser) throw new Error('Browser context unavailable'); await context.browser.perform({ type: 'navigate', url: process.env.TEST_LOGIN_URL, }); // Further actions use this same context. Keep secrets in the environment. }, }, reportPolicy: { id: 'project-evidence-review', reviewReport(report) { return redactReportSecrets( report, [process.env.TEST_SECRET].filter(Boolean), ); }, reviewAttachment(evidence, bytes) { // Replace with the project's actual image/video review policy. return { action: 'omit' }; }, }, }; } ``` The module may instead export functions and reportPolicy directly. The configureTesting factory receives existing library helpers so an adopting module need not duplicate serialization/redaction or resolve workspace packages. ## Runtime configuration ```json { "capabilities": { "name": "user-browser-mcp", "freshAgents": true, "newBrowserContexts": true, "contextProof": "Installed tool exposes isolatedContext; verified with distinct named contexts.", "screenshots": true, "video": false }, "module": "./testing-hooks.mjs" } ``` Capabilities are attestations from actual installed tool contracts/probes, not switches that create missing capabilities. Browser tests require a unique new context for every attempt; the engine rejects missing/reused context identities. The coordinator routes project-function actions to that exact handle. Read [runtime mappings and limitations](testing-runtimes.md). ## Host bridge Use `next --output DIR` to read the pending UUID request at `DIR/bridge/request.json`. Give the fresh worker that actual packet locator or a saved packet. Complete it with `complete --output DIR --response FILE`. Requests are local files, not an unauthenticated HTTP endpoint. ```json { "id": "", "ok": true, "value": null } ``` An unsuccessful operation uses ok false and a nonempty error string. Transport failures differ from a valid failed test. Required response values: | Method | Value | | -------------------------------------------- | ----------------------------------------------------------------------- | | createBrowserContext | {id: unique-context-id, browserContext: true, handle: JSON tool handle} | | browserAction | Serializable action result; target the supplied session | | instruction, startVideo, closeBrowserContext | null on success | | stopVideo | {evidence: Evidence, path: saved-video-path} | | executeTest | {attempt: Attempt, attachments: [{evidenceId, path}]} | The packet supplies runId, attemptId, one resolved test, and its session. Delegate executeTest with no inherited conversation; pass only the packet plus runner/output/request information and the execute-test skill. An Attempt uses id, startedAt, endedAt, outcome (passed/failed/blocked/incomplete), actual, hypotheses, steps, interactions, evidence, and reproduction. See the exported agent-reporting types and [portable reports](portable-reports.md). The runner owns final identity/timing. Empty collections and null optional records are explicit. ```json { "id": "", "ok": true, "value": { "attempt": { "id": "", "startedAt": "2026-10-11T05:00:00.000Z", "endedAt": "2026-10-11T05:00:01.000Z", "outcome": "passed", "actual": "The endpoint returned status 200.", "hypotheses": [], "steps": [], "interactions": [], "evidence": [ { "id": "response", "label": "Endpoint response", "kind": "diagnostic", "capturedAt": "2026-10-11T05:00:01.000Z", "content": { "type": "structured", "value": { "status": 200 } } } ], "reproduction": null }, "attachments": [] } } ``` The worker uses progress with the same response shape after meaningful steps. Files are copied into durable raw-evidence before they count as attachment proof; missing bytes become unavailable evidence. Completed progress is retained if execution fails. Raw artifacts are local private execution state, not reviewed exports. ## Completion, retries, and reporting `checkpoint.json` is the authoritative runner snapshot: `formatVersion: 1`, a validated `view` containing the shared run/result/attempt model, hierarchy, hook events, warnings and processing status, and an `attachments` index keyed by result/attempt/evidence identity. Attachment paths are relative to the output directory under `raw-evidence/`. Bytes are stored before references are published. One atomic replacement commits the complete snapshot; an interrupted replacement leaves the previous complete checkpoint readable. Request/progress files identify unfinished work. Finished results persist before the next test starts. Cancellation records a marker; the coordinator must stop/finish the active worker and service cleanup requests. An interrupted coordinator can reconnect to the pending request in the still-running process, or render saved partial output. Process-restart resumption of in-memory shared fixtures/hooks is not promised. Use start with --retry --ids and the original output directory for an explicit retry; earlier attempts stay intact. Run/section fixtures are prepared again. The production viewer shows eventual recovery as Flaky, separately from Passed. Programmatic default any-failure aggregation still keeps earlier failures visible. Read [automatic retry configuration and accounting](test-run-viewer.md#automatic-retries-and-flaky-results) for the run/section/test limits, pass ordering and terminal failures. Outputs from the earlier `run.json` / `execution.json` / `attachments.json` format remain readable by `report` when run and view agree. They are never rewritten or silently migrated. Retrying legacy output or legacy leaf-only definition revisions requires a new run and output directory. Unsupported or malformed checkpoints fail explicitly. Read [the production viewer and token contract](test-run-viewer.md) for portable resources, scale behavior, runtime counters and single-user bug export. The HTML report has a fixed layout, nested sections, title/tag/component/level search, outcome filters, planned expectations, recorded interactions, attempts, hook diagnostics, and image/video/diagnostic evidence. Report customization is excluded. The configured reportPolicy reviews the entire text/source snapshot and attachment bytes through shared report-export preparation, without generating an unused Bug-report viewer. Binary redaction, omission, unavailable proof, and narrative sanitization clear affected verification claims. Missing review leaves structured results intact and reports needs-review; use `report --output DIR --runtime FILE` once review is configured. Each reviewed export has a new directory, preventing stale/omitted attachments from leaking from prior exports. No raw fallback is allowed after review errors. Create bug selects a failed attempt (including an observed failure with incomplete proof), opens an editable local draft, and preserves destination metadata. Its report-owned single-user form edits content and evidence, exports a portable ZIP or saves an editable JSON manifest, and retains edits on export failure. It does not share a composer with the launcher or submit automatically. The create-test-bug skill packages reviewed evidence through the shared bug-report/provider interface. Live tracker authentication/transport remains provider work. ## Skills and verification Invoke the repository skill paths directly or install the folders into the host's skill search path: For guided onboarding and portable installed skill names, start with [guided setup](onboarding.md). The new setup/initialize-tests/create-test/test workflows reuse the contracts and coordinator below. - setup/initialize-testing - execution/run-test-plan - execution/execute-test - reporting/create-test-report - reporting/create-test-bug The independent skill test uses fresh agents on a disposable catalog; browser/adapter conformance tests use isolated local fixtures. Runtime-specific MCP compatibility requires installed tools; simulated tests do not prove a live MCP implementation. Run pnpm verify for the full production gate. ## Lazy catalog interfaces and selection `discoverCatalog(sources, {catalogId})` reads syntax/identity and returns entries and unselectable diagnostics. `selectCatalogTests(discovery, {tag, all, any, exclude, ids})` reads selection metadata, returning selected entries, available alternatives and uncertain coverage. Tags match case-insensitively; `tag` is shorthand for one all-tag. IDs are qualified references. No matches executes nothing and never broadens the request. `resolveCatalogTest(discovery, reference)` validates one reached test and needed ancestors. `testContext(test)` assembles its worker context. `parseCatalog` remains an explicit full-resolution authoring check; the runner uses discovery/selection and resolves at execution time. Unknown metadata is retained without acquiring executable authority. The CLI `inspect --catalog DIR [--catalog-id NAME] [--tag TAG] [--all A,B] [--any A,B] [--exclude A,B] [--ids REFS]` reports selection metadata and gaps, not full semantic validation or a successful run. `start` accepts the same selection flags. For authoring validation call `parseCatalog` or explicitly resolve the proposed tests through the public API. Use existing onboarding discovery/readiness targets to choose applications. With several applications, the coordinator asks the human even if a default exists. `resolveCatalogScope(discovery, configuredCatalogs, request)` in `tooling/testing/catalog-scope.mts` consumes those targets; it does not discover a competing set of applications. Configuration entries are `{application, id, path}`. After one application is chosen, include every configured catalog unless narrowed by explicit catalog IDs. The coordinator runs each returned catalog with its stable `--catalog-id` and a fresh output directory; the final summary covers all returned catalogs. Clarify ambiguous tag combinations at this coordinator/skill layer. Representative nested API/CLI/prose/duplicate and malformed-metadata definitions are in [the catalog-contract examples](../../examples/test-plans/catalog-contract/README.md). Invalid examples demonstrate diagnostics, not a recommended execution target. Existing evidence metadata is retained; shared bug-report-button configuration/plugins remain authoritative, with evidence integration owned by #160. This catalog change adds no separate AI capture policy.