Markdown test execution#

This first workflow separates catalog definitions, retained test attempts, and bug reports. Skills coordinate the user's agent/browser tools; portable modules handle discovery semantics, hooks, validation, artifacts, and rendering. There is no mandatory T3 runtime.

Start a run#

Use the repository's pinned Node/pnpm, install its lockfile, and build workspace modules. The runner is tooling/testing/plan.mts; help lists commands. An adopting project keeps its own catalog and configuration; the runner can remain in a separate SnagBadger checkout.

node <runner-root>/tooling/testing/plan.mts inspect --catalog <project>/tests
node <runner-root>/tooling/testing/plan.mts start --catalog <project>/tests --output <new-run-directory> --runtime <project>/testing-runtime.json --tag smoke

Keep start running in a persistent session while the coordinating skill services bridge requests. The output directory must be new. A plan is a directory, not a separate manifest. Without a tag/ID selection, all tests run. Tags inherited from sections participate in selection.

Test definition#

The sample-site catalog contains standalone tests and nested section files you can copy, inspect, and adapt.

Discover .snag.md tests only. Supporting Markdown is ignored; optional section.md supplies shared settings/guidance, and directories form arbitrary-depth implicit sections. Bodies are free-form Markdown. Creation guidance recommends prerequisites, checkbox steps and expected outcomes; headings and steps are optional. Workers receive the original body, and only authored checkbox items become checklist steps.

An explicit ID is optional and falls back to the filename without .snag.md. IDs are section-local. References use catalog-id:section/local-id (root: catalog-id:local-id). In filename order duplicates receive -2, -3, etc., skipping all occupied authored IDs; files are never rewritten to save suffixes. Identical local IDs in another section/catalog are distinct. Renames/new duplicates can change calculated references. Artifacts retain source path, body and resolved configuration; historical reports retain their recorded definitions.

Existing catalogs must rename test files from .md to .snag.md deliberately; a regular .md file is now supporting documentation. Calculated references and execution revisions can change, so start a new run rather than forcing historical retry compatibility. This discovery change does not establish a general saved-artifact migration policy.

---
id: account-sign-in
title: Sign in with a test account
executor: browser
tags: [smoke]
components: [authentication]
levels: [integration]
---

## Prerequisites

A disposable staging account exists.

## Steps

- [ ] Open the sign-in page.
- [ ] Submit valid credentials.
- [ ] Inspect the signed-in account.

## Expected outcomes

The account page displays the expected test user.

Executors are browser, api, and cli. A section can supply the executor default. Explicit test values override it. Tags/components/levels are string lists merged as sets. Levels are project metadata, not directory depth. Unknown frontmatter is preserved without executable authority. Recognized fields validate only when execution reaches a selected test, together with the inherited settings it needs. Invalid reached definitions are recorded with file/reason and skipped; independent tests continue. Unreadable selection metadata is reported as unselectable coverage, rather than a pass. Overridden scalar settings need not validate; unrelated invalid branches are untouched. The execution-v1- revision fingerprints the resolved execution contract: body, authored checklist, extracted optional prerequisites/expectations, inherited instructions, executor, applicable evidence requirements, and declared run/section/test hooks. Titles, tags, ordering, and destinations do not affect compatibility; unrelated sections do not affect each other. This compact fingerprint is not a cryptographic integrity claim. Explicit retries require matching execution revisions; project hook implementations remain the project�s responsibility.

Directories create sections even without section.md. Traversal follows filenames deterministically; numeric order has no precedence. Prefix filenames when start order matters. The current runner remains sequential; concurrent scheduling and lifecycle redesign belong to #154.

Resolved lifecycle scope#

resolveTestLifecycle({ sections }, test) from @snagbadger/agent-catalog exposes root-to-leaf section ancestry and effective hook declarations with phase and section identity. It uses the already-resolved per-attempt inheritance in the test; run hooks belong only to the root. Missing ancestors, cycles, duplicate section identities, multiple roots and misplaced run hooks are rejected. The declaration list preserves nearest-section reference precedence; it is not an invocation schedule. Execution owns entry/exit transitions, reversed cleanup and failure policy. Catalog execution revisions, execution transitions and onboarding support review use this same scope, retaining existing execution revision identities.

Optional section configuration#

---
title: Accounts
tags: [smoke]
destination: { provider: jira, project: ACC }
instructions: [Use disposable staging accounts.]
evidence:
  browser: { screenshot: required, video: required }
hooks:
  beforeEach:
    - { id: authenticate, function: authenticateUser }
  afterEach:
    - { id: remove-fixture, function: removeFixture }
---

Use the staging environment. Keep test accounts independent.

The section body is inherited guidance. Instructions append root-to-leaf. Use instructions: {replace: true, items: [New guidance]} to replace inherited instructions. Tags match case-insensitively, accumulate, and support explicit removal with tags: {remove: [smoke], items: [regression]}. components and levels remain descriptive lists. Executor/retries use the nearest explicit value; evidence overrides individual settings. Root-to-parent guidance precedes one clearly emphasized “Test to execute” block; sibling bodies are excluded. Destination settings inherit as a provider/project pair and an explicit child destination replaces the pair. No credentials belong here.

Evidence settings merge per executor and kind. Supported kinds are screenshot, video, diagnostic, recording, and file; requirements are required, optional, or off. Defaults require screenshot/video for browser tests and diagnostic output for API/CLI tests. Missing required proof makes the attempt incomplete; an observed behavioral failure remains recorded and can still create a bug. Optional missing evidence produces a warning. Hook diagnostics do not count as behavioral proof.

Lifecycle hooks#

Run hooks live only in root section.md. Section hooks run once when that selected section is entered/exited. Test hooks inherit through the hierarchy and may be added in test frontmatter.

• beforeRun / afterRun: shared environment setup and teardown.
• beforeSection / afterSection: shared resources for that section.
• beforeEach / afterEach: isolated test setup and cleanup.

Setup proceeds root-to-leaf; cleanup reverses hook/section order and is attempted after setup/test errors. Cleanup functions must handle partially initialized resources and a null browser when context creation failed. Browser context creation precedes per-test hooks; video recording starts before them when supported. A once-per-section login does not authenticate each isolated context.

Each hook declares a unique id and exactly one function or instruction. onFailure defaults to block; stop ends the run after cleanup. Inherited per-test hooks compose. Explicit overrides use:

hooks:
  beforeEach:
    disable: [authenticate]
    items:
      - { id: authenticate, instruction: Sign in through the UI. }
  afterEach:
    replace: true
    items:
      - { id: cleanup, function: customCleanup }

A function name is a registry reference, never inline executable YAML. The selected project module is trusted developer code. It receives runId, test, section, session, shared (a local Map for fixture state), and browser.perform(action) when a browser context exists. Shared values are not serialized into reports.

export function configureTesting({ redactReportSecrets }) {
  return {
    functions: {
      async authenticateUser(context) {
        if (!context.browser) throw new Error('Browser context unavailable');
        await context.browser.perform({
          type: 'navigate',
          url: process.env.TEST_LOGIN_URL,
        });
        // Further actions use this same context. Keep secrets in the environment.
      },
    },
    reportPolicy: {
      id: 'project-evidence-review',
      reviewReport(report) {
        return redactReportSecrets(
          report,
          [process.env.TEST_SECRET].filter(Boolean),
        );
      },
      reviewAttachment(evidence, bytes) {
        // Replace with the project's actual image/video review policy.
        return { action: 'omit' };
      },
    },
  };
}

The module may instead export functions and reportPolicy directly. The configureTesting factory receives existing library helpers so an adopting module need not duplicate serialization/redaction or resolve workspace packages.

Runtime configuration#

{
  "capabilities": {
    "name": "user-browser-mcp",
    "freshAgents": true,
    "newBrowserContexts": true,
    "contextProof": "Installed tool exposes isolatedContext; verified with distinct named contexts.",
    "screenshots": true,
    "video": false
  },
  "module": "./testing-hooks.mjs"
}

Capabilities are attestations from actual installed tool contracts/probes, not switches that create missing capabilities. Browser tests require a unique new context for every attempt; the engine rejects missing/reused context identities. The coordinator routes project-function actions to that exact handle. Read runtime mappings and limitations.

Host bridge#

Use next --output DIR to read the pending UUID request at DIR/bridge/request.json. Give the fresh worker that actual packet locator or a saved packet. Complete it with complete --output DIR --response FILE. Requests are local files, not an unauthenticated HTTP endpoint.

{ "id": "<current-request-uuid>", "ok": true, "value": null }

An unsuccessful operation uses ok false and a nonempty error string. Transport failures differ from a valid failed test. Required response values:

The packet supplies runId, attemptId, one resolved test, and its session. Delegate executeTest with no inherited conversation; pass only the packet plus runner/output/request information and the execute-test skill.

An Attempt uses id, startedAt, endedAt, outcome (passed/failed/blocked/incomplete), actual, hypotheses, steps, interactions, evidence, and reproduction. See the exported agent-reporting types and portable reports. The runner owns final identity/timing. Empty collections and null optional records are explicit.

{
  "id": "<current-request-uuid>",
  "ok": true,
  "value": {
    "attempt": {
      "id": "<packet-attempt-id>",
      "startedAt": "2026-10-11T05:00:00.000Z",
      "endedAt": "2026-10-11T05:00:01.000Z",
      "outcome": "passed",
      "actual": "The endpoint returned status 200.",
      "hypotheses": [],
      "steps": [],
      "interactions": [],
      "evidence": [
        {
          "id": "response",
          "label": "Endpoint response",
          "kind": "diagnostic",
          "capturedAt": "2026-10-11T05:00:01.000Z",
          "content": { "type": "structured", "value": { "status": 200 } }
        }
      ],
      "reproduction": null
    },
    "attachments": []
  }
}

The worker uses progress with the same response shape after meaningful steps. Files are copied into durable raw-evidence before they count as attachment proof; missing bytes become unavailable evidence. Completed progress is retained if execution fails. Raw artifacts are local private execution state, not reviewed exports.

Completion, retries, and reporting#

checkpoint.json is the authoritative runner snapshot: formatVersion: 1, a validated view containing the shared run/result/attempt model, hierarchy, hook events, warnings and processing status, and an attachments index keyed by result/attempt/evidence identity. Attachment paths are relative to the output directory under raw-evidence/. Bytes are stored before references are published. One atomic replacement commits the complete snapshot; an interrupted replacement leaves the previous complete checkpoint readable. Request/progress files identify unfinished work. Finished results persist before the next test starts.

Cancellation records a marker; the coordinator must stop/finish the active worker and service cleanup requests. An interrupted coordinator can reconnect to the pending request in the still-running process, or render saved partial output. Process-restart resumption of in-memory shared fixtures/hooks is not promised.

Use start with --retry --ids and the original output directory for an explicit retry; earlier attempts stay intact. Run/section fixtures are prepared again. The production viewer shows eventual recovery as Flaky, separately from Passed. Programmatic default any-failure aggregation still keeps earlier failures visible. Read automatic retry configuration and accounting for the run/section/test limits, pass ordering and terminal failures. Outputs from the earlier run.json / execution.json / attachments.json format remain readable by report when run and view agree. They are never rewritten or silently migrated. Retrying legacy output or legacy leaf-only definition revisions requires a new run and output directory. Unsupported or malformed checkpoints fail explicitly.

Read the production viewer and token contract for portable resources, scale behavior, runtime counters and single-user bug export. The HTML report has a fixed layout, nested sections, title/tag/component/level search, outcome filters, planned expectations, recorded interactions, attempts, hook diagnostics, and image/video/diagnostic evidence. Report customization is excluded.

The configured reportPolicy reviews the entire text/source snapshot and attachment bytes through shared report-export preparation, without generating an unused Bug-report viewer. Binary redaction, omission, unavailable proof, and narrative sanitization clear affected verification claims. Missing review leaves structured results intact and reports needs-review; use report --output DIR --runtime FILE once review is configured. Each reviewed export has a new directory, preventing stale/omitted attachments from leaking from prior exports. No raw fallback is allowed after review errors.

Create bug selects a failed attempt (including an observed failure with incomplete proof), opens an editable local draft, and preserves destination metadata. Its report-owned single-user form edits content and evidence, exports a portable ZIP or saves an editable JSON manifest, and retains edits on export failure. It does not share a composer with the launcher or submit automatically. The create-test-bug skill packages reviewed evidence through the shared bug-report/provider interface. Live tracker authentication/transport remains provider work.

Skills and verification#

Invoke the repository skill paths directly or install the folders into the host's skill search path:

For guided onboarding and portable installed skill names, start with guided setup. The new setup/initialize-tests/create-test/test workflows reuse the contracts and coordinator below.

• setup/initialize-testing
• execution/run-test-plan
• execution/execute-test
• reporting/create-test-report
• reporting/create-test-bug

The independent skill test uses fresh agents on a disposable catalog; browser/adapter conformance tests use isolated local fixtures. Runtime-specific MCP compatibility requires installed tools; simulated tests do not prove a live MCP implementation. Run pnpm verify for the full production gate.

Lazy catalog interfaces and selection#

discoverCatalog(sources, {catalogId}) reads syntax/identity and returns entries and unselectable diagnostics. selectCatalogTests(discovery, {tag, all, any, exclude, ids}) reads selection metadata, returning selected entries, available alternatives and uncertain coverage. Tags match case-insensitively; tag is shorthand for one all-tag. IDs are qualified references. No matches executes nothing and never broadens the request. resolveCatalogTest(discovery, reference) validates one reached test and needed ancestors. testContext(test) assembles its worker context. parseCatalog remains an explicit full-resolution authoring check; the runner uses discovery/selection and resolves at execution time. Unknown metadata is retained without acquiring executable authority.

The CLI inspect --catalog DIR [--catalog-id NAME] [--tag TAG] [--all A,B] [--any A,B] [--exclude A,B] [--ids REFS] reports selection metadata and gaps, not full semantic validation or a successful run. start accepts the same selection flags. For authoring validation call parseCatalog or explicitly resolve the proposed tests through the public API.

Use existing onboarding discovery/readiness targets to choose applications. With several applications, the coordinator asks the human even if a default exists. resolveCatalogScope(discovery, configuredCatalogs, request) in tooling/testing/catalog-scope.mts consumes those targets; it does not discover a competing set of applications. Configuration entries are {application, id, path}. After one application is chosen, include every configured catalog unless narrowed by explicit catalog IDs. The coordinator runs each returned catalog with its stable --catalog-id and a fresh output directory; the final summary covers all returned catalogs. Clarify ambiguous tag combinations at this coordinator/skill layer.

Representative nested API/CLI/prose/duplicate and malformed-metadata definitions are in the catalog-contract examples. Invalid examples demonstrate diagnostics, not a recommended execution target. Existing evidence metadata is retained; shared bug-report-button configuration/plugins remain authoritative, with evidence integration owned by #160. This catalog change adds no separate AI capture policy.