Production test-run viewer#

The accepted report layout and runtime accounting are tracked in #100. @snagbadger/agent-reporting/html exports createTestRunReport(view, options). The testing runner's report command uses it directly. Layout studies remain in examples/playground/report-prototype; production assets read validated run resources and contain no illustrative test outcomes.

Portable output#

Keep index.html, viewer.js, viewer.css, run.json, resources/ and attachments/ together. Open the HTML directly or serve the directory over HTTP. Classic resource scripts allow lazy loading on file:// without a fetch/CORS dependency. The compact index holds navigation, status, usage and shortened comment previews. A selected test loads its full attempts, comments, hooks, warnings and metadata separately. Image/video files load when Evidence is opened. A bounded cache retains at most 30 resources; closed tree/comment groups unmount their content.

Tests always use a tree. Search, counted outcomes and inherited tag selection combine and prune empty sections. Multiple selected tags match any selected tag. Overall outcome/attention counts remain run-wide; Tests/section summaries reflect filtered descendants. Each branch and finding group renders 25 entries per page. Section siblings are also paged; controls appear only when necessary. Attention selections clear conflicting filters, expand every containing section and reveal its section/test pages. Breadcrumbs reveal the corresponding section page. Filtering away the selected test selects the first match or an empty state; clearing filters retains the current selection. Sorting is an icon beside Tests. Definition, Attempts, Comments and Evidence are exclusive keyboard-operated tabs.

Requires your attention groups failed, blocked, incomplete and Flaky results, plus AI comments using the run policy's ordered importance levels and actual allowed types. Comment filters combine and remove empty groups; selecting a comment opens its full content and attempt context. Findings on passing tests retain the passing verdict. The fixed warm cream/rust/brown theme uses Technical mono typography.

The 10,000-test check proves bounded rendered rows and initial detail loading, including 100 section siblings. It does not bound index size or media bytes. Offline bug export also stores reviewed attachment bytes in lazy base64 resource scripts (about one-third encoding overhead, in addition to normal media files). Large attachments remain subject to the existing 100 MB per-file/250 MB total bundle limits. No prices are inferred.

Automatic retries and Flaky results#

Execution completes the initial selected-test pass before retrying failed or incomplete tests in subsequent passes. Blocked tests are excluded. ExecutionOptions.retries defaults to one additional attempt and accepts integers 0�20; zero disables automatic retries. Catalog frontmatter retries inherits from the nearest section and overrides the run default; test frontmatter takes precedence. The filesystem runner reads the run default from testing-runtime.json's top-level retries field. Explicit start --retry --ids ... is a new execution invocation with its own additional-attempt budget and retains all previous attempts; it still requires unchanged execution-definition identity.

Every attempt delegates to a fresh agent, uses a fresh isolated browser context when applicable, executes test setup/cleanup, and retains evidence and cumulative usage. Section fixtures close between passes and prepare again; run fixtures enclose the invocation. Cancellation, fatal runtime errors and checkpoint publication failures stop execution. Nonfatal worker interruption is incomplete and eligible for retry; checkpoint save failures are terminal infrastructure failures, never Flaky tests.

The viewer classifies a test as Flaky when its latest attempt passed after an earlier failed or incomplete attempt. It has a separate count/filter and attention group. Raw attempt outcomes and explicit RunPolicy.retrySummary aggregation remain unchanged for programmatic callers; the viewer's recovery classification intentionally remains visible even with latest or any-failure policy. Legacy passed-on-retry aggregation remains supported by summarizeResult.

Navigation owns selection, filtered hierarchy and bounded pages in one module. Run checkpoints own cumulative measurement replacement, partial retention and queue-safe persistence. Reviewed run projection owns evidence correspondence, omissions, attachment paths, verification invalidation and test/section hook scope through the public report-review interface.

Runtime tokens#

Optional Attempt.usage and TestRun.coordinatorUsage extend data format 1 backwards compatibly. Missing/null means unavailable, not zero. parseTokenUsage validates nonempty lists of at most 1,000 unique provider/model buckets. Each bucket is a cumulative runtime measurement within its scope:

{
  "provider": "runtime-provider",
  "model": "actual-model",
  "source": "runtime",
  "measuredAt": "2026-10-11T05:00:00.000Z",
  "completeness": "complete",
  "input": 6000,
  "output": 900,
  "cachedInput": 2000,
  "reasoningOutput": 300
}

Counters are nonnegative safe integers; cached input is included in input and reasoning output in output. Total is input + output. Checkpoints replace the same attempt's previous snapshot; distinct retained attempts sum once. Providers/models cannot duplicate within one snapshot. Aggregation rejects safe-integer overflow.

summarizeTokenUsage reports known totals, whether any measurement is available, coverage and average per completely measured attempted test. All retained attempts of a test must have complete measurements for that test to enter the average. Unknown/partial tests and tests not run do not enter its denominator. Known partial consumption stays in totals, labelled partial. Actual measured zero remains distinct from unavailable usage. Attempt and coordinator breakdowns use their own units.

Runtime.getCoordinatorUsage(runId) optionally returns cumulative coordinator/shared counters for the entire run identity, including retries, excluding delegated attempt consumption. Hooks and other shared work belong to this scope. The runner refreshes between lifecycle operations and at completion; it does not enqueue a second host request inside an executing worker's progress callback. Until refreshed, coordinator counters in progress snapshots are partial. A missing later measurement preserves prior known counts as partial; invalid measurements fail validation.

Filesystem adapters advertise capabilities.tokenUsage: true only when runtime counters are actually accessible. The bridge then issues getCoordinatorUsage with {runId}, expecting a measurement list or null. Older adapters receive no new request. Workers return their own cumulative attempt.usage with progress/final responses. Scope ownership and continued cumulative counters across retries are adapter obligations; the library cannot independently audit provider telemetry. Do not substitute estimated tokens or infer counters from narrative text.

A single user files a bug#

The report owns its form; it never shares a composer with the launcher. It calls public createBugFromResult for one selected attempt and compatible selected comments, then public revision/evidence-removal APIs for edits. Earlier failed journeys remain available even when the latest attempt passed. Mixed-attempt comments are rejected by the report contract.

The form displays provenance, resolved destination, editable content and evidence inclusion. Export report bundle creates a ZIP with reviewed report JSON, independent HTML and included attachment bytes, using public createReportBundle. The SDK importer validates the extracted files. Save bug draft creates editable JSON with destination and attachment references, but no attachment bytes. Both exports omit the full source snapshot at the review boundary, preserving source references without copying excluded material. Original run files remain intact. Removing evidence also removes its direct step/interaction references and invalidates verification as defined by the SDK.

Export failures preserve edits and choices for retry. The portable viewer explicitly reports that live tracker submission is unavailable. A local export never claims a ticket was filed. Privileged provider transport is separate and remains #11; public report/export contracts are reused, UI is independent.

Owned local-network preview#

After generating a reviewed report, PowerShell can expose that exact directory through the supported examples server:

$env:SNAGBADGER_REPORT_DIRECTORY = 'C:\path\to\reviewed-report'
$env:SNAGBADGER_HOST = '0.0.0.0'
pnpm dev:examples

Use the printed port with /playground/run-report/; replace loopback with the machine's LAN address on another device. The development-only route serves selected report assets and includes the owned source identity for T3 verification. It does not generate test results or change portable output. Reports used for revision evidence must come from that revision's checks. Stop the serving process before further worktree mutations.