Runs, results and attempts#

A test run executes selected definitions. Each test result pairs a definition snapshot with its retained attempts and comments. An attempt is one execution, with its own actions, outcome, timestamps and evidence.

Preserve history#

Retrying appends another attempt instead of replacing the earlier failure. A report can still select that earlier attempt after a later pass. The policy chosen by the project determines the summarized result; default policy helpers require an explicit choice of latest, any-failure, or passed-on-retry.

Do not infer a clean history from the latest green result. Read attempts and the selected aggregation policy together.

Comments are independent#

A result comment has a type and importance independent of outcome. Default types are problem, optimization, suggestion and context; default importance is low, medium, high and critical. Projects can configure both scales. An explicit setting allows new comment types, while importance stays within the configured scale.

A comment can describe a problem discovered during a passing test or context for a result that has not run. Its basis distinguishes observations, hypotheses and context. Promotion preserves that distinction; an incidental problem does not inherit the intended test's expectation or verification.

Proof and completion#

Missing required evidence makes an attempt incomplete, even when it observed a real failure. That failure can still support a bug report. Optional missing evidence produces a warning. Hook diagnostics describe setup/cleanup; they do not replace behavioral proof.

The runner persists checkpoints so finished and partial observations remain available if execution fails. Raw checkpoint/attachment output is private execution state until reviewed export.

Use review and promotion, selection/retries, and format reference to work with this history. Illustrative results demonstrate the distinction without claiming a live run.