Conformance

The specification is its pages and annexes together with the conformance suite. Both are normative. Where they disagree, the specification has a defect. Implementations then follow the suite provisionally, and the maintainers resolve the disagreement explicitly. A reference engine is an implementation like any other: where it disagrees with the suite, the engine is wrong.

The suite is language-agnostic data: Workfiles, connector and filter manifests, fixture inputs, and expected results, step traces, and error codes.

The suite distributes the standard library’s canonical manifests and fixtures, under library/. Vendor connectors are not mandated, so the suite also ships its own test connectors: fully specified manifests that exercise every path an implementation must handle. They cover idempotent and non-idempotent actions, each error classification, an action that returns an ambiguous outcome, a deterministic action, a validated_output action, and one trigger of each kind.

Levels

An implementation conforms at a stated level and capability set.

• Level 1 — Validator. Parses the serialization subset, resolves against supplied manifests and overlays, and reports every condition of static validation with the correct error code and source location.
• Level 2 — Executor. Level 1, plus executing the test connectors with correct ordering, skip, failure, recovery, and undo semantics, and correct evaluation of the full core filter set.
• Level 3 — Durable. Level 2, plus checkpointing, resumption, and replay, as recording and replay requires, across the capabilities that it declares.
• Portable profile. Level 2 or higher, with every standard-library capability declared. A Workfile whose references stay within the library runs on every implementation of this profile.

An implementation MUST state its level and its capability set together. A capability that an implementation does not declare is not a conformance failure. Executing a construct that requires an undeclared capability is a conformance failure.

The suite

The suite is a directory of data files. Its layout is normative, so that one suite runs against every implementation.

conformance/
  validation/     one case per file: a document and the codes it must produce
  expressions/    expression fixtures, as JSON Lines
  connectors/     the test connector manifests
  library/        the standard library: canonical manifests, fixtures, and samples
  traces/         execution cases: a workflow, canned action results, expected outcomes

A validation case is a YAML document with the keys case, expect, and workfile. expect is a list of the error codes that an implementation MUST report. An empty expect asserts that the document is valid.

workfile holds the document as parsed YAML. A case that tests the serialization subset instead holds workfile as a string, whose text is the raw document. The condition of such a case does not survive parsing.

A case MAY supply the following keys beside the document.

• manifests — connector and filter package manifests, resolved in addition to the suite’s test connectors and the library’s canonical packages.
• overlays — schema overlays to resolve.
• connections — named connections, each with connector, granted_scopes, and parameters. Without this key, every connector resolves through one complete connection, named default, that grants every scope.
• files — a map of project-relative path to file content, for call callees and for templates.
• capabilities — the capability set to validate under. Without this key, every capability is declared.

A case whose codes require a catalog, a connection, a file, or a scope is marked needs: resolution. A checker that applies only the published JSON Schema skips such a case.

case: route over a validated enum leaves a member uncovered
expect: [flow.route_not_exhaustive]
workfile:
  steps: [ ... ]

An implementation passes a case when the set of codes that it reports equals expect. Message text is not compared.

An expression fixture is one JSON object per line, holding expr, an optional bindings map, and exactly one of result and fault.

{"expr": "{{ 7 / 2 }}", "result": 3.5}
{"expr": "{{ 1 / 0 }}", "fault": "fault.division_by_zero"}

A trace case supplies a workflow, a result for each action invocation, and the expected outcome of every step together with the run’s terminal outcome. It is how Level 2 and Level 3 are tested.

Canned results appear in actions. Entries are matched by step path, and the entries of one path are consumed in attempt order, so a retried step consumes one entry per attempt. A canned outcome is success, failure, or unknown — an ambiguous attempt. A step inside a for_each iteration, a repeat iteration, or a state entry is addressed by position: poll[0].probe, fetch[1].page. A reconcile step is addressed as <step>.reconcile.<id>, and an undo as <step>.undo. A case MAY supply events: the outcome of each wait_for, in order — a delivered payload, or a timeout. Its expect MAY also state bindings, outputs, and codes, the classified code per failed step. A trace of a run that ends held states held_at: the step and its classified code. Its steps then holds only the steps that reached an outcome.

A requirement of this specification that no case exercises is untested, and the suite is incomplete for it.

Advancing this standard

A construct advances from draft to stable when two implementations that share no code pass the suite for it. Independent interoperation is the evidence that the prose is sufficient. A construct that one implementation supports is a proposal, whatever its merits.

Use of the name

An implementation can use the Workfile name only if it passes the suite at a declared level. This is a naming policy rather than a technical requirement.