Conformance Suite

Status: v0.6 · Spec version 0.6

This directory is the conformance suite that the specification defines. The layout and the fixture formats are normative, and the suite section specifies them. Error codes come from the error code registry.

The suite covers every validation code of that registry, every evaluation fault, and every core filter. The suite section also states the standard for completeness: a requirement that no case exercises is untested.

Layout

Directory	Holds
`validation/`	One case per file. A document, and the codes an implementation MUST report for it.
`expressions/`	Expression fixtures, as JSON Lines. One object per line.
`connectors/`	The test connector manifests that the conformance rules require. Fixtures, not a catalog.
`library/`	The standard library: canonical connector manifests with samples under `connectors/`, and canonical filter packages with their fixtures under `packages/`. Normative artifacts, distributed with the suite.
`traces/`	Execution cases: a workflow, canned action results, and the expected outcome of every step.

The schema check

check_schema.mjs is the suite’s one runnable consumer today. It is not a conformance runner: a runner is part of an implementation, and none exists yet. The script checks the structural layer alone.

cd conformance
npm install
npm run check

It asserts four things: every valid workfile and every trace workfile passes the JSON Schema; the structural error cases fail it; error codes in fixtures are registered in the error code registry; and the fixture files satisfy their format contracts. The script holds the list of structural cases, so a schema regression or an unrecorded schema improvement fails the check.

Validation cases

A case passes when the set of codes an implementation reports equals expect, exactly. Message text is not compared. An empty expect asserts the document is valid.

Every case resolves against the manifests in connectors/ and the canonical packages in library/. A case MAY add to that context with the keys that the suite section defines:

manifests — further connector or filter package manifests, inline.
overlays — schema overlays, inline.
connections — named connections, each with connector, granted_scopes, and parameters. Without this key, every connector resolves through one complete connection, named default, that grants every scope.
files — project-relative paths to file content, for call callees and templates.
capabilities — the capability set to assume. Without this key, every capability is declared.

workfile is normally the parsed document. In a case that tests the serialization subset, workfile is a string that holds the raw document text, because the condition does not survive parsing.

The structural subset of these cases is also checkable with the JSON Schema. The schema cannot decide a case whose code needs a catalog, a connection, a file, or a scope — those cases are marked needs: resolution and a schema-only checker skips them.

Expression fixtures

One JSON object per line: expr, an optional bindings map, and exactly one of result and fault. A string binding in a position that expects a timestamp or a duration converts by the timestamp rule.

File	Covers
`core.jsonl`	Arithmetic, equality, ordering, membership, indexing, and conversion faults
`operators.jsonl`	Precedence, short-circuit evaluation, the conditional, and literals
`collections.jsonl`	The collection filters
`selection.jsonl`	`where`, `keep`, `each`, and `has`
`strings.jsonl`	The string filters
`numbers.jsonl`	The number and conversion filters
`time.jsonl`	Timestamps, durations, and the time filters
`escaping.jsonl`	The escaping and hashing filters

Where the core filter annex defers to fixtures, the fixtures are normative. The escaping fixtures fix the exact character sets and quoting forms; fault.template_render has no expression fixture, because it occurs only in a template body — traces/template_render_fault.yaml covers it.

Trace cases

A trace case supplies workfile, optional inputs, canned actions, optional events, and expect.

actions entries are matched by step path and consumed in attempt order per path. A retried step consumes one entry per attempt. An entry’s outcome is success, failure, or unknown — an ambiguous attempt.
A reconcile step is addressed as <step>.reconcile.<id>, and an undo invocation as <step>.undo.
events supplies the outcome of each wait_for, in order: { step, outcome: received, event: {...} } or { step, outcome: timeout }.
expect states the run’s terminal outcome and the outcome of every step. It MAY state bindings (path to value), outputs, and codes (the classified code per failed step).
A run that ends held is stated with held_at: the step that caused the hold and its classified code. steps then holds only the steps that reached an outcome, because a held step has none yet.

A step that a trace does not reach still appears in expect, with the outcome not_run.

The recovery traces exercise the full on_unknown and undo semantics: on_unknown_halt, on_unknown_retry, reconcile_effect, reconcile_no_effect, and undo_before_redo.