Design and Rationale

Status: Draft · Companion to the Workfile specification · Non-normative

Workfile makes composed work describable without requiring code for an ordinary workflow. The specification defines the format and its manifests. This document explains the design boundaries without adding requirements.

---

Why a format at all

Most work is composed of small, discrete steps. Consider an email that must be classified, recorded as a lead, opened as a ticket, or ignored as spam. Teams have several options to turn this into a workflow:

A script. Code can express all of it. The author then owns everything that is not the work itself: acceptance of the email,the formal classification process, integration with a ticketing and CRM system, retries, backoff, error classification, credentials, a place to run, and a way to resume after a failure. The description of the work ends up in source code, mixed with the machinery that runs it.

A builder tool. A no-code drag-and-drop tool removes that machinery, and holds the workflow as a graph in one vendor’s database. An export is a serialization of the editor’s state. It is not a document that a person writes, reviews, or generates.

An agent’s own tool calls. A model can call each action in turn. Every intermediate result then passes through the model’s context, which costs tokens and time. The model decides the sequence again on every run, so two runs of one request can differ. A transcript is all that remains afterwards.

The same artifact is missing from all three. In the script, the description of the work exists, but it is mixed with the code that executes it. In the builder, it belongs to one vendor. With the agent, it is never written down at all.

A format supplies that artifact. A Workfile states the work and nothing else, so:

• a person can read it, and review it before it runs;
• a model can write it, because the grammar is small and one construct has one form;
• any engine that implements this standard can run it;
• it lives in version control, and a diff shows what changed.

The two authors of a composed workflow

The format divides knowledge between two authors.

The workflow author states what should happen as a series of composed units of work. This lives in the Workfile.

The catalog author declares how a given API behaves: its errors, its retry policy, its idempotency, its correlation keys. The catalog author has a responsibility to track upstream vendor changes and maintain the catalog.

Every design decision in the format follows from keeping these separate.

Keeping the language small

Workfile is a small language, and it will stay small. The expression layer is pure, total, and not Turing-complete. It has no user-defined functions, no macros, no recursion, no unbounded iteration, and no mutation. Logic that needs more power belongs in wf.run, in a wf.agent, or behind a connector.

Because of this design:

• a file is checkable before any side effect occurs;
• a run can replay from its record without contacting a dependency;
• a run can suspend and resume correctly;
• a model can generate a file mechanically, and a validator can tell it what is wrong.

Why store exists, and why it is small

A workflow often needs one small piece of state that outlives a run: a cursor, a deduplication flag, a round-robin counter.

store therefore holds get, put, compare-and-set, and increment on single keys, and nothing more. It defines no enumeration and no query, because those would make it a database and invite schema, indexing, and migration into the format.

Why sync exists, and why store still does not query

A workflow that moves records between two systems needs one more thing than a cursor. It needs to know, for each record, whether it has handled that record before, whether the record has changed, and which record in the destination is its counterpart.

That question has the same four fields in every integration: a fingerprint, a local identifier, and the two instants that bound when the record was seen.

sync therefore holds it, and its surface is six operations over those four fields.

sync answers a closed set of questions. It has no query by fingerprint, scope listing, or operation across scopes. unseen answers which records a pass did not see.

A record map that needs more is the connector’s map. A second identity that survives the source renumbering its ids, an enumeration of a scope to diff against a remote index, or a mapping row written in the same transaction as the destination row: each of these is a property of one system’s database, and sync stops where a database begins. Such a connector holds the map beside the rows it maps, under its own migrations and backups, and sync is not in the picture. This is the boundary that keeps sync a memory rather than a database.

Change detection is what makes a re-run cheap. A workflow that can tell an unchanged record from a changed one does the work once. Without it, every run repeats every write, and a format whose promise is that the machinery is handled would be handing back the most expensive part.

The position comes from what persisted. A poll trigger owns a cursor, and that cursor advances on what the trigger read. A workflow that writes as it walks needs the other one: the position that the destination reached. sync.watermark is that position, because sync.record is what writes it. A run cut short leaves the watermark on the last record it wrote, and the next run continues from there.

lib-sync is a capability, so an implementation that will not hold entries declines it and a file that needs it is rejected at validation rather than at run time.

Why there are no transactions

A workflow often writes several things that must succeed or fail together: a record, the rows that link to it, and the file it references.

The unit of atomicity is the action. A set of writes that must succeed together is one action, and the connector holds whatever the target system provides to keep them together. A workflow that needs the guarantee names one action, and the connector author decides how to keep it.

undo is the compensating mechanism, and it covers a different case. Two systems share no transaction. A payment in one service and a ticket in another therefore cannot be joined by one, and the file states how to reverse each effect on its own.

Where a transaction is available, compensation is the weaker choice. A compensating action can itself fail, and a failed compensation leaves the effect in an unknown state. A transaction either commits or it does not, and the state is known after either result.

The test for a connector author is short: two writes that must never be seen apart belong in one action.

Why into exists on triggers

Two sources of one logical event are common: an RSS item and a spreadsheet row that both mean “a post to publish”. Without a projection, every step that reads the payload has to branch on which trigger fired, and the conditional is repeated in each step.

into moves that reconciliation to one place, before the first step. The requirement that every entry bind the same names with the same structure is what makes the result checkable: a path that resolves for one trigger resolves for all.

Triggers that differ in behavior rather than in payload shape are a different problem, and are best addressed via separate Workfiles.