Connector Guide

Status: Draft · Companion to the Workfile specification · Non-normative

This guide is not part of the specification. Nothing here adds to or overrides the spec; where the two disagree, the spec wins. What this document answers is the question the spec deliberately does not: how do I connect Workfile to a system it has no connector for — including my own application?

The short answer, which the rest of this page expands: you write a manifest and a function. The manifest is the same format a published connector uses. The function runs in your own process, against your own database, with your own configuration. Nothing about that path is second-class, and nothing about it requires publishing anything.

---

Where a connector comes from

A catalog publishes connectors for systems many deployments share: a CRM, a chat product, a payments API. Those manifests are promises made to every tenant at once, so they are held to publication discipline — no tenant-configured field typed as an ordinary field, error tables that are vendor-truthful, idempotency declared only where the API honors it.

A private connector is a manifest you write for one deployment. It is resolved from your own project, it is never published, and the publication rules that exist to protect other tenants do not apply to it. Your manifest has exactly one tenant: you. So it declares your fields, with your types, and needs no extension point and no overlay.

This is worth stating plainly because the spec’s rules about tenant-defined schemas read, at first, like a prohibition on typing your own data. They are not. They bind a manifest that a catalog publishes for many tenants.

The practical consequence: the system a Workfile cannot reach is a missing connector, not a missing feature. wf.run cannot substitute: this revision registers no language for it, and when one is registered it is sandboxed pure compute with no actions, no state, and no egress by default (where-logic-lives). If the catalog lacks what you need, and you are not waiting for someone else to publish it, write it. For an application you already own, that is usually an afternoon.

Which kind of connector to write

The reference engine executes an action through one of four sources (ENGINE §2.3). Two of them matter when you are starting.

Source	Write this when	Ships as
Action server	The action touches your own application: your database, your ORM, your domain logic.	A manifest plus a function in Python, TypeScript, or C#, served by the SDK.
Descriptor	The action is an authenticated HTTP request against a documented API.	A manifest plus a descriptor.yaml, as data.
MCP server	The system already has an MCP server.	A manifest that maps each action to a tool.
Library	The connector is one of the standard library’s canonical namespaces.	Engine code.

Everything below is the action-server path, because that is the one with no documentation until now. A descriptor connector is data, and catalog/hubspot/ in this repository is a complete worked example of one.

An action-server connector, end to end

The manifest is an ordinary connector manifest. Declare the arguments, the result, the error codes your function raises, and the operational promises the engine acts on:

manifest: 1
connector: directory
version: 0.1.0

auth:
  type: none

actions:
  sync:
    input:
      feed: { type: "enum[members]", required: true }
    output:
      created:   { type: "int" }
      updated:   { type: "int" }
      unchanged: { type: "int" }
    validated_output: true
    idempotent: true
    errors:
      upstream_unavailable: { class: retryable }
      invalid_record:       { class: fatal }
      exception:            { class: fatal }
    retry: { attempts: 3, backoff: exponential, initial: 1s, max: 30s }

Then a function, bound to the manifest:

from workfile import Connector, ActionFailure, ActionUnknown

directory = Connector.from_manifest("manifests/directory.yaml")

@directory.action("sync")
def sync(arguments, ctx):
    ...
    return {"created": 12, "updated": 3, "unchanged": 2_940}

The connector reaches the engine one of two ways, over the same protocol. When your application hosts the engine, Engine().register(directory) hands it over at startup. When wf run or wf serve is in charge, the project’s workfile.yaml names a command such as python -m workfile serve, and the SDK serves every connector it finds (ENGINE §2.9). The SDK surface shown here is illustrative until the SDK milestone lands (ENGINE §3, M3); the contract it carries is fixed.

Either way the engine validates the manifest exactly as a catalog manifest is validated, and refuses a connector whose handlers do not cover every declared action. A name that collides with a connector already in the resolved catalog is an error, never a silent override.

The callback contract is three outcomes and nothing else:

Your function	The engine records
returns a value	success — validated against output where the manifest declares validated_output
raises ActionFailure(code)	a failure, classified by the manifest’s errors table
raises ActionUnknown()	an ambiguous outcome; the step’s on_unknown policy applies
raises anything else	a failure with the code exception

That last row is deliberate: a defect in your callback is a recognized failure event, not a crash. Declare exception in the error table and decide what it means.

ctx carries the invocation context the spec defines: run, step, attempt, generation, connector, action, connection, the idempotency_key the file supplied, timeout_ms, and dry_run. Credentials are not there and never will be: they belong to your own configuration on this path.

Your connector already has your application’s configuration. The example manifest declares no connection parameters, because a Django connector reaches the database Django already opened. Do not route a DSN through Workfile. Connection parameters exist for values that differ per account of an external vendor — a subdomain, a region — not for your own wiring.

Sync and async callbacks both work. A synchronous function runs on a thread, so ORM calls that block are fine.

Connector-owned state

A connector may hold durable state, and a sync connector must: an identity map from external record to local row, a content hash, a cursor.

That state is yours. It lives in your application’s storage, in your migrations, under your backups. The engine does not manage it, does not version it, and offers no API for it — because on this path you already have a database, and a second, weaker store beside it would be a downgrade.

The canonical store namespace is deliberately not that store. It holds single keys with no query and no enumeration, for stated reasons. Identity needs lookup by external id, lookup by local id, and a natural-key fallback. That is a table, and it belongs in your schema.

What connector-owned state owes, in return for the engine leaving it alone:

• Survive a restart. It is durable storage, not process memory.
• Be inspectable and exportable by an operator. It is the thing they will need to look at when a sync misbehaves.
• Be keyed by connection. ctx.connection is supplied precisely so that one connector can serve several tenants without their state colliding.
• Recover when it is absent. See below; this is the one that bites.

Natural-key recovery

An identity map can be legitimately empty when the content is not. The common cause is refreshing a non-production environment from a sanitized copy of production: content tables arrive with fresh primary keys, and the map is deliberately emptied, because a preserved map would point at rows that no longer exist.

So the write path is three steps, in order:

1. look up the external id in the identity map;
2. failing that, look the record up by its natural key — the field that identifies it independently of either system, usually an email or a slug;
3. only then create.

Without step 2, the next run after a refresh duplicates every record, or collides on a unique index. The worked example asserts exactly this, and removing step 2 from it produces UNIQUE constraint failed: members.email — which is the failure a real deployment reports.

The granularity rule

A step is a feed, not a record.

This is the single most consequential decision in a sync connector, and it is easy to get backwards, because imperative pseudocode describes one record at a time. The Workfile shape that follows the pseudocode literally — a wf.for_each over three thousand records, one action per record — is the wrong one.

# The shape to write: one step, one feed.
steps:
  - members:
      directory.sync: { feed: members }

Two facts about any run record make the case.

Record size. Every invocation is an entry that names its step path, attempt, connector, action, and connection. A feed-granular run has one such entry and one result. The same work as a three-thousand-iteration wf.for_each has three thousand, and those fields are what an audit needs, so retention cannot remove them. Retention is not the lever for that shape; granularity is.

Round trips. Each invocation crosses the engine’s host boundary once out and once back. Three thousand crossings are three thousand times the cost of one, however small that one is, before any vendor is contacted.

The general principle is the one the catalog already applies to pagination: transport and per-record mechanics belong inside the connector, and a well-made action returns the complete result for its declared filters. On the descriptor path, the descriptor declares the walk and the executor follows the pages (ENGINE §2.3). Your transform, your hash, your merge policy, and your per-record transaction all sit behind one action, and the Workfile records that the feed ran and what it did.

The test for co-located writes is short, and the design rationale states it: two writes that must never be seen apart belong in one action. A mapping row and the record it maps are the usual pair. A workflow that names two actions for them has no way to say they must not be split, and a later author who splits them for readability gets divergence and no diagnostic.

The same rule decides where a resolver runs. A resolver resolves a set: it takes the keys of a batch and returns a map the document indexes. Under an action source the key set is not known before the walk, so a resolver placed inside the loop runs once per element. Put it before the loop, over an enumerable set, or inside the connector’s own walk.

A null argument reaches you as null; the engine never omits a key the file wrote. Treat null as an omitted argument unless the vendor gives null a meaning of its own, and say which in the field’s description, because a Workfile has no other way to say “absent”.

Cursors you own

The poll trigger owns its cursor, and a Workfile can neither read nor set it. That rule binds the file, so that a workflow holds no transport detail and stays portable. It does not bind you.

A feed-granular connector declines the poll trigger and holds its own cursor, which is the right choice whenever the cursor must be derived from what actually persisted. A run cut short by an upstream failure has still written whatever pages succeeded, and the next run must resume from there rather than from where the failed run began. Only the connector knows that point.

The example stores its cursor keyed by feed and connection, advances it only past records that persisted, and moves it back by a second before storing — because two records can share a modification instant and an inclusive filter would otherwise skip one.

Where the connector goes

A private connector is resolved from your project. The project’s workfile.yaml lists where connectors come from (ENGINE §2.9):

catalog:
  - dir: ./catalog                         # descriptor connectors, as data
  - command: [python, -m, workfile, serve] # the connectors your code serves

A descriptor connector is a directory under that dir:

catalog/
  directory/
    manifest.yaml
    descriptor.yaml
    fixtures/

wf --catalog <dir> adds a directory for one invocation. Nothing is published, and the registry plays no part.

Whichever path, the resolution is pinned per run: a run records the manifest text it executed against, so editing a manifest never changes a run already in progress.

Classifying what the vendor actually sends

An error table keys on codes, and the useful ones are not always in the transport. The most common real-world failure of a self-hosted CMS is an HTTP 200 carrying a stack trace instead of JSON — nominally a success, with an unusable body.

A connector may synthesize a code that no transport header carried. On the action-server path, raise ActionFailure("upstream_fault") when you see it. On the descriptor path, the descriptor’s error recognizers run over the body before the status, and over the raw response text as well as the parsed body, so a recognizer can name a response that is not JSON at all.

Classify it, declare the class, and the engine does the rest. Misclassifying this as success is how a cursor advances past records that were never processed.

Give a failure its detail. A code names the class of what happened; details carries what the vendor said about it, and a catch handler reads it as error.details. A conflict that names the existing record’s id, or the vendor’s error body that a diagnosis needs, belongs there — otherwise the handler re-queries for what the error already told you. On the action-server path, raise ActionFailure("duplicate_term_slug", details={"existing_id": 42}). On the descriptor path, the descriptor names the part of the response that becomes details, and the whole body is the usual choice.

Credentials that already live somewhere

The CLI keeps the connections that wf connect writes in the user’s config directory. That is the right default for a laptop and the wrong one for a deployment whose credentials are already held elsewhere — in an encrypted table, in a secret manager, behind an OAuth flow that the deployment’s own admin owns and that other parts of the application read.

Supply them instead:

def current_connections():
    return [
        {
            "name": "acme-wordpress",
            "connector": "wordpress",
            "parameters": {"subdomain": "acme"},
            "credentials": {"kind": "oauth2", "client_id": ..., "access_token": ...},
        }
        for row in ExternalCredential.objects.filter(active=True)
    ]

engine = Engine(store="postgres://...", connections=current_connections)

A list is a fixed set. A callable is re-read before every run, in your process, so a rotated secret or a token your own flow just refreshed is the one the run uses. The engine holds only the material you last handed over, in memory, for the life of the process (ENGINE §2.7).

With connections supplied, the engine neither reads nor writes the CLI’s connection file. Refreshing an expiring grant becomes yours, which it already was: your flow issued it.

Personal data

Mark it in the manifest. A field declaration takes sensitive, and a value so declared does not outlive the run that used it — it is removed from the record when the run concludes, whatever the deployment’s retention policy says about everything else.

input:
  email: { type: "string", required: true, sensitive: true }

The value still reaches your function and still binds in run state. Only the record omits it. This is the connector author’s declaration, made once: every workflow that calls the action inherits it, and no workflow author writes anything.

What to check before you call it done

The catalog’s own definition of a finished connector applies to a private one too, minus publication:

• every error your function raises is in the errors table, with a class;
• idempotent is declared only if re-issuing really does produce one effect;
• validated_output is declared only if the result really does satisfy the schema;
• personal data is marked sensitive;
• the action is feed-shaped, not record-shaped;
• a configuration invariant — a feed declares its recovery keys, a merge policy declares its fields and its function together — is checked by your own tests, because a manifest declares types, errors, and promises, and a workflow can only check outcomes;
• there is a test.

A manifest boolean is a promise with an operational consequence: idempotent decides whether an ambiguous outcome is re-issued or halts the run for a human. An unkept promise is the worst kind of connector bug, because files type-check against it.

The worked example

guide/examples/member_sync/ is a complete version of everything above: a manifest, a connector over sqlite3, and a Workfile. The Python SDK’s smoke plan runs it (ENGINE §3, M3), including the emptied-identity-map scenario and the cursor’s behavior across runs.

sqlite3 stands in for the application’s ORM so the example needs no dependencies. Against Django the mapping is mechanical:

In the example	In a Django application
CREATE TABLE members	a Member model
CREATE TABLE sync_map	a SyncMap model, with the migration that creates it
SELECT … WHERE email = ?	Member.objects.filter(email=…).first()
INSERT … ON CONFLICT DO UPDATE	SyncMap.objects.update_or_create(...)
conn.commit() per record	with transaction.atomic(): per record

Nothing else changes. The connector function, the manifest, and the Workfile are the same.