The runner and the payload

> Evidence: 16 file:line witnesses as of 2026-08-18.

Evidence: 16 file:line witnesses as of 2026-08-18. Confidence: high, most rules here have a named witness. What changes it: the first module built against this doc (../PLAN.md forge-1) and its instance breakdown.

Every module and every app that runs work has two halves: the machinery that accepts, dispatches, runs, records, and reports work, and the domain code that does the work. This doc names that boundary, states it as a rule, and shows the three places it was already drawn without anyone enforcing it.

Audience: anyone laying out a new module, or deciding where a piece of code goes.

The rule#

A module is a runner wrapped around a payload, and the two never import each other's vocabulary.

The runner knows about jobs, items, callers, tenants, dispatch, attempts, outcomes, usage, and logs. It does not know what a part type is.
The payload knows what a part type is. It receives one item and a context, does its work, and returns an outcome. It does not know what a queue is, how many attempts it has had, or where its result is going.

The runner calls the payload through one small, named verb set. The payload talks back through an equally small one (report progress, meter usage, log a line, ask for review). Anything the payload needs that is not in that verb set is a request to change the contract, not a reason to reach around it.

Why this is the load-bearing rule#

Everything else in this tree hangs off it. If the seam holds, then:

swapping a payload (a new part-type matcher) leaves the runner, and every caller's integration, untouched
swapping a runner's provider (Cloud Tasks for arq) leaves every payload untouched
running two versions of one capability side by side is two payloads under one runner, not two services with two auth stories
the runner is written once, tested once, and its behaviours (retry, cancel, heartbeat, metering) are true for every capability at once, instead of copy-pasted into each

The estate's copy-paste history (charter, "The problem") is what happens when the seam is not named. Speedway's checkpoint loop exists three times because nothing said it was the runner's job (../evidence/20260817-speedway-recon.md §3, "Is the seam clean or tangled").

Three witnesses#

The boundary was drawn independently in three stacks and it lands in the same place each time. That is the evidence this rule rests on. It says nothing about which vendor should sit under it.

versable-runner: `app/` vs `lib/`#

app/ (12 files, 1,656 lines) is the runner: auth, dispatch, task handler, store, cache, jobs, observability, usage, serialization, api, main. lib/ plus overrides/ is the payload: the method registry and everything it imports. Two services (services-api, runner-service) were forked from this and diverged hard enough that one owns its lib/ and the other syncs it, and app/ stayed byte-identical in 8 of 12 files; the other 4 differ only by features one fork picked up and the other has not (../evidence/20260817-runner-four-file-diff.md; diff -rq over the two app/ trees returns exactly api.py, jobs.py, main.py, observability.py). The whole of app/ is runner. The domain never leaked in.

The runner calls the payload through one seam, visible as app/jobs.py's imports from lib: process_pipeline (lib/pipeline/processor.py), the PIPELINE_METHODS registry (overrides/lib/pipeline/methods/__init__.py), the Pipeline type, and NonRetryablePipelineError. The payload talks back through the return value, the error type, and the usage sidecar. One nuance worth stating: lib/ also holds a small shared kernel the runner imports (Config, PrintLogger, the cache ABC), so "lib is payload" is loose; the precise statement is that nothing in app/ names a domain concept.

speedway: `modules/run.server.ts` + `queue.server.ts` vs the four modules#

processRun (app/lib/modules/run.server.ts:7-58) is 58 lines of pure switch-and-delegate: load the run, mark the stage running, dispatch on run.module to one of four payload processors, re-roll the stage, chain the orchestrator. queue.server.ts is the Cloud-Tasks-or-in-process dispatch abstraction, uniform across every task type. Neither imports anything from the domain.

The payload modules (partType, scrape, normalize, content, 460 to 1,150 lines each) talk back through a small consistent verb set: getRun, updateRun, settleRunOnce, createJobLogger, rollUpStage, and the usage engine's recordUsage.

The lapse, and it is instructive: the checkpoint loop (resume from cursor, batch, stamp heartbeatAt, finalize with the finishRun / markError / markCancelled trio) is a runner behaviour, and it lives inside three payload files nearly verbatim (partType.server.ts:114-187, normalize.server.ts:440-501, content.server.ts:628-733). The seam is clean at dispatch and leaky one layer down.

walmart-mvp: `orchestrator.py`, `jobs.py`, `worker.py` vs the stage packages#

The runner is orchestrator.py (gating, evaluate_job, the error lifecycle, the stage-log contract), jobs.py (arq task registration, thin per-stage wrappers), worker.py, dispatch.py, models.py, and routes/_common.py. The payload is one package per stage: ingest/, taxonomy/, spec/, scraping/, images/, content/, walmart/, each exposing a *_service_v2.py entry point the runner calls by name. Gating is decided over two structures every part carries regardless of domain, Part.stage_coverage and open PartError rows.

One leak, documented in the code as deliberate: orchestrator.py carries CONTENT_SCHEMA_KEYS and IMAGE_FIELD_KEYS (:78, :90), two lines of Walmart vocabulary kept there so spec/ and content/ need not import each other.

And one duplication, which is a different thing. poll_feed_status builds its own backoff, time ceiling, abort flag, and terminal states (walmart-mvp/backend/app/jobs.py:636-771), beside the machinery orchestrator.py already provides. It is not a payload leak: jobs.py is on the runner side of walmart's own seam (instances/walmart-mvp.md, runner file list), and rg -n "def poll_feed_status" over that tree returns exactly one hit, in jobs.py. walmart/service_v2.py only calls it. An earlier version of this doc placed the function in service_v2.py and read it as a payload reaching for something the runner withheld; both halves were wrong.

The consequence is real either way: "how does a long-running background operation get retried or timed out here" has two answers depending which runner file you are in.

What the lapses teach#

The two cases are not the same shape, and separating them matters because only one of them witnesses the rule below.

Speedway's is a payload leak. Three payload modules (partType.server.ts:114, normalize.server.ts:440, content.server.ts:628) each hand-implement the same checkpoint loop, because the runner offers no helper. After the correction above, speedway is the only witness in the estate for that form.

Walmart's is duplication inside the runner. One runner file grew a second job-lifecycle machine beside the first. Nobody reached across the seam; the runner simply answered the same question twice.

Both still argue for the same fix, which is why the rule survives losing a witness: any lifecycle primitive the runner withholds gets rebuilt by whoever needs it, worse and N times. Speedway shows it rebuilt in the payload; walmart shows it rebuilt in a second corner of the runner. The runner's verb set has to be complete enough that neither happens. The runner owes the payload, at minimum:

The payload needs to	The runner provides
do work over many items and survive a restart	a checkpointed iteration helper: give me the next batch, I hand back progress, you persist the cursor and the heartbeat
wait on something slow outside (a vendor job, a feed)	a poll-with-backoff-and-ceiling primitive that owns re-enqueue and terminal states
stop when asked	a cancel signal it can check between units of work, and a hook to cancel vendor-side work
say what it spent	one metering call, idempotent, that never throws
say what happened to one item	one outcome type: result, terminal error, or needs-review
write something a human will read later	one logger, scoped to the job and item
trace a model call	one tracing wrapper, no-op without keys; the payload never imports the tracing client (versable-runner's `lib/providers/gemini` importing Langfuse directly is the leak)

04-dispatch-and-workers.md and 07-observability.md specify these.

A fourth witness, measured at extraction (2026-08-18)#

When services-api/app/ was lifted into the foundry_runner package (runner step 0, ~/Code/Versable/foundry-runner, commit dcdd0dd), the seam got a number. Of 1,656 lines across 12 modules, exactly one line did payload work (app/jobs.py:327, the process_pipeline call). Two more lines read the payload without executing it: one read the registry to validate a submitted step list, one walked the payload's pydantic argument models to render the prose guide. Everything else was lifecycle. So the seam is one call plus two discovery reads, and both reads are the kind this rule already names as runner-side vocabulary (which capabilities exist, how to describe them), not domain knowledge.

Config said the same thing before anyone looked: of about fifty settings in services-api's Config, the runner half touched twelve, and every one was already named RUNNER_*. The seam had been in the naming all along.

One debt the extraction carries and step 1 retires: the guide is still payload-authored (the payload's describe() writes its own prose and the runner renders it), where canon/11 wants it manifest-generated. Known, owned by the runner plan, not a lapse to relitigate.

Where the seam sits in a repository#

Two acceptable layouts, both witnessed, and a third worth naming because the extractor uses it:

Package split: app/ (or runner/) and lib/ (or payload/, modules/), with the registry as the only import from runner into payload. versable-runner and speedway.
Directory-per-stage: a flat runner core and one package per capability, each exposing one entry point by name. walmart-mvp.
Process boundary: the runner shells out to the payload as a CLI and reads its output files back (the extractor webserver, server.py:22, invoking the CLI engine as a subprocess). It buys real isolation and a payload that runs standalone from a terminal, and it costs any enforceable contract, since the interface is argv plus a folder layout, and it leaks the moment the runner reads the payload's private layout directly (server.py:21). Acceptable for a payload that is genuinely a tool; the contract in that case is the argv and the file schema, written down, and the runner reads only through it.

What is not acceptable: a runner file that imports a domain type, or a payload file that imports a queue client, a store client, or a status enum. Both are cheap to grep for and worth a lint once template code exists.

Do-nots#

Do not put a runner behaviour (checkpoint, heartbeat, backoff, cancel, metering, finalization) inside a payload, however local it feels. Add it to the runner's verb set and call it. (speedway partType.server.ts:114, normalize.server.ts:440, content.server.ts:628)
Do not answer the same lifecycle question twice inside the runner either. A second machine beside the first means the verb set is incomplete, and the cost is that "how does this get retried" has two answers. (walmart jobs.py:636-771, beside orchestrator.py)
Do not let the runner name a domain field, even to avoid a payload-to- payload import. Give the payloads a shared package instead. (walmart orchestrator.py:78,90)
Do not read the four divergent runner files in versable-runner as extension points. They are version skew, and the fork is not evidence about what varies between capabilities. (../evidence/20260817-runner-four-file-diff.md)