Evidence: 14 file:line witnesses as of 2026-08-18. Confidence: high, most rules here have a named witness. What changes it: the first module built against this doc (
../PLAN.mdforge-1) and its instance breakdown.
The map of everything a production module or app has to answer, with how
each of the three instances in the matrix answers it today and where the
contract lands. Five instances are broken down in ../instances/; the matrix
compares three of them because those three were read first and are different
enough to argue with each other. This is the index for canon/; each numbered
doc below takes one row or a cluster of rows and states the rule with its
reasoning.
Audience: anyone deciding what a new module has to cover, or checking whether an existing one is missing something.
How to read the matrix#
The three instances are different enough that agreement between them is evidence and disagreement is a real choice:
- versable-runner (
services-api,runner-service): Python, Cloud Run, Cloud Tasks, GCS as the only datastore. A pure module: no users, no UI. - speedway: TypeScript, React Router SSR on Cloud Run, Firestore, GCS, Cloud Tasks in prod and in-process in dev. An app with its runner inside.
- walmart-mvp: Python FastAPI, Postgres on Cloud SQL, Redis + arq, GCS.
An app with its runner inside, four Cloud Run services from one image
(
walmart-api,walmart-api-prod,walmart-worker,walmart-worker-prod, read from live state 2026-08-17): api and worker split by role, prod and non-prod split by environment.
Stance column: settled (all instances agree, or the disagreement has an
obvious winner; the contract fixes it), recommended (the contract names
a default and the trade-off; a module may deviate with a reason),
open (nobody has built it well yet, or the instances split and the
contract has to design forward). Evidence: ../evidence/ and ../instances/.
The matrix#
| # | Concern | versable-runner | speedway | walmart-mvp | Stance | Doc |
|---|---|---|---|---|---|---|
| 1 | Runner/payload seam | app/ vs lib/, 12/12 files runner; survived a fork | modules/run.server.ts + queue.server.ts vs modules/{partType,scrape,normalize,content} | orchestrator.py, jobs.py, worker.py vs ingest/, taxonomy/, scraping/, content/, walmart/ | settled | 01 |
| 2 | Caller identity | one shared password for all callers, X-API-Key or bearer; nothing verifies meta | session cookie, hand-rolled password auth, no IdP | bearer JWT HS256 for the SPA; no machine-to-machine scheme | open, design forward | 02 |
| 3 | Tenancy | none; meta.{service,organization,user} is a self-declared tag | org (billing) → workspace; every path scoped workspaces/{wid}/… | X-Org-Id header checked against membership; per-resource require_membership | settled: tenant is mandatory on every job | 02 |
| 4 | Roles / RBAC | none | member < admin < owner per org | owner/admin manage, member otherwise | recommended: module receives scopes, app owns roles | 02 |
| 5 | Submit surface | POST /jobs {data, method, params, meta} and POST /jobs/run-file; 422 on unknown method | form action per module; two-phase create with setupHold | multipart upload; synchronous dupe check in-request | recommended: one job-submit shape, O(1) in items | 03, contracts/module-surface |
| 6 | Job state ownership | module owns; caller polls | app owns (job doc + runs subcollection) | app owns (Job row) | open: module-owned, caller-owned, or mirrored | 03 |
| 7 | Derived vs stored status | derived from blob listings; nothing to go stale | job status derived in a transaction from the stages map; run status stored | stored at two altitudes: Job.status/stage/stage_state with one writer (evaluate_job), Part.status with six writers | recommended: derive where listing is cheap; else one mutation point per altitude | 03 |
| 8 | Dispatch / queue | Cloud Tasks, self-push, one fanout task then one task per item | Cloud Tasks HTTP task in prod, setImmediate in dev, shared secret on callback | arq over Redis, deterministic job ids | recommended: queue behind a port with an in-process mode | 04 |
| 9 | Concurrency | queue maxConcurrentDispatches and nothing else | one active run per job+module, transactional | arq max_jobs=2 after an OOM | recommended: name the semaphore; per-tenant fairness is app-side | 04 |
| 10 | Retry / backoff | handler owns 3 attempts, queue is a backstop at 5 | stall recovery ×3 from cursor; no queue retry config | none for the current pipeline (job_timeout=3600 only) | settled: the handler owns the budget; the queue is a backstop | 04 |
| 11 | Heartbeat / crash recovery | none needed: a dead task is redelivered; result writes are guarded | heartbeatAt per batch, STALL_MS 3 min, transactional requeue | none for the current pipeline; legacy pipeline has it | settled: any run longer than one dispatch needs a heartbeat | 04 |
| 12 | Cancel | marker blob + delete pending tasks; in-flight finishes | cooperative flag checked between batches; vendor cancel for scrape | none for the current pipeline | settled: cooperative cancel is mandatory; state after cancel is defined | 04 |
| 13 | Resume / checkpoint | per-item, so resume is "redeliver the item" | cursor on the run doc, copy-pasted in three payload files | upsert-idempotent re-run, no cursor | recommended: checkpoint as a runner helper, not payload convention | 04 |
| 14 | Idempotency | deterministic task names, if_generation_match=0 on writes | settleRunOnce first writer wins; intent doc before paid calls; per-event usage ledger | deterministic arq ids; part_error_id; publish dedup window | settled: idempotency is structural, at every write that costs money or ends a run | 04 |
| 15 | Storage of inputs | payload.json + items/{idx}.json in GCS | files in GCS, rows in Firestore | files in GCS, JobFile rows in Postgres | recommended: bytes in object storage, keyed by tenant/job | 05 |
| 16 | Storage of outcomes | results/, errors/, usage/ per item in GCS | run doc + chunked log subcollection + denormalized stages on the job | Part row per item, PartError per (part, field) | recommended: per-item outcomes addressable by index, one place | 05 |
| 17 | Results reporting | GET /jobs/{job_id}/results?offset&limit, meta filters, /errors, /stats | list views read the denormalized rollup | routes per resource | settled: pagination + filter on every list; per-item fetch | 03, contracts/module-surface |
| 18 | Logs, per job and per item | usage/{idx}.json sidecar; no per-item log stream | workspace events + per-run chunked log, resumable across workers | stage log capped at 20 lines, latest-run-only; stage summaries JSON | open: the per-item log a caller can read is the missing piece | 07 |
| 19 | Tracing | Langfuse via lib/ (observe on the Gemini provider), keys default empty; the payload owns the tracing decision, a seam leak | none | Langfuse, optional, per process | recommended: Langfuse behind a no-op default, wrapper provided by the runner | 07 |
| 20 | Usage metering | duration, attempts, tokens, LLM cost per item and per model | idempotent per-event ledger, dead-letter + replay, decoupled from run outcome | in-process buffer flushed per 50 events, best-effort by design | recommended: speedway's shape; cost per item is a first-class number | 08 |
| 21 | Limits / quotas | none | checkLimits before work, soft/hard/overage modes | none | recommended: module reports, app enforces; module needs a hard ceiling per caller | 08 |
| 22 | Caching | GCS-backed KV keyed by pipeline input | cross-workspace scrape cache (SERP in Firestore 7d, page text in GCS 1mo) + in-process TTL microcache for the auth chain | Redis for queue + rate slots, not results | open: cross-service result cache; Redis on Render exists | 06 |
| 23 | Rate limiting outbound | none | OXYLABS_CONCURRENCY etc. as env caps | Redis-backed fleet-wide slot limiter per vendor | recommended: walmart's shape when workers scale past one | 04 |
| 24 | Config | one Config, every var has a default, zero-env import works | one flat object from process.env at load | pydantic Settings, .env + env | settled: one typed config object, defaults for local, no raw reads | 09 |
| 25 | Secrets | Secret Manager, injected by reference; 8 secrets | env | env | recommended: reference-injected, never printed | 09 |
| 26 | Human in the loop | none | review queue, "review-quiet" as an orchestrator trigger | PartError rows are the queue; resolve/skip re-evaluates the job | open: module emits "needs review" as an outcome; app owns the queue | 10 |
| 27 | Completion signalling | poll | in-process chaining; its extractor client receives signed webhooks and runs a reconciliation sweep | poll; Walmart feed status by self-re-enqueue with backoff | open: poll is the floor, webhook is the target, sweep is the guard | 03 |
| 28 | Capability discovery | GET /usage self-documenting text for LLM agents; /docs Swagger | none | none | open: a machine-readable manifest per module | 11 |
| 29 | Versioning | none | /build-info reports commit, branch, kit, build time | GET /api/build-info reports commit SHA | open: contract version, module version, payload schema version | 11 |
| 30 | Health / readiness | /health, /health/deep with GCS round-trip and effective config | /livez (/healthz is edge-reserved on *.run.app), /build-info | /api/build-info | settled: liveness, deep health with config echo, build identity | contracts/module-surface |
| 31 | Provisioning / environments | deploy.sh, idempotent, env-parameterized; dev shares prod's service account | release.sh and deploy/deploy.sh, not interchangeable | branch→env mapping, one image four services (api/worker × prod/non-prod), Cloud Build | recommended: walmart's branch mapping; per-env identity | 12 |
| 32 | Local dev / debugging | RUNNER_DISPATCH=local in-process semaphore | in-process when Cloud Tasks vars unset | storage_backend=local, stub_feed_submit | settled: every module runs in-process with zero env; the debug path is a contract concern | 13 |
| 33 | Output delivery | S3 for rendered images (AWS_*) | exports built in-app | GCS public bucket for rehosted images | open: how a module hands artifacts back | 05 |
| 34 | Data retention | none | none | none | open | 05 |
| 35 | Graceful degradation / multiple versions | two forks of the same runner run side by side by accident | n/a | two pipeline generations in one database and one codebase | open, and a recurring failure mode: three times the estate has run two versions of one capability (runner forks, walmart's generations, the extractor's VM and Cloud Run deployments), every time by accretion, never able to route between them (instances/extraction.md) | 14 |
| 36 | Conformance | none | none | none | open: a checklist and a smoke suite a module can run against itself | contracts/conformance |
| 37 | Outputs and exports (M from N, partial reads, edits) | /results for run-file jobs returns a resume-file dict, a transform in disguise | exports built in-app over parts and runs | rehosted images to a separate public bucket; exports over Part rows | settled by App V5's output layer: outcomes raw and immutable, outputs a declared transform | 15, guides/01 |
| 38 | Data ownership split (which store, which layer is truth, where config lives) | one store, GCS; config in env | Firestore + GCS; per-field provenance on parts; config in config.server.ts and data/ | Postgres + GCS + Walmart's side; per-org flags in DB | recommended: module owns its record only, app owns identity/tenant config/catalog, reference data is a snapshot, config classified by content into five buckets; four splits compared | guides/02 |
Note on row 27, the extractor#
The extractor column is not a single answer, and an earlier version of this matrix carried a claim about it inside the speedway cell.
The service speedway talks to does deliver callbacks. Its client documents a
service that returns 202 with a job_id and "delivers rows by webhook and/or
polling", HMAC-SHA256 signed
(speedway/app/lib/extractor.server.ts:5-9); speedway receives and verifies
them at speedway/app/routes/tasks/extractor-webhook.tsx:20-21. The automation
surface documented in instances/extraction.md offers only poll. Those are
different surfaces on different deployments, so "the extractor polls" is true of
one and false of another.
Two properties of that webhook are lapses rather than models. The signing key
is the API key (extractor.server.ts:8-9, and :252
createHmac("sha256", config.extractorApiKey)), so a caller cannot rotate
authentication without breaking signature verification, and anyone holding the
API key can forge a callback. And there is no event id, so the receiver has
nothing to dedup on. contracts/module-surface.md requires a separate
per-caller callback secret and an X-Foundry-Event-Id; this instance is the
reason both are stated explicitly.
What the matrix says#
Three shapes fall out of it, and they organise the rest of canon/.
Settled by agreement. The seam (1), the retry budget living in the handler (10), idempotency at every costly write (14), tenant on every job (3), one typed config (24), pagination on every list (17), and an in-process mode (32). Three implementations agreeing without coordination is about as good as evidence gets; the contract fixes these.
Settled by an obvious winner. Where the instances split and one is plainly better: heartbeat and cancel (11, 12; walmart's current pipeline has neither and can delete a job mid-run), usage metering (20; speedway's ledger is the shape, versable-runner's cost-per-item is the number), checkpointing as a runner helper rather than a payload convention (13; speedway copy-pasted it three times).
Open, and the contract has to design forward. Caller identity (2) is the weakest point in the estate: a shared password, a session cookie, and a JWT with no machine path, none of which can express "app A on Render, on behalf of org X, calling module B on GCP, with a budget". Job state ownership (6), per-item logs a caller can read (18), completion signalling (27), a capability manifest (28), versioning (29), and running several versions of one capability at once (35) have no good instance to copy. These are where the first modules will teach the tree the most.
The fourth witness: App V5 (enhancement-product)#
Recon landed after the matrix was written
(../evidence/20260817-enhancement-product-recon.md), and its breakdown is
written at ../instances/enhancement-product.md. It is a Mongo-poll pull
queue on Render with
Redis and Postgres beside it, and it agrees with the three above on the seam
(one call, process_pipeline, lib/tasks/task_runner.py:319-320), the
handler-owned retry budget, one typed Config, and per-item structured
error codes. It adds mechanisms none of the other three has, now folded into
the canon docs named:
| It has | Where | Folded into |
|---|---|---|
| claim-time fairness: pick a random job first, then a task in it | lib/tasks/claimer.py:80-81 | 04, concurrency |
| two-tier heartbeat: Redis 15 s TTL fast path, 30-min Mongo lock sweep backstop | lib/redis/worker_heartbeat.py, claimer.py:404-449 | 04, heartbeat |
| fleet-wide retry budget and circuit breakers over Redis, breaker-outermost fixed by decorator | lib/redis/retry_budget.py, lib/breakers/* | 04, 14, contracts/runner-verbs |
deferred as an outcome that does not burn an attempt | lib/tasks/task_runner.py:341-350 | 03, 04, contracts/runner-verbs |
| pause and resume as job and item states; running work finishes | api/jobs.py:150-170, lib/tasks/admin.py | 03 |
| mixed-version fleet safety: workers claim only tasks whose names they registered | lib/scheduler/scheduler.py:276-286 | 14 |
| queue namespaced by git branch for PR previews on one Mongo | lib/config/__init__.py:88-93 | 12 |
credits: worker credit-unaware, async idempotent charge, Postgres UNIQUE + ON CONFLICT DO NOTHING | lib/redis/credit_dispatch.py, .claude/notes/credit-flow.md:52-54 | 08 |
| Sentry centralized to one init; Langfuse spans stamped with team and user | lib/sentry/__init__.py, task_runner.py:305-311 | 07 |
| preflight on prod boot, Slack-reported once per role and commit | lib/config/__init__.py:228-237 | 12 |
And the same lapses in a different costume: a shared X-Api-Token for
machine callers (api/auth.py:188-206, row 2), flat team and user
ownership with one hardcoded admin team and no roles (row 4), the runner
importing from api/ for a credit-cycle read (a seam leak with a comment
explaining it), and payload code raising runner-typed exceptions to steer
routing (row 1, the deliberate version of the same coupling).
Not on the list yet#
Scheduling (auto-refresh of stale SKUs) is stage two of the replatform and
is deliberately deferred. Bi-directional PIM sync is out of scope. Anything
the Workflow Console needs that is not a row here gets added when v6/ is
written.