Skip to content

0014. Deliver outbound data in two modes over one pipeline

Date: 2026-08-24 Status: Accepted

Context

Conflux reads. Every verb on IntegrationContract fetches from a shop, verifies a connection, or accepts what a shop reports. Every counter on SyncBatch describes a run coming in: what was inserted, what was updated, what was left untouched, and what reconciliation removed because the source no longer had it. Nothing in either of them moves a record the other way.

The declaration for that other direction is already in place. The direction matrix from ADR-0004 lets a system declare push_out per domain, and ADR-0005 refuses a SyncRule whose target cannot receive the domain that rule carries. The target side of every rule is therefore valid, stored, and read by nothing that delivers.

Two consumers want that data, and they want it in two different shapes. rapidmail reads Conflux over the API and sets its own pace. The User Platform is written to, and Conflux holds the position on its behalf. That difference is real, and it is also the only thing that differs. Before either one can happen, something has to establish four facts: what changed, where it has to go, what the payload looks like, and what the recipient already has.

Push-out ships first, because it is what the User Platform needs and it is the half with nothing behind it at all. Pull-out is not dropped. It is a later phase, and it is the reason this record is written now rather than then. A write mode built as its own vertical would leave the read mode to answer those same four questions a second time, its own way.

None of this exists in code yet. The stages arrive with the tickets that follow this one in CFX-206, and each of them is built against a test-double writer, because there is no live target to write to while the layer is being assembled. No system declares push_out either, so a Conflux running the finished pipeline still delivers nothing.

Decision

Outbound delivery is one pipeline of five stages, and only the fifth knows whether Conflux is being read or is writing. The four before it answer the four questions both modes ask, in the same order, from the same rows:

  1. Change signal. What changed. A content_changed_at timestamp that moves only when a meaningful field moves, and a deletion_reason that says why a row went away. The timestamp arrives with CFX-139, the reason with CFX-212.
  2. Routing. Where it has to go. The active routes for a changed row, gated on connection health, with self-suppression so a route never delivers a change back to the connection that produced it. CFX-140.
  3. Projection. What the payload looks like. A persisted row becomes its canonical DTO, the inverse of what a transformer does on the way in. CFX-213.
  4. Delivery state. What the recipient already has. The per-rule cursor and the per-record ledger. CFX-214.
  5. Write or serve. The only stage that differs. Writing is CFX-215 through CFX-218. Serving is the pull-out phase, and it reads stages 3 and 4 through an API instead of calling a writer.

Stage 5 is a leaf, not a spine. Everything above it is built without knowing which of the two modes it feeds, so the read mode arrives as a second leaf rather than as a second pipeline. The stage boundaries are queue boundaries, and trace context already crosses those. One delivery is therefore followable from its change signal to its write once the backend from ADR-0011 is collecting spans.

The write-side contract stays separate from IntegrationContract. A system can be source-only, target-only, or both, and the matrix already says which per domain. WooCommerce is a source today and a plausible target tomorrow. The User Platform is a target and never a source. Write verbs on the read contract would hand every integration a set of methods it has to declare and must never use. AbstractIntegration would carry a no-op body for each of them. A separate contract lets a class opt in by implementing it, so "can this connection be written to" is answered by the type and the matrix rather than by whether a method throws. Which verbs that contract has is ADR-0015's question, not this record's.

A scheduled watermark scan reads forward from a per-rule cursor. There is no outbox. A scan is cheap: one indexed query per active rule, on the same schedule that already carries sync dispatch. It is self-healing, because a run that dies leaves the cursor where it was and the next run covers that ground again. It batches naturally, because it holds a set of rows rather than one event. It adds no write to the ingest path, which matters most when a bulk sync is writing tens of thousands of rows and an outbox would double every one of them. The decisive reason is the last one: the cursor a scan reads from is the position pull-out would hand a consumer, so the read mode inherits the mechanism instead of needing one of its own. An event-driven kick can be added on top later if delivery latency turns out to matter, and it would move this same cursor.

The cursor and the ledger are two objects, because they answer two different questions. The cursor says what to send next. It is one position per rule, and it has an exact pull-out equivalent, because a reading consumer needs the same answer to the same question. The ledger says what happened to each record and which identifier the target gave back for it. It has no pull-out equivalent at all, because a reading consumer holds that state in its own system and Conflux never learns it. One object holding both would bind a per-rule position to a per-record history, and the read mode would inherit a table it can never have a row in.

The ledger is keyed by source identity, (sync_rule_id, data_domain, source_id), not by the local row. Keying it by customers.id is the obvious choice, and it is wrong, because that row does not survive the connection lifecycle. ADR-0013 makes disconnect and uninstall ordinary states rather than incidents. A disconnect soft-deletes the connection's business data, an uninstall removes it for good, and a later reinstall syncs fresh rows carrying fresh identifiers. A ledger keyed on the local row would lose its reference on every one of those rebuilds. The next delivery would then create a second record in the target where it should have updated the first. source_id is the identity the source system gave the record, so it outlives a rebuild for the same reason the mirror can be rebuilt at all.

The cursor advances past every record it attempted, including the ones that failed. Holding it at the first failure until that record succeeds is correct exactly once and then stops the rule. One record a target refuses, for a reason that target considers permanent, would hold every later change behind it for as long as the refusal lasts. A poison record must not stall a stream. Failures are written to the ledger instead, with an attempt count and a next attempt time, and a separate pass drains them on their own schedule. A target that is down is a different case and is already covered. It opens the circuit breaker from ADR-0009, so the run stops rather than marking a thousand records failed one at a time.

Consequences

Pull-out later costs a controller, not a second warehouse. The read mode needs stages 3 and 4 and nothing else that is new: a projection it shares, and a cursor it hands to a consumer instead of advancing itself. What it adds on top is an endpoint and the authentication for it, and ADR-0006 already decided the second half of that. It adds no second store of outbound data, and no second definition of what a change is.

Production behaviour is unchanged until a system declares push_out. None declares it today, and the matrix is the only thing routing consults. A rule whose target does not declare the direction resolves to no route, and the runner delivers nothing. The layer can therefore be merged stage by stage while every existing connection behaves exactly as it did. The cost sits on the other side of that same coin: nothing in the pipeline is exercised against a real target until one declares the direction. The first live rule is also the first real test.

The invariants that keep both modes on one pipeline are obligations on every ticket that follows, not intentions. The payload shape lives in the DTO layer and never inside a writer: a writer maps a canonical DTO into one target's request shape, and a field only one target wants is either defined in the DTO or not sent at all. content_changed_at stays delivery-neutral: it records that a record changed, and it is never suppressed, delayed or rewritten for a reason only a write cares about. Cursor semantics are identical in both modes: the position is (content_changed_at, id), the lower bound is inclusive, and the guarantee is at-least-once. Push state lives in the ledger: customers, orders and products get no push-specific column, because a column there is one a reading consumer would have to be told to ignore. Break any one of these and pull-out stops being a controller.

At-least-once means a record can be delivered twice and never skipped. The guarantee leans that way on purpose. The overlap a scan reads back over, and a cursor that advances past failures, both trade a possible repeat for never losing a change. What makes a repeat free rather than harmful belongs to the ledger, and CFX-214 builds it.