Outbound Delivery¶

In one sentence¶

Conflux delivers outbound data over a single five-stage pipeline whose first four stages do not know whether the data will be read from Conflux or written into somebody else's system.

What it's about¶

Conflux has spent its life reading. A shop like WooCommerce is asked for its customers, orders and products, and those records land in Conflux's own store. This page explains the design behind the other half (CFX-206): getting that data back out again, either because a consumer reads it from Conflux or because Conflux writes it into a system somebody else owns.

Those two ways out look like two features. They are one. Both have to settle the same four facts before anything can move: the change itself, its destination, the shape of the payload, and what the recipient already holds. Only the final step differs. Either the consumer takes the data at its own pace, or Conflux hands it over and holds the position on the consumer's behalf.

Writing goes first. The User Platform needs it, and it is the half of the problem with no code behind it whatsoever. Reading, called pull-out, comes later. The point of settling the shape now is that the later half then costs an endpoint rather than a second pipeline.

Terms used on this page¶

Why the outbound half needed its own layer¶

Every verb on the integration contract points inward. An integration fetches from a shop, verifies a connection, or takes in what a shop reports about itself. Nothing on it hands a record the other way.

The accounting points the same direction. SyncBatch counts what a run brought in: inserts, updates, rows left untouched, and the rows the deletion log removed. It has no counter for anything that left.

The declaration for the other direction is already in place. A system can declare push_out per domain in its direction matrix (ADR-0004), and a SyncRule whose target cannot receive the domain that rule carries is refused at save time (ADR-0005). Every rule's target side is therefore stored and validated, and nothing that delivers ever reads it. It is a declaration, not yet a behaviour.

That gap is not the size of a method. A push() verb on the read contract would leave all four facts above unestablished, and none of them is a shop's business. Which rows changed since last time, which rules carry them, and what a given target already holds are questions about Conflux's own state. They are answered once, by a layer above the integrations, and only the final step of that layer talks to a target at all.

The write side does get a contract, and it is a separate one from the read contract. A class opts into it by implementing it, so "can this connection be written to" becomes a question the type and the direction matrix answer between them. See ADR-0014 for that decision and for why write verbs stay off IntegrationContract.

The five stages¶

graph LR
    CS["1 · Change signal<br/>content_changed_at, deletion_reason"]
    RT["2 · Routing<br/>active routes, health gate"]
    PR["3 · Projection<br/>row to canonical DTO"]
    DS["4 · Delivery state<br/>cursor and ledger"]
    SV["5a · Serve<br/>consumer reads"]
    WR["5b · Write<br/>outbound writer"]
    CS -->|"1 · what changed"| RT
    RT -->|"2 · where it goes"| PR
    PR -->|"3 · what the payload is"| DS
    DS -->|"4 · what the reader already has"| SV
    DS -->|"5 · what the target already has"| WR

1. Change signal establishes what changed and hands it on (edge 1). A content_changed_at timestamp moves only when a meaningful field moves, so a re-import that writes the same values again produces no delivery. A deletion_reason records why a row went away. CFX-139 adds the timestamp and CFX-212 adds the reason.
2. Routing establishes where a change has to go (edge 2). It reads the active routes for a changed row, gates them on connection health, and suppresses the route that would send a change back to the connection it came from. CFX-140.
3. Projection establishes what the payload is (edge 3). A projector turns the persisted row into its canonical DTO, and the row's own delivery metadata is wrapped around it, so what leaves this stage is a self-describing envelope rather than a bare payload: a consumer reads which domain it holds instead of inferring it from the keys the record happens to carry. The payload shape belongs to the DTO layer, never to a writer. A field only one target wants is therefore either defined in the DTO or not sent at all. A soft-deleted row projects like any other, with its deletion marked and its last known state inside: the delivery scan never reads one, but the deletion path that will is better off with the payload than with an identifier alone.
4. Delivery state establishes what the recipient already has (edges 4 and 5). This is the cursor and the ledger, and it is the last stage that is identical in both modes. CFX-214.
5. Serve or write is the only stage that differs. The two edges out of stage 4 are the two ways it can go. A reading consumer is handed the projection and the cursor over an API (edge 4). A write hands the projection to an outbound writer, which maps it into one target's request shape (edge 5). The write half already has its contract, the capability interfaces that say which verbs a given target offers, the base class, the registry and the test doubles behind it, its own chain of jobs, and the guard rails described above; see Outbound Writer Contract. Serving is the later pull-out phase.

Only stage 5 knows which mode it serves. Nothing above it is written against either one, which is what lets the read mode arrive as an added endpoint instead of a parallel pipeline. Keeping it that way is an obligation on every ticket in CFX-206, and ADR-0014 lists the invariants that carry it. Each stage boundary is also a queue boundary, so one delivery becomes followable end to end as soon as the tracing backend from ADR-0011 collects spans. See Tracing for how spans are emitted today.

Why the projection stage keeps no catalogue¶

A stage that keeps its own list of what it can project is a stage a fourth domain can be left out of. That failure already has a precedent here: reconciliation once carried a mapping of its own from domain to table, with a fallback for anything the mapping missed, so a domain nobody had added to it reconciled nothing at all, with no exception and no log. Nothing in the projection stage holds such a list.

Three declarations carry it instead. A model declares itself projectable by implementing the contract for it, and it answers its own delivery metadata off the columns every mirrored table already has, so the stage never reads a column off a model it does not know. A row names its domain through its own class. And one enum names, per domain, the table it is mirrored as, the DTO its records take, and the projector between the two. Adding a domain is those declarations and nothing else, because the entry point has no catalogue to update.

Putting the DTO pairing on that enum reversed an earlier decision. It used to sit next to the envelope, argued for on the grounds that the enum is read by the direction matrix, by sync rule validation and by connection capabilities, none of which has any business knowing about the data layer. It moved anyway. Which table a domain is mirrored as and which DTO its records take are facts about the domain rather than about a direction: reconciliation already needed the first of them on the way in, and the inbound transformer, reconciliation and the outbound projector all address the same three pairs. One declaration serving both directions beats two that can drift, and the readers that only ever ask the enum for its cases pay nothing for the methods they never call.

That leaves one pairing no type can express: that the projector a domain names really does produce the DTO the same domain names. The envelope closes it from the other side. It refuses a record that is not what its domain says it is, so a domain wired to the wrong projector throws at the boundary instead of shipping a payload under the wrong label.

For the shape a new domain has to follow, see Implement a Projector. For the contract signatures and the envelope's fields, see Projection Contract.

Why a watermark scan, not an outbox¶

The obvious way to move a change out is an outbox: every write on the ingest path also writes a row saying that this record changed and has to be delivered. Conflux does not do that. A scheduled watermark scan reads forward from the per-rule cursor instead.

The engineering economics favour a scan, which costs one indexed query per active rule and adds nothing to the ingest path. What settles it is pull-out. A reading consumer would be handed the very cursor a scan reads from, so the read mode inherits the mechanism instead of building one of its own. An event-driven kick can be layered on later if delivery latency starts to matter, and it would advance this same cursor. See ADR-0014 for the full comparison with an outbox.

Why the cursor and the ledger answer two different questions¶

The cursor answers "what next", and the ledger answers "what happened". The cursor is one position per rule. Pull-out has an exact equivalent of it, because a reading consumer asks the same question a writer does. The ledger is one row per delivered record, and it holds the identifier the target gave back for that record. Pull-out has no equivalent of the ledger, because a reading consumer keeps that account in its own system. Keeping the two apart is what makes pull-out a controller rather than a second warehouse. See ADR-0014 for why that split is a condition of the shared pipeline rather than a preference.

Why the batch is the unit of work¶

Delivering under one rule could be one job: scan from the cursor, deliver, retry what failed, and exit whenever the work is done. A backfill makes that a job that lives for hours inside a worker, and hours is exactly what nothing around a queue is built for. A deploy restarts the workers and kills it mid-pass, the worker timeout has to be set for the longest pass anybody might ever start, and the memory one pass accumulates is nobody's to account for. One batch of records per job makes every one of them an ordinary job of a minute or two, and the ordinary answers apply again.

It also buys a handle. The jobs of one rule share a batch, and that batch is a row somebody can look at: when the chain started, how many jobs it has got through, whether it is still open. Cancelling it stops the chain at the next job boundary, which is a stop button a long-running pass did not have — between one tick and the next it was a black box. The same shape lets the framework carry the exclusivity that keeps two chains off one rule, as a lock held for the length of a job rather than one refreshed by hand for the length of a pass.

What it costs is the single report at the end. Each job says what it did, and what the whole chain did has to be read off the batch row and the log filtered by the rule, rather than off one object handed back when the work finishes. Nothing needs that aggregate enough yet to pay for persisting it, and where something does, the per-job report, the cursor and the batch are three places to hang it on.

Why a chain stops or parks a record¶

A delivery chain meets two kinds of trouble, and telling them apart is what keeps a rule moving. Some faults belong to the rule: the route closed while the chain was working, the target's breaker is open, no writer is registered for that target's driver, or the writer has no create where the ledger says a record has to be created. Every record under that rule would meet the same fault in the same way, so there is nothing to be learned by trying the next one. The chain stops, writes nothing about the fault, and leaves the cursor where the last batch that concluded put it. Other faults belong to one record: the target refused the write, or reported that the record it once held is gone, or took a create and handed back nothing that addresses the result. Those land in that record's ledger row, and the job carries on with the next candidate.

The retry account for a record-level fault lives in the ledger rather than in the position, for the reason already given above: the cursor keeps moving forward regardless, so there is nowhere else to hold it. A failed row carries its attempt count and the time of its next attempt, and the jobs at the start of the chain — the drain, which reads failed rows rather than walking the cursor — pick those up once they come due, wherever they now lie relative to the cursor, before the chain walks forward, so a rule that changes faster than it delivers cannot starve its own retries. A record that runs out of attempts, or that was created without an identifier coming back, is parked for an operator instead of retried for ever.

Neither kind of stop rolls anything back. What was delivered stays delivered, the cursor stands where it stands, and the next chain resumes from there rather than restarting — which is the same property a disconnect and a reconnect rely on. ADR-0029 sets out which fault goes which way, why a missing verb ends a chain instead of marking a thousand records, and why an open breaker does not put the target connection into an error state the way an inbound sync failure does.

Why a manual stop and a volume trip are kept apart¶

A rule can also be stopped by a person rather than by a fault, and it can be held back by a third thing that is neither: a volume breaker that judges the withdrawals a step is about to make implausibly large. The two look similar from outside a rule, both leave creates and updates untouched or halted depending on scope, but they answer different questions and are recorded differently on purpose.

A manual stop is the kill switch. It reaches as far as whoever set it decided: every rule, every rule writing into one target connection, or one rule, and it halts every write in that scope, not only withdrawals. It exists because a person needs a lever that works regardless of what the data says, and it stays in force until a person, or the console, ends it.

A volume trip is never a decision anybody made. It is the system noticing that what a rule is about to delete or deactivate does not look like an ordinary change, and it only ever touches that one rule's withdrawals: a create or an update for the same rule keeps going while the hold is in force. ADR-0035 sets out why holding withdrawals, rather than stopping the rule the way a manual stop does, is the right size of reaction to a finding that is usually, but not always, a false alarm.

Why the volume breaker counts per rule and before every step¶

The breaker asks its question before a step withdraws anything, never while it is withdrawing. A tally kept while records go out would let a whole threshold's worth of deletions leave before anything noticed, which defeats the point of having a limit at all. Asking first means the worst a false trip can do is refuse a step that would have been fine; asking after would mean the worst it can do is nothing, because the damage is already done by the time the count catches up.

It counts per rule rather than per target connection, even though a connection can carry several rules. A rule is one source, one target, one domain, and a large shop collapsing overnight would otherwise dilute itself against every smaller shop sharing the same connection: a second, healthy rule's ordinary volume would raise the share the first rule is allowed, hiding exactly the collapse the breaker exists to catch. Counting per rule keeps one rule's trouble from borrowing another rule's room.

What it counts is deliberately narrow: how many records a step is about to withdraw against how many the ledger already knows the target holds. It has no opinion on whether any single deletion is right, that question belongs to the deletion cause (ADR-0019) and the rule's own policy (ADR-0016). The breaker exists for what gets past both: ten thousand deletions that are each individually well-formed, and only implausible in how many of them arrived at once.

A held row counts towards that number only while its record is still deleted. A record that came back in the source is no longer a withdrawal; once the hold ends it goes out as an ordinary update, so counting it would only inflate what support has to confirm.

When support releases a hold, the release belongs to the rule, not to a chain or a step, and it stays in force until the rows it confirmed are gone. A chain can be anywhere when support acts. Held rows are handed back only by the drain, at the start of a chain, so a release spent by whichever step met it first could be spent by a scan that withdraws nothing, or by a chain that dies after one chunk, and the next chain would recount and trip again over the very withdrawals support had just confirmed. Keeping the release in force until no held tombstone is left means every step meanwhile gets it, and it ends exactly when there is nothing left for it to cover, or earlier, when a new hold trips: that hold's finding already counts the confirmed rows still held, so whatever support decides about it covers them too, and a discarded hold must not leave the older allowance behind to let out what support just declined. A newer release replaces an older one: the older answered a finding that nobody has confirmed since. The allowance is added on top of the normal limit rather than put in its place. Replacing the limit would make a release brittle: one more deletion arriving between the finding and the release would push the count past the confirmed number and void the whole release, although a single extra deletion is exactly what the normal limit exists to let through.

Why delivery paces itself against the target¶

A burst does not need a backfill to happen. A price change across a catalogue, a bulk consent import in the shop, a large retry drain after an outage: all of these can put more writes into a minute than a target accepts, on an otherwise ordinary day. A target that publishes a rate enforces it, and delivery now paces itself against that rate rather than finding out the hard way.

The budget belongs to the target connection, not to the rule. A target's limit is a fact about the workspace the connection points at, and every rule writing into that connection meets the same target, so every rule spends from the same budget.

A rate-limit response is a statement about the target, not about the record it answered. The record that met it is not broken, and nothing about it needs a retry counted against it. What changed is that the target asked Conflux to slow down, so delivery blocks the whole connection for the wait it named, and every other record waits behind it too.

A short wait is worth sitting out; a long one is not. The chain never holds a worker while it waits: it releases its own next job, which gives its worker back and comes again once the wait is over. A wait long enough to threaten the chain's own liveness is different. The run ends instead, so the heartbeat stays truthful rather than going stale behind a job that is only waiting, and a rule that is waiting out a long block reads as throttled rather than as failing. The waiting job holds to the same bound, so one that wakes to a longer block ends the run the same way. The block belongs to the connection, so a rule on it that would otherwise turn stale while it runs reads as throttled instead, not only the one that met the rate-limit response.

See ADR-0036 for the reasoning behind each of these choices.

Why the ledger is keyed by source identity¶

The ledger is keyed by (sync_rule_id, data_domain, source_id), the identity the source system gave the record, rather than by the local row id. Disconnecting and uninstalling take the local rows away, and a reinstall brings the same records back under fresh identifiers (ADR-0013). A ledger keyed on the local row would lose its pointer at every such rebuild, and the next delivery would then add a second record to the target instead of updating the one already there. See ADR-0014 for the decision and for why the obvious local-id key is the wrong one.

What the duplication model costs¶

For every source record, Conflux creates a record of its own in the target, and it never searches the target for one it already holds. ADR-0015 argues why that is the right rule and why the alternatives are worse. It is also the most expensive rule on this page, and the bill arrives in three parts.

The ledger cannot be rebuilt. Every pointer into the target exists once, in that table and nowhere else. The target cannot be asked which records are Conflux's, because nothing distinguishes them from the ones it holds natively. Lose those pointers and every record Conflux created stays in the target out of reach. No update reaches it, a withdrawn consent included, and no later deletion finds it either. Reinstalling after an uninstall therefore builds a second full set of records rather than picking the first set back up. Uninstall still has to purge the table, because each stored reference names a person. Pruning the table to save space is not a free optimisation either, because a pruned row is a record no one can reach again.

One person can carry two consent states. The target workspace ends up holding two records for somebody it already had. One is its own contact, with whatever consent that contact was given. The other is the record Conflux created, with the consent the shop had on file. Neither copy is ever reconciled against the other. Two sources produce the same split, with one shop's record standing in for the workspace's contact: a tenant whose WooCommerce shop and PrestaShop shop both sold to the same person delivers two records, not one (ADR-0016).

A backfill's side effects scale with dataset size. One tenant with 50,000 customers means the same number of creates in the target. Every one of those is a change that target instance may be configured to act on. There is nothing to resolve against, so there is nothing to skip, and no cheaper shape is available. A rule's first activation nonetheless delivers what the rule already carries, because that is what connecting a shop asks for, and the throttle, the kill switch and the volume breaker bound it (ADR-0037); a backfill started by hand later is what the confirmation described under What a write can set off in the target is for.

What a write can set off in the target¶

A write from Conflux is indistinguishable from a person editing that record by hand. The target sees an ordinary update on one of its own records, and it applies it.

That cuts both ways. A write triggers nothing by itself, and no flag on the request can rule anything out either. What runs is whatever that target instance has been set up to run when one of its records changes, up to and including messages to people. Where nothing is set up, a write is inert.

The exposure therefore belongs to one customer's setup, not to the target's API. Code cannot check it, because the configuration lives in a system Conflux does not own and does not read. The answer is procedural instead. A backfill somebody starts by hand against a live target needs a confirmation from somebody who has checked that instance's configuration and found a change of that size safe there. The start dialog says how many creates the backfill would send, and that number has to be typed back. The bound and the number are fixed when the dialog opens, so records arriving meanwhile do not move them. Conflux counts again against that bound when the start is submitted and refuses the start only where the backfill would now send more creates than were confirmed. The confirmation is stored with who gave it, for which rule and target, and what the backfill was going to cover. It is a record that somebody looked. It does not stop the target from reacting, and it cannot, for the reason above. There is no console path to a start, because a confirmation is only worth its link to a signed-in person (ADR-0046). A dependent rule, such as consent, offers no start by hand at all. Its records are never created, only written onto a parent the target already holds, so a number of creates says nothing about them, and a consent backfill into Positive User would send one double opt-in mail per unconfirmed contact (ADR-0046). The initial backfill a rule's first activation runs carries no such sign-off: connecting a shop is already the merchant asking for their data to be there (ADR-0037).

The same indistinguishability has a cost pointing the other way. A manual edit made inside the target is overwritten by the next delivery for that source record, and nobody finds out. Conflux does not read the record back, so it cannot report the overwrite. The editor is told nothing either, because the target saw a routine update and applied it as one. What a rule may ask of a delivery is bounded by two per-rule dials, update_mode and delete_policy. See ADR-0016 for the defaults and for why the policy sits on the rule rather than on the system.

What happens when a record is deleted at its source¶

A deletion needs two things to agree before it reaches somebody else's system. The policy says what the customer permits, and the recorded cause says what actually happened.

Only source_event ever propagates: the source stated that the record is gone. A connection_lifecycle teardown never does, because a disconnect or an uninstall removes rows from Conflux's mirror and says nothing about the records those rows were delivered to. A record whose cause was never recorded is read the same conservative way. See ADR-0019.

A deletion Conflux never delivered has nothing to withdraw, so a tombstone whose ledger row holds no remote reference is left alone rather than delivered as a create followed by a delete.

What a delivered deletion leaves behind differs by verb, and the difference is what makes a later restore behave. After a delete the reference points at nothing, so it is cleared and a restored record is created afresh. After a deactivate the record is still in the target, so the reference stays and the row records that the record is inactive. A restored record is reactivated through that reference first, and only then does the rule's update mode decide whether its content is written. The outcome is recorded as DeliveryStatus::Withdrawn, which is not terminal: a record restored in the mirror moves its change signal past what was delivered and is offered to the scan again. See Outbound Delivery State for the ledger row this leaves.

Why a restore is reactivated even under create_only¶

An update writes fields. It does not make an inactive record active again, so a restore needs a verb of its own, and reactivate is that verb. A writer that can deactivate can always reactivate, because CanDeactivateRemoteRecords carries both. A target that could only do the first would leave every restored record inactive for good.

create_only has no say over the reactivation. The mode exists to leave alone what somebody may have edited in the target, and the inactive state is not such an edit: Conflux put it there itself. Undoing it overwrites nothing. What create_only does govern, the record's content, is still left alone afterwards.

The ledger remembers the deactivation in a column of its own, remote_deactivated_at, rather than reading it off the status. A reactivation that fails turns the status into failed, and the retry still has to know the record is inactive. A target that no longer holds the record at all ends delivery as remote_gone, the same answer an update gets. Creating the record again could undo a deletion somebody made on purpose.

Dependent records¶

A record of a dependent domain, today a customer's consent, is never created in a target on its own: it is written onto the record of its parent there, the contact. Its rule is bound to a rule of the parent domain between the same two connections, the anchor rule, and runs only while that rule runs. RecordDelivery takes each of its candidates through five steps, and the first that applies decides.

1. Anchor. Which target record the consent belongs to is read from the parent domain's ledger, the reverse direction first, afresh on every delivery. Where the target does not hold the parent yet, the row waits as awaiting_anchor, using no attempt. When the anchor rule creates the parent, every dependent of that parent is made due again, so the consent follows right after its contact. See ADR-0042.
2. Echo. Where the target's own mirror of the same consent already holds the value, nothing is sent. For consent that means subscribed and confirmed, with null on either side counting as no difference. A mirror row the target's integration does not report in full (Positive User's SMS, declared through HasFullyReportedConsentChannels) takes no part here or in the conflict step, and the consent is delivered. See ADR-0043 and ADR-0045.
3. Conflict. Where the rule back from the target is due for the same consent too, the later consent_updated_at wins, a tie going to the candidate in hand, and the winner marks the other side superseded. This runs under a lock on the pair of rules. See ADR-0043.
4. Delivery. The writer's update() is called with the anchor as the remote reference. There is never a create, and the rule's policy is fixed to overwrite and ignore.
5. Outcome. A target that lost the parent makes the row remote_gone, which for a dependent record is looked at again on its next change. When the anchor rule withdraws the parent with delete, the dependent rows written onto it are withdrawn with it, so a parent created again later receives its consent again.

A dependent rule delivers no backfill on its first activation: a consent already in place for an existing contact reaches the target once it changes. See Outbound Delivery State for the ledger rows these steps leave, and Sync Rules for what makes a dependent rule valid.

What this page does not cover¶

• No live target. Not one system declares push_out, so a Conflux running the finished pipeline still delivers nothing. Every stage is built and will keep being built against the test doubles in Outbound Writer Contract — there is no concrete writer yet. The first live rule is therefore also the first real test of the whole chain.
• No feedback channel from the target. Nothing reads a record back, so an edit made inside the target never reaches Conflux and the two copies are never compared. Detecting that drift would be a per-rule policy in its own right, and it would have to say which copy wins when the two disagree. Nobody has decided that.
• No deduplication across sources. Two of a tenant's shops selling to the same person produce two records in the target. The same holds for orders and products. Merging duplicates would mean deciding, field by field, which source's value is the true one, and ADR-0003 refused to make that call.
• No field-level provenance. Nothing tracks which system last wrote a given field, and no rule decides per field which system wins. That combination is what a golden record is, and ADR-0003 declined the model. Granularity stops at the rule on purpose. An active rule gives the source authority over everything it carries, and the only other setting is off.
• No shop write-back. Writing an order status back into WooCommerce is a plausible use of this layer and not part of it. WooCommerce is a source today, and the direction matrix is what would have to change first.