Outbound Delivery State¶

The cursor¶

There is one cursor per SyncRule. It holds the delta position — (content_changed_at, id) up to which delivery under that rule has been attempted — and, while a backfill runs, the backfill's bound backfill_until (T0) and its own position over the dataset that already existed at the rule's first activation.

A delta scan does not resume at the stored position itself. It subtracts a configurable overlap, starts there, and treats that lower bound as inclusive. The overlap exists for the record whose content_changed_at landed behind the position while the previous pass was already running. Reading strictly after the position would step over it once and never come back to it. The price is a record that can be read a second time, and the ledger join excludes it before it is returned. A backfill scan takes a different lower bound, with no overlap; see below for why.

The cursor lives in its own table, delivery_cursors, with one row per rule and a unique key on sync_rule_id. It holds position_changed_at and position_id for delta, and backfill_until, backfill_changed_at and backfill_id for the backfill. There is no mode column.

Columns on sync_rules were rejected for three reasons. SyncRuleObserver::saving() validates the direction matrix whenever an active rule is saved, so a cursor advance would fire a configuration check and could fail on it. sync_rules is configuration read by Filament, the policies and DeliveryRoutes, while a cursor is high-frequency runtime state. And a disconnect soft-deletes a connection's rules, which would carry the cursor off with them and make the reconnect promise below depend on RestoreConnectionDataJob's timestamp filter.

The name carries the direction split. The cursor has an exact pull-out equivalent, so it takes no outbound_ prefix; the ledger has none, so it does.

SyncRuleObserver::created() and updated() open the cursor at a push-out rule's first activation: created active, or switched on and holding none yet. Opening it fixes the moment of activation as T0, starts delta from T0, and opens a backfill bounded by T0 over everything the rule already carries. A reactivation, a restore after a disconnect, or any other save of an active rule finds the cursor already there and changes nothing. Each of the two hooks fires only for the save that created the rule or switched it on, so a later save of the same instance does not look the target up again. Where a delivery step meets a rule with no cursor at all, DeliveryCursor::forRule() falls back to opening the same backfill cursor there and then. For a rule with no ledger row at all, such as an existing rule whose target only now declares push_out, that is its first activation for delivery purposes and the rollout of CFX-200's promise for existing connections, so it logs at info. Where ledger rows exist, a cursor went missing and it logs a warning, since the listener should already have opened one. See ADR-0037 for why.

A rule of a dependent domain, such as customer_consents, opens its cursor in delta only, with backfill_until left null, on its first activation and in the forRule() fallback alike. A parent that already existed in the target when the rule was activated receives its dependents once they change. A consent backfill into Positive User would send one double opt-in mail per unconfirmed contact, so a dependent rule cannot be backfilled by hand either (ADR-0046). See ADR-0042.

Only the time half of the position filters a delta scan. Its lower bound is position_changed_at minus the configurable overlap, inclusive, on content_changed_at alone; position_id records the exact stop point and orders records that share a timestamp, but does not itself narrow the query. A bulk write producing many records at one timestamp is therefore re-read on each pass and suppressed by the ledger join, which converges but reads more than it needs to. A backfill scan has no ledger join to converge with, so it does filter on position_id, for the reason below.

The two modes take different lower bounds. A delta scan reads back over the configured overlap, inclusively, and relies on the ledger join to make a re-read cost one indexed read and no write. A backfill scan reads content_changed_at < backfill_until, strictly after the exact stored position, (content_changed_at, id) as a tuple, and applies no overlap. It has no ledger filter to break a tie with, so an overlap would stop it terminating: a batch spanning less than the overlap would put the next lower bound at or below the current one and return the same rows for ever.

What backfill trades for termination is the overlap's protection. A record written while a backfill is running, whose commit becomes visible only after the pass has read past its timestamp, is missed by that pass. A backfill walks a dataset that already exists, so this concerns concurrent writes only. They share one row and hold two positions, and a record written while a backfill runs carries a change signal at or after T0, so delta picks it up.

Aborting and starting a backfill¶

A backfill ends in two ways: it reads past its bound (finishBackfill()), or somebody aborts it (abortBackfill()). An abort needs a backfill in progress and ends the initial one as well. A start (openBackfill()) needs none in progress, so a new backfill follows one that ended and never replaces one that runs. Both are done by App\Services\Backfills from the rule page's header actions, which are described under Sync Rules resource.

Every write to the three backfill columns is a compare-and-set on backfill_until. advanceBackfillTo(), finishBackfill() and abortBackfill() update the row only WHERE backfill_until still equals the value the model was read with, and openBackfill() only WHERE backfill_until IS NULL. A delivery step reads the cursor once, delivers a batch and writes back. Where an abort or a start landed in between, the condition no longer holds. The write changes nothing and logs "Backfill write dropped", with the cursor, the rule, the operation (advance, finish or abort) and the bound the step had read. The next step reads the cursor afresh: it finds no backfill and runs delta, or it finds the new backfill and starts at its beginning.

MySQL reports an update that changes nothing as zero rows. The method therefore checks that the row still matches the bound before it calls a write dropped.

The delta side needs no such condition. advanceTo() and touch() save only the attributes they changed, so they never write the backfill columns and cannot undo an abort.

A stale step cannot hit a new backfill as long as the new backfill carries a different bound. The old bound is the old backfill's own start, which lies before the abort and so before the new start. Bounds are stored to the second, though. A start, an abort and a new start within one second would reuse the bound, and the dialog in between rules that out in practice.

The two writes an operator makes, openBackfill() and abortBackfill(), leave updated_at alone. The delivery chain reads that column as its heartbeat (see The heartbeat), and a start or an abort says nothing about whether a chain is running. The writes of a delivery step, advanceBackfillTo() and finishBackfill(), do set it.

The batch running when an abort lands is delivered to the end, and only the batches after it are prevented. A chain that meets a cursor with no backfill in progress touches it and carries on in delta. The records the backfill had not reached are picked up by delta once they change. A restarted backfill starts from the beginning, and writes a record it already delivered and that has not changed since as suppressed instead of sending it again.

The service logs an abort at info level, "Backfill aborted by hand.", with the rule, the bound that ended and the user. That line is the only record of an abort.

Backfill confirmations¶

Each backfill started by hand leaves one row in backfill_confirmations, written in the same transaction that opens the backfill on the cursor. It records that somebody confirmed the target's reaction to a create had been checked, and what the backfill was going to cover. It is never updated. The backfill the first activation opens needs no confirmation and leaves none. See ADR-0046 for why a start needs one.

There is no deleted_at: the row records a decision, and it goes only with its rule. SyncRule::deleteReferencesOutsideDeliveryState() deletes a rule's confirmations together with its runs, stops and policy acceptances. Two indexes serve the two reads: (system_connection_id, created_at) for the latest confirmation on a target connection, and (sync_rule_id, backfill_until) for the confirmation of the backfill that is running.

Both numbers are counted by DeliveryScan::backfillPreview() for a rule of a root domain, over the rule's domain and source connection, against one bound. The record count reads the same window as the backfill itself, so on a propagating rule it includes every tombstone of a deletion the source reported. The backfill withdraws those the target holds and skips the rest. A create is counted among the live records only, where the runner would send one. That is a record with no ledger row, or with a live row that holds no remote_id, is in no terminal status, and records no delivered version or an older one than the record's. The missing reference decides, not the missing row: a record whose earlier create failed has a failed row and no reference, and it is created again. Where that row still waits for its retry, the drain creates it rather than the backfill. A record whose row is soft-deleted, remote_gone or exhausted is no create, because delivery skips such a row. An undeliverable record counts as a create. A source row corrected since the refusal would be created by the backfill, because neither delta nor the drain offers the record again. The database cannot tell whether it was corrected, so the count errs high. A record whose row already records its version is suppressed, not created. The terminal statuses are read from DeliveryStatus::isTerminal().

The start dialog counts once, when it opens, with the present second as the bound, and keeps the bound and both numbers while it is open. Backfills::start() receives that bound and the confirmed number. It refuses a rule Backfills::canStart() rejects and a bound in the future, counts once more against the bound, and refuses the start only where the backfill would now send more creates than confirmed. In normal operation the count shrinks, as delta delivers records from inside the window and changed records move past the bound, so a smaller recount starts and the confirmation keeps the confirmed number. The count can grow too. A change signal stamped just before the bound can become visible after the dialog counted. A data restore brings records back live without moving their change signal. A corrected undeliverable record counts as a create, and so can a record whose ledger row somebody edited. That is why a greater recount is refused. ADR-0046 says when values the browser holds are safe to take, and names the one forged bound that is not.

The rule page reads a confirmation by rule and bound to say who confirmed the running backfill. The target connection's page shows its latest one.

The ledger: outbound_deliveries¶

Each row covers one source record under one rule. It answers two things about that record: what the last delivery attempt did with it, and under which identifier the target holds it now.

A unique index on (sync_rule_id, data_domain, source_id) gives each source record exactly one row per rule. That triple is also the lookup performed before every delivery, because a row holding a remote_id means an update and no row means a create. The key is built from source identity rather than from a local row id. A rebuild after a disconnect or an uninstall issues fresh local ids for the same source ids, so a key on the local id would point at nothing afterwards. See Outbound Delivery for what that choice buys and ADR-0014 for the decision itself.

Status values¶

exhausted and remote_gone are the two terminal statuses, and DeliveryStatus::isTerminal() is where that is stated. A record in either is passed over on every later chain however often its source row changes, with one exception: under a dependent rule a remote_gone row is offered again on its record's next change. The other seven are not terminal: an awaiting_anchor row is offered again on its record's next change or once its parent's create wakes it, and a delivered, suppressed or withdrawn row is offered again as soon as the record's change signal moves past delivered_content_changed_at, a failed row once next_attempt_at has come due, an undeliverable row on every later pass, and a held row once its rule's hold ends: the drain hands it back as a withdrawal if the record is still a tombstone, or as an ordinary delivery if the record has come back live. For a withdrawn row that means the record was restored in the mirror. A row that lost its remote_id (a delete) is delivered as a create. A row that kept it (a deactivate) is reactivated first, whatever the rule's update_mode, and then updated or suppressed as that mode says. A target that no longer holds the record at reactivation ends it as remote_gone.

Some of the values are about the target and some are about the record, and that split is what keeps them from collapsing into one another:

• undeliverable reports, it does not gate. The refusal is decided locally, before the target is contacted, so re-evaluating it on a later pass costs a projection and no request at all — which is why the scan needs no filter for it and the runner needs no guard. What the status buys is that an operator can list what is stuck and read why, instead of the record disappearing quietly into a scan that offers it forever.
• remote_gone looks similar and behaves the opposite way. There the target has spoken, re-offering costs a request, and delivering again would undo somebody's deletion — so the runner does have to check it and skip.
• failed carries a retry schedule because trying again is the thing that might work. undeliverable carries none, for the same reason in reverse.
• withdrawn is the only one of the five that records a write the target accepted. The others say the record did not go out, or went out unchanged. This one says a removal reached the target and did what the rule asked.

awaiting_anchor is the status a record gets when it cannot go out until a record in another domain has gone out first. Only a dependent domain has such a dependency: a customer, order or product is deliverable on its own, while a consent needs its customer in the target. See ADR-0042 for how a dependent record finds its target record.

Indexes¶

Two reads justify an index, and each gets one.

What it does not hold¶

No payload, and no hash of one. A hash is there to answer which fields have moved since last time. Nothing in delivery asks that question, because an update carries the full canonical payload every time. The ledger records outcomes and identifiers, and nothing about content. The full-payload rule is the overwrite default of update_mode. ADR-0016 has the reasoning.

The chain¶

Delivery under one rule is a chain of queued jobs rather than one long run. App\Services\DeliveryRunner holds the two steps a job can take: scanBatch() walks one batch forward from the cursor and says whether the cursor moved, drainChunk() retries one chunk of the failed rows whose next attempt has come due and says where the next chunk would start. Neither step decides a status itself. Each hands its candidates to App\Services\RecordDelivery, which decides create against update from the candidate's ledger row, calls the writer, and translates the writer's answer into the row. Everything around those two is thin: a scheduled command starts a chain per route, and each job runs one step and picks the job that follows it.

Dispatch and schedule¶

queue:prune-batches runs daily alongside it, deleting finished and cancelled batches older than 48 hours so job_batches does not grow by one row per chain forever.

The chain of jobs¶

One chain is one bus batch, named delivery:{sync_rule_id}, and one row in job_batches. It starts with a single DeliverDrainChunkJob and never holds more than one job at a time: each job chooses the job that continues the chain, and the batch closes as finished once a job chooses none. Every job logs its step's report with batch_id and next, the job that follows (drain, scan, backfill, or null when the chain ends there), so a chain can be read off the log filtered by sync_rule_id.

The drain jobs come first. Each one retries the rule's failed rows whose next_attempt_at has come due, and, for a dependent rule, the awaiting_anchor rows a parent's create woke by setting their next_attempt_at, one chunk in id order, whatever their position relative to the cursor. The drain exists because a record that failed and has since fallen behind the scan window would otherwise never be attempted again. It is a set of rows rather than a walk forward, so the cursor stays where the scan left it, and its position lives in the job's payload as the id of the previous chunk's last row rather than in the database. A chunk that comes back empty hands over to the first scan batch. The drain runs first because its set is finite and shrinks within a chain, every row attempted either lands or gets a next attempt in the future, while the set the scan walks can grow faster than the rule delivers; a chain that scanned first would never reach its drain for such a rule, and its retries would wait for ever.

The scan jobs come last. Each one delivers one batch forward from the cursor and advances the cursor — forward only, past the last candidate whose delivery concluded. A scan job is followed by another scan job while the batch it read was full: a batch of scan_batch_size records may have left more in the window, and the ledger join excludes what was just delivered, so the next batch reads on. A batch that came back short, empty included, has read everything the window held. Whether the cursor moved is logged but does not decide: a full batch of late arrivals behind the position moves it nowhere and still leaves the window unread past it.

A backfill batch runs in between, never ahead. Once a delta scan batch comes back short and DeliveryCursor::backfillPendingFor() finds the rule's backfill still in progress, DeliverScanBatchJob hands over to one DeliverBackfillBatchJob rather than closing the chain. That job runs DeliveryRunner::scanBatch() in backfill mode, delivering the next batch of what the rule already carried at its first activation, and hands straight back to a DeliverScanBatchJob regardless of whether its own batch was full or short. Delta is checked again before the chain ends, so a change that arrived while the backfill batch ran is never left waiting behind it. A short or empty backfill batch, one that has read past backfill_until, ends the backfill (DeliveryCursor::finishBackfill()) instead of continuing it. The chain is through once a delta batch comes back short and the rule's backfill is no longer in progress.

Nothing is attempted twice in one chain. A failed row with a scheduled retry is the drain's alone: the scan's ledger join leaves it out wherever it lies, due or not, so a scan job never touches a record the drain owns.

Every job covers exactly one batch or one chunk and carries OUTBOUND_DELIVERY_JOB_TIMEOUT_SECONDS (300 by default) as its $timeout, so no job outlives a single step and a deploy or a worker timeout meets an ordinary job. Size it to comfortably cover a whole batch of OUTBOUND_DELIVERY_SCAN_BATCH_SIZE writes at the target's latency: a batch that hits the timeout is not resumed, it is re-run from the same cursor position on the next tick, and a target too slow for the batch would then time out on every attempt. The timeout is enforced by queue:work, which registers the alarm around each job, and only by it: under queue:listen --timeout=0, which the repository's dev supervisor config uses, no job ever times out, so the lock arithmetic below assumes the daemon. Both classes are #[DeleteWhenMissingModels]: deleted rather than run when the rule row has gone by the time they are picked up, which a purge between dispatch and execution can cause. Both begin by checking $this->batch()?->cancelled() and return without touching anything when it is. Both jobs are pushed to the queue named by OUTBOUND_DELIVERY_QUEUE, delivery by default and never the default queue. One scan batch is up to OUTBOUND_DELIVERY_SCAN_BATCH_SIZE sequential remote writes, so on the shared queue two slow targets would occupy the worker pool for minutes while webhooks and bulk syncs waited; a queue of their own decouples delivery latency from the rest of the platform. A worker has to consume it, because a job on a queue nobody listens to never runs: the repository's dev supervisor runs one (laravel-delivery-worker), and the infrastructure needs one before this is deployed.

No job is retried, and that holds for a timeout as much as for a thrown exception. $tries is 0 and $maxExceptions is 1, which reads as one rule: being released by the middleware is not an attempt, and the first exception to leave handle() fails the job for good. A job killed by its own timeout meets the same rule rather than a gentler one: the queue worker counts a timeout against $maxExceptions exactly as it counts a thrown exception, so the first timeout fails the job too, instead of releasing it back onto the queue to run again. A failed job cancels its batch — allowFailures() is not set — so either cause ends the chain, which is what ADR-0029 asks for. A permanently broken rule therefore leaves one cancelled batch and one failed_jobs row per tick, which is visible and intended.

Cancelling the batch is also the operator's stop button. Bus::findBatch($id)->cancel() ends a chain at the next job boundary, the cursor stands where the last batch put it, and the next tick starts a fresh chain if the route is still open.

One job per rule¶

Both job classes run through App\Jobs\Middleware\OneJobPerDeliveryRule, which takes a cache lock keyed delivery:{sync_rule_id} for the length of the job and releases it in a finally. A job that does not get the lock is released for OUTBOUND_DELIVERY_RELEASE_AFTER_SECONDS (5 by default) and offered again, without that counting as an attempt. The lock's TTL is the job timeout plus one minute: a job cannot outrun its timeout where the daemon enforces it, so a lock still standing after that belongs to a job killed hard, and it is gone long before the staleness window lets the dispatcher restart the chain, which is what keeps the restarted job from being released against it.

The middleware enqueues the successor, and it does so only after the lock has been released. A job's handle() merely remembers which job should follow it, through App\Jobs\Concerns\ContinuesDeliveryChain. That ordering is the reason for a middleware of Conflux's own rather than Laravel's WithoutOverlapping: a successor enqueued from inside handle() exists while its predecessor still holds the lock, so a free worker can pick it up, find the rule locked, and delay or drop it. Enqueued after the release, a successor can never meet its own predecessor. It is also what lets a whole chain run nested on the sync queue, each successor firing synchronously within its predecessor's own call stack rather than waiting on a separate worker.

Collisions can therefore only come from a tick, and a tick starts a chain only once it has judged the previous one dead and cancelled its batch. ShouldBeUnique would not do: it deduplicates at dispatch and holds its lock until the job ends, so a successor enqueued at the end of a job would be dropped as a duplicate.

The heartbeat¶

A chain is alive when a batch named delivery:{sync_rule_id} is open — neither finished nor cancelled — and its heartbeat is younger than OUTBOUND_DELIVERY_STALE_AFTER_SECONDS (900 by default). The heartbeat is the younger of two timestamps: the rule's DeliveryCursor::updated_at, which every step past the route check writes (through advanceTo() where the batch moved the cursor, through touch() where it did not), and the batch's own created_at. Starting or aborting a backfill by hand does not write it. The second one is what covers a chain whose first job has not run yet: that job is what creates the cursor row, so a chain waiting in a queue backlog would otherwise read as dead on the very next tick. Neither timestamp is a new column.

A batch has no lifetime of its own — a job lost in transit leaves its batch open for ever — so an open batch with a stale heartbeat is a chain that died mid-way. The tick cancels it through Bus::findBatch() before starting the next one, so job_batches collects no ghosts and a job of the dead chain that does turn up finds its batch cancelled and ends. The batch half of the condition is equally what keeps a chain that has just finished from blocking the next tick: a finished batch is not alive, however fresh the cursor.

OUTBOUND_DELIVERY_STALE_AFTER_SECONDS has to exceed the dispatch interval plus the job timeout. Below that, a chain waiting in a backlog, or one slow batch, reads as dead and is cancelled and restarted on every tick. The default of 900 clears the default cron interval plus the default timeout, 300 s plus 300 s, with a further 300 s of margin.

The retry schedule¶

App\Services\RetrySchedule turns the number of failed attempts a record has accumulated into the time of its next one, or into the answer that there is none.

With the defaults a failing record waits one minute, five minutes, thirty minutes and two hours, and is exhausted after the fifth failure. attempts counts failures since the last success: a success resets it to zero, and a new version of the source record does not.

How a step ends¶

Whether the chain goes on is not a result. A step that ends route_closed or breaker_open adds no successor and the batch closes as finished, and after a completed step the question is answered by ScanBatchReport::$full for a scan and DrainChunkReport::$lastLedgerRowId for a drain.

All three are ordinary return values on App\Enums\DeliveryResult: the job succeeds and logs its report. A step can also end by exception — no writer registered for the target's driver, a writer lacking the verb the ledger decision needs, a unique key violation on a ledger insert, or a mirror record carrying no position or source_id — and then there is no report: the failed job and the cancelled batch are the record. ADR-0029 sets out which faults end a chain, which ones park a single record, and why the two are kept apart.

What a step skips¶

DeliveryCandidates::dueForDelivery() holds every rule by which a candidate a step read is passed over. A skipped candidate has nothing written for it, neither a new row nor a change to an existing one, and the cursor still moves past it.

In delta the scan's query already leaves every one of these out, and it has to. The overlap offers a record again in every batch until the cursor has moved a whole window past it, and a record that is read and dropped moves the cursor nowhere. One batch's worth of them at the front of the window, and every batch would be made of them alone: the chain would end without progress, the next chain would repeat it, and the rule would stand still behind them for good. So the delta join keeps only a record with no ledger row, one whose change signal is newer than what a live delivered or suppressed row records, and one whose live failed row has no scheduled retry (the shape the drain leaves behind, see below). For a dependent rule it also keeps a live awaiting_anchor or remote_gone row whose record changed after the version the row records, and an awaiting_anchor row a parent's create woke, whose version is cleared. The filter then drops something in delta only when a row changed between the query and the read. In backfill, which builds no ledger join and re-reads nothing, and in the drain, the filter is what decides.

• The record itself is soft-deleted and is not a deletion this rule delivers: its delete_policy is ignore, its deletion_reason is not source_event, its ledger row holds no remote_id, or that row is already withdrawn. In delta the scan's query leaves such a tombstone out entirely; in backfill and the drain it is read and DeliveryCandidates::isDeliverableTombstone() drops it. A tombstone that does qualify is delivered as a withdrawal rather than skipped. See Outbound Delivery for how.
• Its ledger row is soft-deleted. A trashed row still occupies the unique key and behaves as a row that is in the way, so it may be neither delivered under nor overwritten. Nothing soft-deletes a ledger row today; this is a guard.
• Its ledger row is in a terminal status, remote_gone or exhausted. A dependent rule's remote_gone row is the exception: it is due again once its record has changed.
• Its ledger row is failed and next_attempt_at is still in the future. Such a record stays inside the scan window for the length of the overlap and is reachable through the drain afterwards.

A row that is delivered is not skipped here. Whether the record is newer than what was delivered is decided against the row's own timestamps, and a record that is not newer is written as suppressed.

A drain chunk goes one step further for the first of these. Its query already pins the status and the due time, so a row it selected and then dropped has lost its record, soft-deleted or gone. The chunk clears that row's next_attempt_at, and the row stays failed: no later drain selects it, and should the record come back, its change signal moves, the scan's ledger join offers it again, and a failed row with no next attempt counts as due.

What skipped counts in the report¶

DeliveryReport is per job, not per chain: every job logs the report of its own step, and what the chain did as a whole is the job_batches row plus the log filtered by sync_rule_id. Within one job, DeliveryReport::skipped counts the candidates the step read and dropped.

No step keeps a memory across jobs, and none needs one. A delta scan job re-reads nothing it has dropped, because the query leaves out what is not deliverable, and nothing it has delivered, because the join excludes a delivered record from the very next batch; the only re-reads are records whose change signal landed behind the position, and those are delivered. A drain chunk reads the rows after the previous chunk's last id, and a row it drops is parked with its next_attempt_at cleared, so it is counted on the one occasion there is to count it. In backfill the scan continues strictly after the exact position and counts a dropped candidate once as it passes it.

Throttling¶

DeliveryThrottle keeps one request budget per target connection, taken from the writer's requestsPerMinute() over a 60-second window. Every writer call spends one request of it; a suppressed or held record asks the writer nothing and spends nothing. The budget is shared by every rule that delivers into the same connection, because the rate belongs to the workspace the connection points at, not to any one rule.

A rate-limit response blocks the whole connection rather than failing the one record it answered: the wait it named, capped at one day (DeliveryThrottle::MAX_BLOCK_SECONDS), or 60 seconds where it named none, and a block already running longer is never shortened. See ADR-0036 for why.

A live block already in force when a step starts ends it as DeliveryResult::Throttled before the cursor is read, so nothing in the batch is attempted and nothing in the ledger changes. A budget spent partway through a batch, or a 429 the target answers partway through one, is only discovered mid-batch, once RecordDelivery calls DeliveryThrottle::take() or the writer itself answers: by then, earlier candidates in the same batch may already be delivered and recorded. That step also ends as Throttled, and nothing is booked for the record that met the throttle. Its cursor stays on the last record whose delivery concluded, and a throttled drain repeats from the id it started after.

App\Jobs\Middleware\WaitForDeliveryBudget runs on every job of the chain, before the rule's lock is taken. It asks how long the target connection's next request must wait and, where that is more than zero and no more than throttle_max_wait_seconds, releases the job for that long and takes neither a worker nor the lock meanwhile. A longer wait, such as a block another rule's 429 set while the job waited out the budget, passes the job on: the step's block check ends it as Throttled, and the job closes the run as throttled. A job of a cancelled batch passes straight through. The dispatcher (delivery:dispatch) checks the same block before starting a new chain and skips a connection that is still blocked, counting it as throttled in its summary line. Since every rule on that connection is skipped, only the rule that met the 429 has a throttled run, and the rule state reads the others as throttled from the block itself (see Rule state).

Runs¶

Each delivery chain writes one row of its own, in delivery_runs, one per chain rather than one per step. Where a chain's own log lines answer "what happened to this batch", a run answers "how did this chain end and what did it do overall", without anybody reading the log and adding it up.

A run is running from the moment DeliveryRuns::open() creates it until something ends it, and every run passes through running exactly once. What ends it, and who writes the ending:

First writer wins. DeliveryRuns::close() updates the row only where its status is still running, so whichever of the chain's last job, its failure handler, or the tick reaches it first is the ending that sticks; a failure handler that fires after the tick has already abandoned the run cannot turn it back into faulted, and the reverse holds too.

Opening a run does not touch a release. It only creates the row. A volume breaker release belongs to the rule, not to a run: see Stops for how long it stays in force.

Runs are pruned, not soft-deleted. DeliveryRun uses MassPrunable; its prunable() query removes every row, closed or still running, whose started_at is older than run_retention_days (config/conflux.php, default 30), and model:prune runs it daily. A run this old is treated as history regardless of how it ended, in line with the project's exception for a log-shaped table: soft deletes would hide the rows without ever freeing the space.

Stops¶

A DeliveryStop (table delivery_stops) is either a manual stop, the kill switch, or the volume breaker's hold on one rule's withdrawals. It is live while not soft-deleted; ending one soft-deletes it and records how and by whom, so deleted_at is the moment it ended and every stop that ever existed stays readable as history.

Scope (DeliveryStopScope) says how far a stop reaches: global (nothing else set), connection (system_connection_id set, reaching every rule that writes into it), or rule (sync_rule_id set, reaching that rule alone). Only the columns matching the scope are set; the rest stay null.

Cause (DeliveryStopCause) says why: manual, the kill switch, halts every write in its scope; volume_breaker is always scoped to rule and holds back only that rule's withdrawals, never its creates or updates.

Resolution (DeliveryStopResolution), set once a stop ends: resumed (a manual stop ended, or a hold discarded without releasing anything), released (a volume breaker hold ended with an allowance attached; the only cause a release can apply to), self_healed (the system ended a hold because nothing was left to hold, with no person involved).

At most one live stop exists per target and cause. MySQL has no partial unique index to say so, so setting a stop takes a cache lock keyed by scope, target and cause, and returns the stop that is already live rather than creating a second one alongside it.

Reading which stop applies:

A hold carries the finding it tripped on: pending_withdrawals, known_records and threshold, the same numbers the volume breaker computed at the moment it tripped. A release carries allowed_withdrawals, the number support confirmed, and allowance_consumed_at, set once the release is no longer in force; a hold that is discarded rather than released carries neither.

A release stays in force until the rows it confirmed are gone. At most one release per rule is in force: the released volume breaker stop whose allowance_consumed_at is still empty. Recording a release stamps allowance_consumed_at on every older unconsumed release of the rule, in the same transaction and under a row lock, so a newer release replaces an older one and the older can never apply later. While a release is in force and no hold is live, every step of every run adds its allowance to the limit. The first step that finds no held row with a tombstone left stamps the release consumed, again under a row lock, because every row it confirmed has been withdrawn or came back live; stamping twice changes nothing. While a release is in force, each run may withdraw up to the normal limit plus the confirmed number, until the held rows are gone. A new hold ends the release: holdWithdrawals() stamps it consumed when it creates the hold (not when it finds one already live), under the same cache lock, because the new finding already counts the confirmed rows still held, and discarding that hold must leave no allowance behind. Tying a release to one run instead would lose it on a chain already past its drain, the only step that hands held rows back, or on a run that ends early.

The volume breaker's limit. Before every step, on a rule whose delete_policy propagates, the breaker counts what withdrawing now would mean: the scan window's pending withdrawals, plus held rows whose record is still a tombstone, plus what the current run has withdrawn so far. A held row whose record came back live is left out, because it goes out as an update once the hold ends. The limit is the greater of the configured floor and the configured share of what the ledger already knows the target holds (rounded up), plus the allowance of the release in force. The count exceeding that limit is what trips a hold. Once a release is spent, the run that withdrew its rows keeps credit for as many of its own withdrawals as the release covered (the smaller of allowed_withdrawals on its row and its withdrawn), because those withdrawals stay in the count; a new run starts from zero and gets none. A count within the floor plus the allowance passes without counting what the target holds, since that count reads the rule's whole ledger.

Setting or ending a stop, and creating a new hold, forgets every user's cached badge count on the Delivery page, so those changes show up without waiting out the badge's 60-second cache. The count is cached per user, because each user counts only the rules they may access; DeliveryHealth::UNHEALTHY_COUNT_CACHE_KEY holds a generation number that every per-user key includes, and DeliveryHealth::forgetUnhealthyCounts() moves it on rather than finding each user's key. Finding an already-live hold still there costs nothing extra: only the run that actually creates one forgets the count.

Rule state¶

DeliveryHealth::stateOf() derives a DeliveryRuleState for a rule when it is read, from its stops and its runs; nothing stores it. Deriving it rather than writing it is what makes a dead scheduler visible: nothing marks a rule stale by hand, so when the tick stops running, every rule under it reads as stale on its own instead of freezing in whatever state a job last wrote.

Six values, declared from most to least severe. The most severe one that holds is the state:

A rule with no open route, no run ever, and no hold has no state at all, null: delivery has nothing to say about it, which reads differently from ok in the admin panel. That holds under a manual stop too, because a global or connection stop reaches pull-in-only rules that never deliver; a stopped rule whose route has closed but that has run before still reads stopped.

Staleness is timed from the latest completed run's ended_at, or from the rule's own updated_at where it has never completed one. A run that ended any other way, route_closed, breaker_open, stopped, faulted or abandoned, does not reset that clock, so a rule stuck failing its route check for long enough still reads stale rather than hiding behind its last attempt. The one exception is a chain that is still writing. A backfill keeps its one run running for hours, and every step of the chain writes the cursor, so a running latest run whose cursor was written within the window is alive, not stale. A dead chain's run is abandoned by the dispatcher and stops being running, so it still goes stale.

The Delivery page's groups are this derivation run over every open push-out route the viewing user may access (SyncRule::accessibleBy()), kept where the result is not ok; the navigation badge is that same count, cached per user for 60 seconds. The page's stops follow the same scope: the global stop for everyone, a rule's stop where the user may access the rule, a connection's stop where they may access the connection.

Rule status in the admin panel¶

The admin panel shows one status per rule, built around this state: a delivery state other than ok comes first, and only then does the panel say what the configuration and the connections add (Config problem, Inactive, Pull out, Route closed, No run yet). See Sync Rules resource for the full order.

The Delivery page¶

Support opens this page (/admin/delivery) to see what is not delivering normally right now, without hunting through individual rules. Four tiles sit above the page, each linking to its group and each following the search below:

Below the tiles, every stop and every unhealthy rule the user may access appears exactly once, placed by which applies to it, in one of three groups:

1. Stopped by people. One card per manual stop, rule- or connection-scoped. A connection stop's card lists the rules it affects (only the ones the user may access), and names a volume breaker hold still live on one of them. A rule under both a rule stop and its target connection's stop is listed under both stop cards, since both have to end before it delivers again.
2. Held by the volume breaker. One card per hold whose rule is not already on a stop card; a hold whose rule sits under a stop, its own rule stop or a connection stop on its target, is named on that stop's card instead, not repeated here.
3. Not delivering, no stop. One card per remaining unhealthy rule (Failing, Stale, Throttled), with its state and DeliveryHealth::describe()'s sentence. A rule stopped only by the global stop is not a card here or in group 1: the "All delivery is stopped" card above the groups counts it, under "Also stops N push-out rules not listed below" (shown only when there is at least one).

Backfills in progress. Below the three groups, a fourth section lists every active rule the user may access whose backfill is still running, under the same search and grouped by tenant the same way. Each card shows how far the backfill has come, as records passed out of records before its bound (1,240 / 5,000 (24 %)), a progress bar, the moment it started (its T0, in UTC), and the rule's state where the worklist found one. Where the rule's push-out route is closed, for example because its source or target connection is inactive, the card also says "Paused: the route is closed.", asked once for all listed rules through DeliveryRoutes::openAmong(). Nothing here is a problem to act on: a backfill delivers the existing dataset through the same throttle, kill switch and breaker as delta, and the section only says how far it has come. The counts cost two queries over the mirror per rule and many backfills start at once when a system first declares push_out, so each rule's counts are cached for 60 seconds per cursor and backfill bound, so a backfill that was aborted and started again is counted afresh; the rule page reads them live.

While a global stop is in force, the page never says every rule is delivering normally. A search that matches nothing says "Nothing matches the search." instead. Neither message appears beside a listed backfill.

Cards are grouped by tenant, tenant names ordered alphabetically. Within a tenant, stop and hold cards are ordered oldest first; group 3's cards are ordered by status urgency, then by rule title, since an unhealthy rule with no stop has no start time for its problem. When the "Global delivery" tile reads "Scheduler down" or "Runs not completing", group 3's stale cards collapse into one line naming the count and the cause; Failing and Throttled cards stay as their own cards.

Every card shows a tenant name and its short id above it, the rule or connection title, and a collapsed "IDs" row with the full tenant, rule, source and target UUIDs. A stop or hold card shows both the source and the target host; a group 3 rule card shows only the source host. Clicking any id, short or full, copies it and a "Copied" tooltip confirms, the way Filament's own copyable entries do.

Search. The field above the groups (?q=, so a link from an alert opens the page already filtered) takes either a name, matched case-insensitively as a substring of a tenant or connection name, or a string of at least 8 hex characters or dashes, matched case-insensitively as a substring of any id involved (tenant, rule, source connection, target connection): a full UUID, or any 8-or-more-character part of one, including the short id shown on each card. The short id is the last 8 characters of a UUID, not the first: the ids are UUIDv7, whose leading characters are a timestamp, and a tenant created through the widget shares that timestamp with the connections and rules created alongside it, so the first 8 characters would not tell them apart.

The dispatcher heartbeat¶

delivery:dispatch records the moment it starts, at the start of every run, in a heartbeats database row (App\Services\Heartbeats, keyed delivery:dispatch) rather than in the cache, since the local stack's array cache store does not outlive the process.

The "Global delivery" tile (App\Services\GlobalDelivery::status()) resolves one of four states, first match wins:

Rows 2 and 3 are about the whole installation, not one tenant, so every user sees them, the same as the global stop.

conflux.outbound_delivery.dispatch_heartbeat_stale_after_seconds (env OUTBOUND_DELIVERY_DISPATCH_HEARTBEAT_STALE_AFTER_SECONDS, default 600) is this state's own staleness window. Do not confuse it with OUTBOUND_DELIVERY_STALE_AFTER_SECONDS above, whose similar name covers a different question: whether one rule's delivery chain is still alive, not whether the dispatch command itself has run recently.

Guard settings¶

Lifecycle¶

A ledger row outlives the record it describes. Soft-deleting a customer, an order or a product leaves its delivery rows alone, in line with the project's rule against cascade deletes.

Delivering a tombstone follows the rule's own policy. The scan reads a soft-deleted record only where delete_policy propagates and the recorded cause is source_event, and a delivery of one is recorded as withdrawn rather than as an update. See Outbound Delivery for how.

A disconnect leaves the ledger standing. That is what makes a reconnect within the grace period cheap (30 days by default, configurable via config/conflux.php). Every source record still has its row and its remote_id, and the first scan after the reconnect finds nothing newer than what those rows already record, so it sends nothing.

An uninstall purges the table. A row points at a record inside somebody else's system, and that record stands for a person. The row therefore belongs to the personal data an uninstall exists to remove. See ADR-0013 for both states and what each of them takes away.

Purge ordering¶

SystemConnection::pruning() hard-deletes a connection's SyncRule rows. Both delivery_cursors and outbound_deliveries reference sync_rules with restrictOnDelete, so their rows have to go first. Both purge paths reach that code: the scheduled model:prune for an expired grace period, and PurgeConnectionJob for an uninstall. The scheduled path fails quietly if the order is wrong, because pruneAll() catches per record and reports, so the connection would be skipped and retried on every run while nothing is ever deleted.

The removal happens inside the same chunk that already collects the rule ids, before the rules themselves are force-deleted. No model event was added to SyncRule to release the constraint on its own, so a direct forceDelete() on a rule that still holds delivery state still throws. The purge is therefore the single path authorised to release it.

A single rule is refused, not cascaded¶

Force-deleting one rule while its connection stays alive would drop the ledger while the blueprint service recreates an equivalent rule on the next activation, and that rule's first delivery of each record would create a second copy in the target. The Filament force-delete action is therefore disabled for a rule that holds delivery state, with the reason in its tooltip, and the foreign key remains a real guard behind it.

Reversible means more than a disconnect¶

A soft-deleted rule, a rule an admin switched off, a deactivated connection, a connection in Error or Pending: in each case DeliveryRoutes::candidates(), which filters on ->active()->betweenHealthyConnections(), stops returning the rule as a route, so nothing scans it while the cursor keeps its position. Whatever undoes the state resumes delivery where it stopped rather than restarting it, and each of the four has its own reversal: a connection returning to Active covers the last two, while a soft-deleted or deactivated rule has to be restored or switched back on in its own right.

What the reconnect promise rests on¶

The claim above, that the first scan after a reconnect sends nothing, holds only while nothing moves content_changed_at during a disconnect. DisconnectConnectionDataJob and RestoreConnectionDataJob write through the query builder and therefore never touch it. A guard on each of them asserts the job leaves the cursor and the ledger untouched, and a third test drives a delivery, a disconnect, a reconnect and then a scan, asserting the already-delivered record stays suppressed by its ledger row while a pending one is still offered. CFX-212's criterion on which deletion causes move the change signal is what would make it a rule.

A soft-deleted ledger row still counts¶

The unique index on (sync_rule_id, data_domain, source_id) spans soft-deleted rows, so the scan reads them too: both the join in DeliveryScan and the lookup that pairs a record with its row use withTrashed(). This is the same reason AbstractTransformer looks every mirrored row up that way. Having read the row, the delta join leaves the record out, and the backfill's due filter drops it.

Excluding them would put the two halves of the system into disagreement. The scan would report the record as never delivered, the step would create it in the target, and the insert recording the new remote_id would then hit the index the trashed row still occupies. Under ADR-0015 the remote create has already happened by then, so the record could never be delivered again. Counting the row suppresses the record instead, and restoring the row recovers it.

Nothing soft-deletes a ledger row today. The column is kept because whether the ledger is ever pruned is an open decision, and a reversible removal is one of the mechanisms that decision may want. Until it is taken, a trashed row behaves as a row that is in the way, not as a row that is absent.

Recoverability¶