0029. Stop a delivery run on a fault of the rule, park a record on a fault of the record¶

Date: 2026-09-10

Status: Accepted

Context¶

The delivery runner (CFX-216) is the first piece of code that writes a DeliveryStatus at all. The four stages under it were built one at a time and each of them settled a question in isolation; the runner is where all four meet a live target and start producing outcomes.

Two of those settlements decide what the runner has to cope with. Under ADR-0015 the ledger row decides create against update before the writer is asked: a row holding a remote_id means update by that reference, anything else means create, and nothing interrogates the target to check. Since ADR-0021 a writer states which verbs it offers in its type, and a target that has no create or no update is an ordinary target rather than a broken one. Put together, the decision the ledger makes can land on a verb the writer in front of it does not have.

A record can go wrong on its own account too. The target can refuse the write for reasons that pass — a timeout, a rejected field, a rate limit — or answer that the record it once created is not there any more, or take a create and hand back nothing that addresses the result, which leaves a record in the target that nothing can ever reach again. And a chain as a whole can be stopped by something that has nothing to do with any record: either side of the route stops being healthy, or the circuit breaker from ADR-0009 trips because the target is down. A rule has one job at a time, and a job that finds the rule taken is not a fault of anything: it waits and runs once the rule is free.

So every fault a chain meets has to be routed to one of two places: the chain, or the row of the record it struck. Both directions of the mistake cost something, and neither cost is theoretical. Routing a record's fault to the chain stalls the rule behind one bad record, which is the failure the cursor design already refuses by advancing past every record it attempted. Routing a rule's fault to the records writes one ledger row per record for a single fact, gives each of those rows a retry schedule that can never succeed, and lets the changes themselves pass the cursor with nothing recording that they were never delivered.

Decision¶

A fault that would fall the same way for every record of the rule ends the chain, and nothing is written for it. There are six, and none of them is about the record the step happened to be holding:

A fault that belongs to one record lands in that record's ledger row, and the cursor moves past it. A transport failure counts an attempt and takes a failed row with the time of its next one. A target reporting the record gone takes remote_gone. A create that comes back unaddressable takes exhausted, because the record now exists in the target and a second attempt would make a second copy of it. The job carries on with the next candidate in the batch.

exhausted is a status of its own, rather than failed with a null next_attempt_at. It covers both ways a record runs out of road: the retries are used up, or the create produced something unaddressable. DeliveryStatus::isTerminal() returns true for it and for remote_gone, and those two are the whole of the terminal set — a delivered or suppressed row is offered again as soon as the record's change signal moves.

Two of the six are ordinary return values; the other four fail the job and cancel the batch. A closed route and an open breaker are DeliveryResult cases, returned rather than thrown, because the step did what the situation called for and a later chain resumes where this one stopped: the connection comes back, the breaker cools down. The job adds no successor, the batch closes as finished, and the next tick starts a chain again once the route is open. The other four are exceptions that reach the queue and land in failed_jobs, and a failed job cancels its batch, which ends the chain, because nothing about a later attempt would fix them. A driver without a writer, a writer without a verb the rule needs, a ledger row that exists where the scan said there was none, a mirror row the scan should never have handed back at all — each is a configuration or an invariant, and each wants somebody to look at it rather than a retry. The last of them is raised rather than skipped for the reason the whole ADR turns on: a candidate passed over silently is a change that leaves no trace of never having been delivered.

Whichever way a chain ends, nothing is rolled back. What was delivered stays delivered, and the cursor stands where the last batch that concluded left it, so the rule resumes rather than restarts.

Retries come before fresh deliveries. A chain opens with the drain and ends with the scan. The drain's set is finite and shrinks within a chain, because every row it attempts either lands or is given a next attempt in the future. The scan's set is the rule's changes, and a rule whose changes arrive faster than it delivers keeps its scan moving indefinitely; a chain that scanned first would never reach its drain for such a rule, and a record that failed once would never be retried. Putting the drain first bounds that: fresh changes wait for the due retries of that tick, which are at most what has come due since the last chain, and never the other way round.

The alternative that was not taken: a ledger status per record for a missing verb¶

A missing verb could be recorded like any other per-record outcome, with a status of its own on every candidate that needed it. It was rejected because the fault belongs to the rule. Every record under that rule needing that verb fails in exactly the same way, so one row per record is a thousand copies of one fact, and each of those rows would then be picked up by a retry pass that cannot succeed until somebody changes the configuration. Halting is loud, loses nothing, and once CFX-217 validates a rule's policy against the target's verbs at save time, the case stops arising at all.

The alternative that was not taken: skipping with a log line¶

The cheapest reaction to a candidate the writer cannot take is to pass over it and log. It was rejected because the cursor moves past a skipped candidate, and once it has cleared the overlap window a delta scan has nothing that brings the record back. The changes would be gone with only a log line saying so, which is exactly the shape of failure that is discovered months later by somebody comparing two systems by hand.

The alternative that was not taken: failed plus a null time in place of exhausted¶

Reusing failed and reading next_attempt_at is null as "given up" adds no enum case, and the data already distinguishes the two states. It was rejected because it hides the meaning in a combination of two columns. Every rule that has to leave such a record alone would have to read both, an operator filtering the ledger would have to know the convention, and a row written with a null time by mistake would silently become a record nobody delivers again. An explicit status keeps the skip rules status-based and gives the admin panel something to filter on.

Consequences¶

CFX-217 moves the verb check to save time. Validating a rule's delivery policy against what the target's writer can actually do is that ticket's job, and it makes UnsupportedDeliveryVerbException unreachable through the ordinary path. The exception stays as the assertion behind the validation, in the place where the write would otherwise happen.

A misconfigured route fails its job on every tick until somebody fixes it. A system that declares push_out for a domain and has no writer registered for its driver produces a failed job and a cancelled batch every five minutes, and the same holds for a rule whose target lacks the verb the rule needs. That is deliberate. The alternative is a route that quietly delivers nothing, which looks identical to a route with nothing to deliver.

An operator finds parked records by status. exhausted and remote_gone are queryable, each row carries the last_error that put it there, and next_attempt_at is null on both, so nothing picks them up again on its own. Getting a record out of either state means changing the row, and no interface offers that yet: until one does, an operator resets it in the database.

The connection is not set to Error on an open breaker. This differs deliberately from the inbound BulkSyncJob, which does mark its source connection. For a target the breaker's own cooldown is the intended recovery, and the next tick tries again by itself. Setting the connection to Error would close the route on the health gate every route lookup passes through, and keep it closed until a person brought the connection back by hand — a manual step for an outage that resolves itself.

A job that finds the rule taken waits; a job that throws takes the chain with it. Exclusivity is a job middleware holding a lock for the length of one job, rather than a lock held for the length of a whole pass. A job that cannot take the rule's lock is released and offered again seconds later, and that release counts as no attempt. The only collision it has to cover is a tick that judged the previous chain dead, and that tick has already cancelled the dead chain's batch, so a job of the cancelled chain that does turn up ends the moment it starts: nothing reaches the ledger and nothing moves the cursor. No job is ever retried after an exception, either, and a timeout is held to the same rule rather than a looser one: $maxExceptions counts a timeout against the same budget it counts a thrown exception against, so the first timeout fails the job exactly as the first exception does. The first exception to leave a job fails it for good and cancels its batch, which ends the chain, because every exception in the table above is a fault of the rule that a second attempt would meet again.

A record parked in a terminal status stays parked however often its source row changes. That is the point of the two terminal values, and it is the one place where delivery deliberately ignores a change signal. For remote_gone it protects a deletion somebody made in the target on purpose; for exhausted it stops a record the target has refused five times from being offered a sixth, and stops an unaddressable create from being repeated into a second unreachable copy.

A fresh change waits behind the retries that are due when its chain starts. For a rule whose target is healthy that is a few rows; for a rule whose target rejects everything it is the whole due set, once per tick, and the breaker halts the chain before the scan if the target is down.