0035. Hold withdrawals rather than halt a rule when the volume breaker trips¶

Date: 2026-09-24 Status: Accepted

Context¶

A delivery rule can be asked to withdraw far more records from a target than any real change would produce: a truncated export that reads as ten thousand deletions, a misconfigured filter that marks a whole catalogue gone, a bug in a source integration. ADR-0019 and ADR-0016 already narrow what can reach delivery as a deletion at all, but neither one catches a mass event that is itself wrong: both only ask whether a single record's deletion should propagate, never how many are propagating at once.

Whatever catches that has to assume most of what it catches is a false trip. The volume breaker counts what a rule is about to withdraw against what the target is known to hold, and a customer whose shop genuinely cleared out half its catalogue is indistinguishable, from the count alone, from a customer whose export broke. The only way to tell them apart is a person looking at the finding, and the people who can do that are internal support, working from the admin panel. Neither the source system's owner nor the target system's owner has any account there. So a trip is resolved on support's schedule, in hours or, if nobody is looking, days, and every trip a real deletion causes costs that support time whether or not anything was ever wrong.

That constraint decides more than the words "hold" and "release" suggest. A design that stops the whole rule until support clears it makes every false trip as expensive as the worst true one: no creates, no updates, nothing moving for as long as the finding sits in a queue. A design that lets anyone outside support see or clear the hold has no such person to hand it to.

Decision¶

The breaker holds back withdrawals only. Creates and updates for the same rule keep flowing while a hold is in force; VolumeBreaker::assess() is asked before every step and only ever changes what that step does with a deletion. A row it holds back takes the ledger status held rather than withdrawn, remote_id and the delivered version stay as they were, and the row is offered again once the hold ends.

A hold is recorded as a DeliveryStop, scoped to the one rule, with the finding attached. It carries the pending count, what the target is known to hold, and the threshold the count crossed, so support reads the finding without re-deriving it. Recording it as the same model a manual stop uses means one table and one admin surface answer "what is stopped and why" for both a person's decision and the breaker's.

Support settles a hold one of two ways. A release names the exact number of withdrawals to let through, taken from the finding. The release stays in force until the rows it confirmed are gone, withdrawn or come back live, and it applies to every step meanwhile, whichever run the step belongs to. Tying it to one run or one step would lose it whenever that run could not use it: only the drain hands held rows back, so a chain already past its drain when support acts, or one that ends early, would spend the release without withdrawing anything, and the next run would trip again. The allowance is added on top of the normal limit rather than replacing it: the confirmed withdrawals go out, a few that arrive meanwhile still fit under the normal limit as they would on any day, and a count past the sum trips a new hold. That new hold ends the release in force: its finding already counts the confirmed rows still held, so releasing it covers them again. A newer release replaces an older one, since the older answered a finding nobody has confirmed again. Discarding ends the hold without releasing anything and leaves no release in force; the next step counts again with no allowance, and a finding that has not actually changed trips the same hold at once.

A held row counts as pending only while its record is still deleted. A held row whose record came back live goes out as an ordinary update once the hold ends, so it is not a withdrawal and the breaker does not count it. A release therefore never has to cover records that are no longer gone.

A hold also ends on its own, without a person, when it turns out to have been a false trip that repaired itself. If nothing is held and the scan's window holds no pending withdrawal by the time a scan has read it through, the hold is lifted as self-healed. This is the one case a system may close without support, because nothing is released. It exists for the most common false trip: an export that came back truncated, where the next full import brings the records back before anyone has looked.

Alternatives considered¶

Halt the whole rule. Rejected. It multiplies the cost of a false trip by every create and update the rule would otherwise have delivered while the finding waits in support's queue, which for a customer that is actively selling can be days of silence for a single bad batch.

No latch at all, and rely on delete_policy to keep mass deletion out. Rejected. A policy is a per-record gate; it has no way to ask "how many at once", so a config that already allows deletion propagation offers no protection against an event that produces too many of them in one pass.

An unbounded release. Rejected. Clearing the hold without a number would wave through whatever the next scan finds pending, including records that arrived after support looked at the finding, so support would be signing off on a count it never saw.

Automatic release after a grace period. Rejected. A permanently truncated export, the shape of fault a grace period cannot tell apart from a real mass deletion, would then have its records deleted in the target the moment the period ran out, which is exactly the outcome the breaker exists to prevent.

Consequences¶

Every propagating rule now carries a fifth ledger status, held, alongside a stop record with the finding on it, which is more state than a rule that halted outright would need. Removing held later would mean deciding again where a paused withdrawal goes instead: back to failed loses the finding, and a new status is the same cost this decision already pays.

A held rule keeps delivering everything that is not a withdrawal, so its DeliveryRuleState reads withdrawals_held rather than stopped: a customer watching their own data still sees updates and new records arrive, with only removals paused.

A release is bounded to what support confirmed plus the normal limit, so a customer whose deletions keep arriving in volume after the release still needs a second look. That is deliberate: an allowance answers the finding support saw, not an open door for the rule.

Self-healing depends on the count matching the scan's own window and reading it through to the end. It costs one extra count per completed scan, and it means a hold that would have ended on its own still shows in the admin panel until a scan actually clears its window, rather than the moment the underlying records reappear.