Respond to a Stopped or Held Delivery Rule¶
This guide is for platform support. It walks through what to do when outbound delivery for a customer's rule needs a look, whether you were paged by a Slack alert or called by a customer whose data has stopped moving into their target system.
Where the alert comes from¶
A volume breaker trip logs an alert-level line: Volume breaker held back the withdrawals of a
delivery rule., with the rule id, the pending count, what the target is known to hold, and the
threshold it crossed. Whether that line reaches Slack is a deployment setting, not something this
guide controls: the slack channel has to be in LOG_STACK, and LOG_SLACK_WEBHOOK_URL has to
point at a real webhook. If alerts have stopped arriving, that is the first thing to check, not the
breaker.
A customer can also reach you first, before any alert fires: they notice records missing in their WooCommerce shop, or the reverse, records that should have been removed and were not.
Step 1: Open the Delivery page¶
In the admin panel, open Delivery in the navigation. The badge next to it counts every open
delivery rule whose state is not ok among the rules you may access; it is cached for up to 60
seconds, so a rule that just tripped may take a moment to show.
Four tiles sit at the top: Global delivery, Stopped by people, Held by breaker, and Not
delivering, no stop. If the alert or the customer's complaint points at a single rule you already
know, skip straight to it: paste its UUID, or the alert's rule id, into the search field above the
groups (it also takes ?q= in the URL, so a link from an alert can open the page pre-filtered) or
open it directly by address. If instead many rules look stale at once, read the Global delivery
tile first, not the individual rules: it names whether a global stop, a dead scheduler ("Scheduler
down") or queue workers not finishing runs ("Runs not completing") explains all of them together,
before you chase any one rule's cause.
Below the tiles, every stop, hold and unhealthy rule you may access appears exactly once, grouped by why: Stopped by people (one card per manual stop, listing the rules a connection stop affects), Held by the volume breaker (one card per hold), and Not delivering, no stop (one card per remaining unhealthy rule, with its state and why). Each group follows your record scope: you see the global stop and the "Global delivery" tile always, but a rule's or a connection's stop, and a rule's own card, only where you may access that rule or connection. Each card opens that rule's or connection's page.
Step 2: Find the rule and read its state¶
Match the customer's rule, or the alert's rule id, to a card, or search for it directly as above. Open it: the rule page's status slot shows the same state, with a sentence saying what it needs whenever it needs something. The delivery state, on the card and in the slot, is one of:
| State | What it means for the customer |
|---|---|
| Stopped | Nothing moves under this rule: no creates, no updates, no withdrawals. Someone set a stop at global, target-connection, or rule level, and it stays in force until someone ends it. |
| Throttled | The target asked Conflux to wait (its rate limit), and every rule delivering into that connection waits until the time the tooltip names. Nothing is lost; delivery resumes on its own. See A rule shows as throttled. |
| Stale | The rule has an open route but no completed run recently. Creates, updates and withdrawals are all waiting, not refused; the scheduler may be behind, or something upstream of the chain is broken. |
| Withdrawals held | Creates and updates keep arriving. Only deletions and deactivations are paused, waiting on the finding below. |
| Failing | The last run completed but delivered nothing and every attempt failed. Data is not reaching the target. |
| Ok | Nothing needs your attention. |
Stopped outranks every other state, then Throttled, then Stale, then Withdrawals held, then Failing: a rule
that is both stopped and stale still shows as Stopped, because that is the more useful thing to
tell a customer. The rule page and the connection's flow can also show statuses about the rule's setup rather than its delivery (Config problem, Inactive, Pull out, Route closed, No run yet); see Sync Rules resource.
Step 3: A stop (the kill switch)¶
A stop can sit at three levels: global (every rule, set only by someone with delivery.stop_global),
one target connection (every rule writing into it), or one rule. Whichever applies, nothing under it
moves until the stop ends.
To end a manual stop on the rule itself, use Resume delivery in the rule page's status slot. For a connection-level stop, the slot links to the connection page instead, whose Resume delivery ends it; for a global stop it links to the Delivery page's Resume all delivery. Ending a stop does not replay anything that was skipped while it was in force; the rule resumes exactly where its cursor stood.
A stop on a customers rule also stops every consent rule that depends on it, whichever way the
consent rule runs, and that includes consent withdrawals: a contact who unsubscribed in the shop
stays subscribed in the target until the stop ends. The consent rule then reads Stopped too,
described as "Stopped through the customers rule" with the stop's reason. To end it, lift the stop
on the customers rule, with Resume delivery on that rule's page; the consent rule resumes with
it. On the consent rule's page, Open the customers rule in the status slot takes you there; the consent rule offers no Resume of its own for that stop. Stopping only the consent rule leaves the customers rule running. See ADR-0042.
Console fallback¶
When the admin panel is out of reach, the same actions exist as commands:
The stop's first argument is its reason, which is required. Replace --global with
--connection=<id> or --rule=<id> to scope the stop or resume to one target connection or one
rule. A stop set from the console is recorded without a user, which is how
it is told apart from one set by a person in the UI.
Step 4: A hold (the volume breaker)¶
Withdrawals held means the volume breaker judged the number of deletions or deactivations this rule
was about to send as implausibly large next to what the target already holds, and paused them.
Creates and updates were never affected.
The status slot shows the finding: how many withdrawals are pending, how many records the
target is known to hold, the threshold that was crossed, and when the hold was set. It offers two
ways to act on it, both requiring delivery.release_withdrawals:
- Release withdrawals. Confirm the exact number shown in the finding. The release stays in force until the records it confirmed have been withdrawn (or came back), across as many delivery runs as that takes, and releasing again replaces your earlier release. While it is in force, each run may withdraw up to what the breaker lets through on any ordinary day plus the number you confirmed, until the held records are gone, so a few deletions that arrive meanwhile do not undo the release, but a new wave past that trips a new hold. A new hold ends your release: its finding already includes the records you confirmed that are still held, so release that hold to cover them, or discard it and nothing is released. A held record that has come back in the source is not a withdrawal any more: it goes out as an update and does not count against the number. Use this only when the rule can confirm the source really did delete these records: check with the customer or their WooCommerce shop's own change log before releasing anything.
- Discard hold. Ends the hold without releasing anything. The breaker counts again on the next run: if the same tombstones are still there, it trips again immediately. Use this when you are not yet sure, or when you expect the underlying data to be fixed before the next run.
A hold can also end on its own, with no action from you: if the pending withdrawals disappear before you look (a truncated export the next import already repaired, for example), the next scan clears the hold as self-healed. If you open the rule and the hold is already gone with no entry from you, that is what happened, not a bug.
A rule shows as throttled¶
The target asked Conflux to wait: this is its rate limit, not a fault in the rule or the record.
The tooltip on the Delivery page and the connection page's subheading both say until when. The wait
applies to the whole target connection. The rule that met the rate limit shows as throttled at
once; the other rules delivering into it keep showing Ok until they would otherwise turn stale,
and then show as throttled instead. Nothing needs doing; delivery resumes on its own once the wait is over.
If the same connection throttles over and over, the target's rate is too tight for the volume this customer sends through it. That conversation belongs with whoever owns that workspace, not with a fix on this side.
A rule shows as failing¶
The last run delivered nothing and every attempt failed. The rule page's Records section opens on the records that did not make it (failed, exhausted, undeliverable), each with its last error, so you can tell a target that refuses everything from a few bad records. A failed record is tried again on its own schedule; the table shows when. An undeliverable one waits for its data to change at the source.
What to tell the customer¶
- Stopped: "Delivery into this connection is paused; nothing is going out until we resume it." Give a rough time frame if you know the cause.
- Withdrawals held: "New and changed records are reaching [target]. Removals are paused while we confirm the numbers; nothing is lost." Do not promise a release time before you have actually looked at the finding.
- Throttled: "[Target] asked us to slow down for a while; delivery picks up again on its own at [time from the tooltip], and nothing is lost."
- Stale: "Delivery looks idle rather than broken; we're checking why nothing has run recently."
- Failing: "Delivery is running but every attempt is failing; we're looking at the error."