Skip to content

0046. Require a recorded confirmation to start a backfill by hand

Date: 2026-10-06 Status: Accepted

Context

A rule's first activation opens a backfill over everything the rule already carries, bounded by the moment of activation (ADR-0037). Without this decision the kill switch is the only way to stop one, and it stops delta as well. Nothing can start a backfill again once one has ended.

A backfill is the run most likely to create records in bulk. Every create can set off whatever the target does on a new record, such as a welcome automation that mails every imported customer. A backfill is usually stopped for exactly that reason, and a restart repeats the exposure.

What a target does on a new record is configured inside the target. Conflux does not own that configuration and does not read it, so nothing Conflux sends can rule the reaction out. What can be enforced is that somebody looked, and that it is known who.

The initial backfill carries no such sign-off, because connecting a shop is already the merchant asking for their data to be there. A backfill somebody starts later is a different act.

A rule of a dependent domain, such as customer_consents, opens no backfill on its first activation (ADR-0042). Its records are never created in the target. Each one is written onto its parent's record by update, and the parent is found through the ledger of another rule. ADR-0042 leaves the guarded backfill to this decision and asks that its guard cover dependent rules explicitly.

Decision

A backfill started by hand requires a recorded confirmation. Aborting one needs none. The two actions sit in the header of the rule page, next to "Stop delivery", and not in the connection's flow panel, which stays an overview.

  • Only a rule of a root domain can be backfilled by hand. The guard rests on a number of creates, and a dependent rule sends none. Which of its records the runner would write depends on two ledgers, on whether the target accepts the record and on echoes of the target's own writes, and remote_gone is not terminal for it. No count gives the operator a number a recount could hold the start to. The target's reaction differs as well. A consent backfill into Positive User asks for every unconfirmed opt-in, which sends one double opt-in mail each, and fires the event automations trigger on for every withdrawal (ADR-0045). The dialog's question about what the target does on a new record does not cover that. The start action is not offered on a dependent rule. Backfills::canStart() holds the conditions a rule must meet, a live, active, push-out rule of a root domain, and the action and Backfills::start() both use it, so the service refuses what the action would not offer. SyncRulePolicy::backfill() stays the question of who may.
  • The guard is a confirmation, not a suppression. It does not stop the target from reacting. It records that somebody checked what the target does on a new record before the run started.
  • One confirmation per start, stored as its own record. Each start writes a row in backfill_confirmations: the rule, the rule's target connection, the user, the bound of the backfill it opened, the number of records in its window and the number of creates confirmed. The row is never updated and has no soft delete. A standing confirmation per connection would outlive the check it stands for, so there is none.
  • The confirmation types back the number of creates. The dialog says how many creates the backfill would send and asks for that number to be typed, next to a ticked box saying the target was checked. Typing the number makes the person read it.
  • The bound and the numbers are fixed when the dialog opens. The bound is the moment the dialog opens, cut to the second. The dialog counts once against it and keeps the bound and both numbers for as long as it is open. Every render and the form check read those values and count nothing. The typed number has to equal the number shown. Records that arrive while the dialog is open last changed after the bound, so they cannot move the numbers.
  • The start counts once more and refuses only a greater number. The service counts again against the bound the dialog fixed. In normal operation the count shrinks. Delta keeps delivering records from inside the window, and they gain a reference. Records that change move past the bound and become delta's. So a smaller recount starts the backfill, and the confirmation stores the number the operator authorised. The count can also grow. A change signal stamped just before the bound can become visible only after the dialog counted. A data restore brings records back live without moving their change signal. A corrected undeliverable record counts as a create, and so can a record whose ledger row somebody edited. A greater recount therefore refuses the start and shows the new number under the same bound, because nobody confirmed the creates above theirs.
  • The values travel with the dialog, and forging them is harmless, with one exception. The bound and the numbers are held by the browser while the dialog is open, so a request can carry other values. An earlier bound only narrows the backfill. A lower number of creates is refused by the recount. A higher one only authorises more than the backfill would send. A bound in the future is refused by the form and by the service, because it would pull records that are delta's into the backfill. The number of records is only shown. The confirmation stores the recount's. The exception is the exact bound of a backfill aborted a moment before, which somebody holding delivery.backfill can forge as the bound of a new start. A delivery step that read the cursor before the abort then passes the compare-and-set on backfill_until and writes its position or its finish onto the new backfill, and a finish ends it at once. The worst outcome is a backfill that ends early or skips part of its window, which another start repairs. Nothing is sent that was not confirmed.
  • A create is a record the runner would create in the target. On a rule of a root domain, that is a live record with no ledger row, or with a live ledger row that holds no remote_id, is in no terminal status, and records no delivered version or an older one than the record's. These are the runner's own checks, read in SQL. A record whose earlier create failed has a ledger row and no reference, and it is created again. Where that row still waits for its retry, the drain creates it rather than the backfill. A record whose row is soft-deleted, remote_gone or exhausted is skipped by delivery, so it is not counted. An undeliverable record counts as a create. Neither delta nor the drain offers it again, so a source row corrected since the refusal would be created by the backfill. The database cannot tell whether it was corrected, so the count errs high. A record whose row already holds its version is suppressed. Tombstones are never creates.
  • The number of records is the backfill's window. It counts what the backfill walks: the records whose last change lies before the bound, and on a rule that propagates deletions, the tombstones of deletions the source reported. The backfill withdraws those the target holds and skips the rest. The dialog's number and the total of the progress shown after the start are the same count.
  • No console path. The confirmation is only worth its link to a signed-in person, and a command has none. delivery:stop keeps its console fallback because a stop only delivers less.
  • A permission of its own. delivery.backfill goes to SuperAdmin and PlatformSupport. SyncRulePolicy::backfill() requires it and canManageRecord(), like stopDelivery(), and gates both actions.
  • Abort and start act through compare-and-set on backfill_until. An advance, a finish or an abort writes the backfill columns of the cursor only where backfill_until still holds the value the writer read. A start writes only where backfill_until is null, that is, where no backfill is in progress, so two starts cannot both win. A delivery step that read the cursor before an abort or a restart finds the condition false, writes nothing and logs it. It cannot undo either. The next step reads the cursor afresh: it finds no backfill and carries on in delta, or it finds the new backfill and starts it from the beginning. This relies on two backfills of one rule having different bounds. A bound is stored to the second, so a start, an abort and a new start within one second would share one. The dialog in between rules that out in practice. A bound forged to the aborted one is the case it does not rule out, named above.
  • The batch running at an abort still finishes. The abort prevents every later backfill batch, not the one in flight. The kill switch remains the way to halt everything at once.
  • An abort is logged, not recorded. The log line names the user, the rule and the bound that ended. An abort only delivers less, so a table row would buy nothing.
  • A restart does not rewrite what the first run delivered. A backfill writes an unchanged, already delivered record as suppressed. Only records that changed, or that the target never got, go out again.

Consequences

  • A manually started backfill cannot begin without a named person having been shown the number of creates. The connection page shows the latest confirmation on a target, and a running backfill's fact on the rule page says who confirmed it.
  • A dependent rule offers no start by hand. Its records that sat in the source's mirror before the rule was activated reach the target once they change. In the standard case there are none: connecting a shop creates the customers rule and its dependent rules together, and the import after that stamps what it brings later than their activation. Two cases leave records behind: a Positive User workspace attached to a shop Conflux already mirrors, and a dependent rule deleted and created again. A manual backfill for dependent rules, with a guard of its own, is left to a later decision.
  • The guard does not prevent a bad outcome. Somebody can confirm without checking. What the record gives is an owner for the decision, and the number they saw.
  • Nobody can start a backfill from the console or from a script. A rollout across many rules means one confirmation per rule.
  • Records that arrive between the dialog opening and the submit do not block the start. They belong to delta. A dialog left open for long starts a backfill whose bound lies that far back, and delta covers everything after it.
  • A delivery step that overlaps an abort or a restart drops its write to the cursor and logs it. Dropped writes are expected after either action, and they are not faults.
  • An abort ends the initial backfill too. The records it had not reached are delivered once they change, or by a later backfill started by hand.
  • backfill_confirmations has no soft deletes: the row records a decision and goes only with its rule. A force delete of a rule deletes its confirmations first, in the same step as its stops, and a rule with a confirmation already holds a cursor, so a force delete is refused anyway.