Skip to content

0032. Exclude record faults from circuit breaker failures

Date: 2026-09-17 Status: Accepted

Context

The circuit breaker counts consecutive failures per connection and opens the circuit when the count reaches a threshold, so a target that is down stops being hammered and the records queued behind it are not all burned through against a system that cannot answer. It counts every Throwable its callback lets out.

ADR-0025 already carved out one class of exception from that count: an answer the target gave that happens to be negative — "no contact with that id" — proves the request was well-formed, the credentials worked and the system responded. An exception marked ProvesIntegrationReachable therefore counts as a success and breaks the failure streak.

Building UserPlatformOutboundWriter surfaced a second class that fits neither arm. Some records cannot be delivered because of what is in them, not because of anything about the target:

  • a customer carrying neither an email address nor a phone number that normalises to E.164 has no addressee in any channel the target offers;
  • a custom attribute whose key collides with a field the target owns would rewrite the contact's identity if it were sent;
  • a projection from a domain the writer does not carry was mis-routed before it got here.

All three are refused before the target is contacted at all. Raised as an OutboundWriteException they would be counted as target failures, and a run over a dataset holding a handful of them would open the circuit on a connection with nothing wrong with it — stopping every deliverable record behind them. They would also be retried on every pass, forever, because that class promises a failure is worth trying again.

Three options were considered.

  1. Reuse ProvesIntegrationReachable. Cheapest, and wrong: that marker asserts the system answered, and here it was never asked. Using it would clear a real failure streak on the strength of a local data check, which is the opposite of what the breaker exists for.
  2. Leave them as OutboundWriteException and raise the threshold. Trades one wrong behaviour for another: a higher threshold makes the breaker slower to notice a target that is genuinely down, and the records still get retried indefinitely.
  3. A second marker with its own semantics. More surface, but it is the only option that states the actual fact — this failure is about the record.

Decision

App\Contracts\ReportsARecordFault marks an exception whose cause is the record being written rather than the system being written to. The circuit breaker treats it as neutral: it neither counts towards opening the circuit nor breaks a failure streak, and in the half-open state it neither closes nor reopens the circuit, so the next attempt gets to probe again with a record that might be deliverable.

That neutrality is the whole difference from ProvesIntegrationReachable, and it is deliberate in both directions. Counting it would take a healthy connection down over a data problem; clearing the streak would be just as wrong, because nothing was sent, so nothing was proven.

App\Exceptions\UndeliverableRecordException is the concrete exception carrying the marker. Like RemoteRecordGoneException and UnaddressableRemoteRecordException, it deliberately does not extend OutboundWriteException — that class promises a retry may help, and here the source row has to change first. An arch rule pins the separation, alongside the two that already existed.

AbstractOutboundWriter::write() rethrows it untouched and records it on the span, but does not mark the span as an error: nothing about the target failed.

Consequences

A writer now has a way to refuse a record that costs neither a retry schedule nor the connection's health. That is what makes it usable for ordinary data-quality problems, which are common in shop data and would otherwise have to be either swallowed or escalated into an outage.

The ledger records it as DeliveryStatus::Undeliverable, with no next_attempt_at. That status reports rather than gates, which is what makes it cheap: the refusal is decided locally, before the target is contacted, so a record that stays undeliverable costs a projection per pass and no request. The scan therefore needs no filter for it and the runner no guard — unlike remote_gone, where re-offering costs a request and delivering again would undo somebody's deletion.

What it buys is that an operator can list what is stuck and read last_error for why, instead of the record disappearing quietly into a scan that offers it forever. Writing that row is the delivery runner's step; what this decision settles is the state it writes and the exception that earns it.

The marker is general, not writer-specific. An inbound path that wanted the same treatment for a malformed source record could carry it too. Nothing does today.

The risk this decision accepts is a marker applied too broadly. A refusal the target made is an ordinary write failure, however clearly it names the field it disliked — a 400 from the target is evidence about the target's willingness to answer and belongs in the count. The rule is where the refusal was decided, not what it was about.