Skip to content

0025. Exclude answered negatives from circuit breaker failures

Date: 2026-08-31 Status: Accepted

Context

ADR-0009 defines the breaker's failure rule as "any Throwable". With the four outbound operations that existed at the time — verify, stream, probe, fetch page range — that rule was exactly right. Each of those either succeeds or fails; there is no third outcome. Any exception really did mean something had gone wrong with the remote system.

CFX-179 introduced the first operation with a third outcome. CanPullConsent::pullConsent() asks a system for one contact's consent, and "there is no contact with that id" is a legitimate, expected answer — ConsentTargetNotFoundException, whose own docblock says the caller should skip the sync and log a warning rather than retry. That is already incompatible with counting it as a failure, in the same commit that wrapped the operation in CircuitBreaker::attempt().

The contradiction stayed unreachable until CFX-180 supplied the first real implementation: until then performPullConsent() was a placeholder throwing BadMethodCallException, and nothing in the codebase could produce a ConsentTargetNotFoundException at all. Positive User answers an unknown contact with a plain 404.

Under "any Throwable", such a 404 counts as a failure. With the default threshold of five, a handful of contacts Conflux does not know about in the target system open the circuit on a connection that is demonstrably healthy — it just answered five questions correctly. Every subsequent call for that connection is then rejected for the cooldown, including calls that would have succeeded. The breaker would be reacting to Conflux's own incomplete data rather than to the remote system's health, which is the opposite of what it exists for.

The obvious narrow fix — teaching CircuitBreaker about ConsentTargetNotFoundException — was rejected. The breaker is generic infrastructure shared by every integration; giving it knowledge of a consent-specific exception inverts the dependency, and the next operation with an expected negative (a consent push against a contact that was deleted, a lookup by an id a shop has since reused) would have to add its own special case in the same place.

Decision

The failure rule is narrowed: an exception counts as a failure unless it is itself evidence that the remote system answered. Such exceptions are marked with the interface App\Contracts\ProvesIntegrationReachable, which extends Throwable so only exceptions can carry it. ConsentTargetNotFoundException is the first implementer.

A marked exception is treated as a success for health purposes, not merely as a non-failure:

  • Closed — it clears the consecutive-failure counter, exactly as a successful call does. A definite answer breaks a failure streak; the system demonstrated it is reachable.
  • Half-Open — it closes the circuit rather than reopening it. Reaching the system and getting an answer is precisely what the probe was checking for.
  • Open — unchanged. The callback is never invoked, so there is no answer to observe; the call is rejected with CircuitBreakerOpenException like any other.

In every case the marked exception is rethrown unchanged. The breaker's health bookkeeping and the caller's error handling are separate concerns: the caller still has to deal with the missing contact.

The marker is deliberately narrow, and its docblock says so: it is for a definite answer from the remote system. A timeout, a connection error, a 5xx, or a malformed response are all genuine evidence of ill health and must never carry it. A 404 qualifies only because the request was well-formed and the credentials worked — the system understood the question and said "no".

Consequences

ADR-0009's decision stands; only its failure rule is refined, and its "any Throwable" wording should be read against this ADR. The three-state machine, the per-connection cache keys, the thresholds, and the placement in the capability traits are all unchanged.

Marking an exception is now a design decision with real consequences in both directions. Mark too much — a 5xx, say — and the breaker stops protecting anything, silently, because nothing will ever open the circuit again. Mark too little and the original CFX-180 problem returns for the next operation with an expected negative. The judgement is per exception class and belongs in that class's docblock, which is why the marker sits on the exception rather than being passed per call site: a call site could get it wrong differently each time.

Because a marked exception clears the failure counter, a connection alternating between real failures and answered negatives will not accumulate towards the threshold. This is intended — the alternating answers prove the system is up — but it does mean a partially degraded system that fails some requests and answers others stays out of the Open state longer than the raw failure count alone would suggest.