Skip to content

0009. Circuit breaker for outbound integration requests

Date: 2026-07-23 Status: Accepted

Context

Outbound HTTP calls from platform integrations — WooCommerce and PrestaShop today, any future AbstractIntegration subclass tomorrow — went straight through Http:: with no protection against a failing or degraded external system. A shop's WooCommerce or PrestaShop instance going down, timing out, or returning 5xxs did not stop Conflux from continuing to dispatch requests at full rate: every queued job, every retry, every scheduled bulk export kept hitting the same broken endpoint, compounding load on an already-struggling system and burning through job attempts and queue capacity for no benefit.

Before implementing this in-house, an existing Composer package, algoyounes/circuit-breaker, was evaluated as a build-vs-buy candidate. It was rejected: 59 GitHub stars and a single maintainer are too thin a foundation for a dependency sitting on every outbound integration call. The state machine a circuit breaker implements (Closed / Open / Half-Open) is well-understood and small enough to own directly, backed by infrastructure (cache) Conflux already depends on.

Decision

Conflux implements its own circuit breaker as a plain service class, App\Services\CircuitBreaker, rather than pulling in a third-party package. It tracks the classic three states per SystemConnection:

  • Closed — the default state. Calls run normally; consecutive failures (any Throwable) are counted. Reaching failure_threshold consecutive failures opens the circuit.
  • Open — calls are rejected immediately, without invoking the wrapped callback, by throwing CircuitBreakerOpenException. This is the point of the pattern: stop hammering a system that has already demonstrated it is failing. Once cooldown_seconds has elapsed since opening, the circuit moves to Half-Open.
  • Half-Open — exactly one probe call is let through (guarded by a short-lived lock so concurrent callers don't all probe at once). A successful probe closes the circuit; a failed probe reopens it and restarts the cooldown.

State is tracked entirely via Laravel's cache facade — one key for when the circuit opened, one for the consecutive failure count, one short-lived lock for the half-open probe — keyed per SystemConnection. No new infrastructure dependency (no separate store, no new table) is introduced; the cache Conflux already runs is sufficient because circuit state is inherently short-lived, best-effort bookkeeping, not data that needs durability guarantees.

The breaker is wired into App\Services\AbstractIntegration, the shared base class both WooCommerceIntegration and PrestaShopIntegration extend, rather than into each integration individually. The four outbound template methods — verify(), streamBulkExport(), probeBulkExport(), and fetchBulkExportPageRange() — route their inner call through CircuitBreaker::attempt(SystemConnection $connection, Closure $callback). Centralizing this in the base class means the protection is structural, not opt-in: an integration author writing a new integration gets the breaker automatically by extending AbstractIntegration, the same way they already get the IntegrationRequestException / InvalidIntegrationResponseException error-handling guarantee described in Integration Contract Design. Duplicating the breaker call inside each integration would make it something every new integration has to remember to add, and something a future integration author could plausibly get wrong or skip.

AbstractIntegration takes the breaker as a constructor-injected dependency with a default value (new CircuitBreaker, PHP's "new in initializers"), so existing direct instantiations (new WooCommerceIntegration(), new PrestaShopIntegration()) keep working unchanged while Laravel's container still resolves CircuitBreaker normally when an integration is built via IntegrationRegistry::resolve().

Consequences

Any future AbstractIntegration subclass gets circuit-breaker protection on its outbound calls for free, with no per-integration wiring required. failure_threshold and cooldown_seconds are configurable per deployment via config('conflux.circuit_breaker') (CIRCUIT_BREAKER_FAILURE_THRESHOLD, CIRCUIT_BREAKER_COOLDOWN_SECONDS), defaulting to 5 failures and a 60 second cooldown.

handleWebhook() is deliberately left outside the breaker. The breaker protects Conflux-initiated outbound calls against a struggling external system; webhook handling is inbound — the external system calls Conflux, not the other way around — so there is no outbound request for the breaker to guard, and wrapping it would misapply the pattern to the wrong side of the connection.

Because breaker state lives in the cache rather than the database, it is deliberately not durable or auditable history: a cache flush silently resets every connection to Closed, and there is no persisted record of when or how often a given connection's circuit has tripped. If that visibility becomes a requirement later, it will need a dedicated decision rather than an extension of this one.

Connection status on circuit open

When the breaker actually opens — failure_threshold consecutive failures exhausted, not on any single failed attempt — BulkSyncJob and FetchBulkPageRangeJob catch the resulting CircuitBreakerOpenException and set the affected SystemConnection's status to SystemConnectionStatus::Error via a direct update() query. This is a deliberate escalation from "this call failed" to "this connection needs human attention": a connection stuck in Error is excluded from the scheduled sync:dispatch command (SystemConnection::active()) and from the manual "Bulk Sync" admin action, which is already disabled for unhealthy connections. Recovery is intentionally manual — the connection owner must reconnect/re-verify — rather than the platform silently retrying a dead connection forever in the background. The existing reconnect handshake (SystemConnectionService::activate()) additionally calls CircuitBreaker::reset() on success, so a freshly reactivated connection isn't rejected by a stale cooldown left over from before the reconnect.