Skip to content

0036. Treat a rate-limit response as a signal from the target, not a failed write

Date: 2026-09-24 Status: Accepted

Context

A target that publishes a rate enforces it: User.com allows 10 requests per second per workspace and answers anything above that with HTTP 429 and a retry-after header. Delivery arrives in bursts even without a historical load. A price change across a catalogue, a bulk consent import in the shop or a large retry drain after an outage all produce more writes in a minute than a target may accept.

Until now a 429 was an ordinary write failure. The record got a failed ledger row and used one of its attempts, and the circuit breaker counted the response towards opening the circuit. A burst therefore burned retries on records that had nothing wrong with them, sent some of them to exhausted, and could open the circuit on a target that was up and answering.

Writers already declare requestsPerMinute(), and nothing read it.

Decision

Delivery paces itself against the declared rate and treats a rate-limit response as a statement about the target.

  • One budget per target connection. A target's limit belongs to the workspace a connection points at, so every rule delivering into that connection spends from the same budget. The budget is the writer's requestsPerMinute(), used in full, and each writer call takes one request of it. Suppressed and held records ask the writer nothing and take nothing.
  • A 429 is not a failed write. The writer throws TargetRateLimitedException with the wait the response asked for. Nothing is recorded against the record and none of its attempts is used. The whole target connection is blocked for the wait, capped at one day, or for 60 seconds where the response named none.
  • A 429 proves the target is reachable. The exception carries ProvesIntegrationReachable (ADR-0025), so the circuit breaker counts it as a successful interaction.
  • Short waits are sat out, long ones end the run. A wait up to throttle_max_wait_seconds (300 by default) keeps the chain alive: the next job releases itself until the wait is over, holding neither a worker nor the rule's lock. A longer wait ends the run as throttled, the dispatcher skips the connection until the block lapses, and the other rules on the connection read as throttled instead of turning stale while it does, not only the rule that met the 429. The next job waits only up to the same bound: one that wakes to a longer block, set meanwhile by another rule's 429, ends the run the same way.

Consequences

  • A burst is delivered completely, only more slowly, and no record is marked failed for it.
  • The bound on a sat-out wait stays below the chain heartbeat's staleness window, and the waiting job holds to it too, so a waiting chain is never taken for dead. Changing one of the two means checking the other.
  • One rule's 429 holds back every rule on the connection. The dispatcher skips them all, so only that rule has a throttled run, and the others read as throttled from the block itself rather than turning stale until it lapses.
  • The budget counts per 60-second window, so up to twice the rate can go out across a window boundary. A target that counts on a sliding window may answer 429, which is handled as above.
  • There is no fairness between rules on one connection, and two connections that point at the same workspace count separately.
  • Losing the cache loses the budget and the block. The next write then meets a 429 and sets the block again.