Back to all articlesIntegration reliability

Decide when a circuit breaker should stop calls to a failing dependency

Alactic Editorial9 min read

Prepared with AI assistance by Alactic Editorial. Examples identified as hypothetical are illustrative, not customer results.

Your integration is timing out, but workers keep accepting jobs and repeating the same unsuccessful request. The backlog grows, and unrelated work waits behind calls with little chance of succeeding.

A circuit breaker can stop that cycle. It suspends calls within a defined dependency scope when enough evidence indicates sustained trouble, then admits limited recovery traffic. The difficult decisions are which failures count, what happens to withheld work, and how to restore traffic without another burst.

The following fictional policy makes those decisions executable. Its numbers are hypothetical design choices, not universal thresholds or evidence of controls available in any Alactic product.

1. Identify the resource and dependency scope

A circuit breaker makes an admission decision before an outbound call starts. Closed permits ordinary traffic; open suppresses it; half-open admits bounded recovery probes. This example adds an explicit recovery-ramp state between probing and normal operation.

Start with the resource being exhausted: worker slots, connections, queue capacity, or the time available to complete a business task. Then identify the operation causing that pressure. A breaker does not replace deadlines, which release occupied resources, or concurrency limits, which contain pressure before failure evidence accumulates.

Scope calls by a plausible shared failure cause. A global provider breaker could disable healthy operations because one route fails. A breaker per individual record may see too little traffic to establish a pattern.

In this fictional example, the scope is customer-record lookups through the primary account connection. Its normal concurrency budget is 20 outbound calls across all application instances. Unrelated work has a separate worker allocation. These are recommended design choices for the example, not product claims.

2. Classify evidence before counting errors

An unsuccessful business request does not necessarily indicate an unhealthy dependency. Define the classification before setting a threshold:

  • Qualifying health failure: a connection failure, dependency deadline expiration, or confirmed transient service-error response.
  • Positive health evidence: a timely, valid response from the affected operation. This may include a documented request-specific rejection that demonstrates the operation is responding normally.
  • Request-specific failure: invalid input or a missing record. The job remains unsuccessful, but the response need not count against dependency health.
  • Credential or configuration failure: suspend the affected route for intervention rather than repeatedly testing unchanged credentials.
  • Rate restriction: apply a separate dispatch pause based on the provider's documented limiting scope.
  • Inconclusive outcome: local cancellation, ambiguous responses, or an error whose meaning has not been established. It proves neither recovery nor dependency failure.

HTTP 429 indicates rate limiting; a Retry-After header may indicate when to retry. Providers apply limits differently. Do not assume that one account's restriction establishes a provider-wide outage.

For the rolling failure percentage below, count qualifying failures and positive health observations only. Record inconclusive and other excluded outcomes separately. This prevents local cancellations from masquerading as remote failures or recovery evidence.

3. Copy this filled decision artifact

Fictional worked artifact: Every duration, count, and limit below is hypothetical. The records are intended to be copied into an operating policy and adapted to one dependency operation.

Shared operating settings

  • Owner: integration operations lead.
  • Scope: primary-account customer-record lookups.
  • Normal allowance: 20 concurrent outbound calls, shared across instances.
  • Call deadline: five seconds, including recovery probes.
  • Deferred work: at most 500 jobs, expiring ten minutes after original enqueue; overflow is rejected with an explicit capacity result.
  • Retry ownership: the dispatcher owns retries; callers must not add independent retry loops.

Decision A: sustained health failure

  • Evidence: at least 30 classified health observations in a rolling 60-second window, with at least 60% qualifying failures. Count each outbound attempt once; label retries separately.
  • Action: open for 45 seconds. Reject interactive lookups as temporarily unavailable; defer eligible background work without retaining workers.
  • Recovery: enter the bounded half-open procedure in section 6.
  • Further failure or insufficient recovery evidence: reopen for 90 seconds, then repeat that procedure.

Decision B: sparse traffic

  • Evidence: five consecutive qualifying health failures within ten minutes, without intervening positive health evidence.
  • Action: use Decision A's open behaviour even if the rolling minimum is unmet.
  • Sequence rules: positive health evidence resets the consecutive-failure sequence. Inconclusive outcomes neither increment nor reset it; failures older than ten minutes expire.
  • Recovery: use the same half-open procedure; do not manufacture business demand simply to fill the sample.

Decision C: account rate restriction

  • Evidence: a confirmed rate-limit response documented as applying to this account.
  • Action: pause affected dispatch separately from the health breaker. Respect provider retry guidance when supplied; otherwise use a documented bounded backoff policy.
  • Recovery: admit one eligible request when the pause ends. Another restriction renews the pause; positive health evidence permits entry to the capped ramp, provided no health or credential suspension remains.
  • Other outcomes: a qualifying health failure opens the health breaker; inconclusive evidence renews the documented backoff rather than releasing normal traffic.

Decision D: credentials and individual requests

  • Credential failure: one confirmed invalid-credential result suspends that credential route pending authorised correction.
  • Verification: after correction, allow one controlled call. Positive health evidence permits the ramp if other gates allow it; another credential failure keeps the route suspended.
  • Other verification outcomes: qualifying health failure opens the health breaker; inconclusive evidence keeps the route suspended for operator review. A rate restriction follows Decision C.
  • Invalid individual request: reject that job with an actionable reason. A timely, documented validation rejection may establish health, but never successful completion of the business task.

Admission requires every applicable gate to permit the call. Clearing a rate restriction does not override a credential suspension or an open health breaker.

4. Check the arithmetic against real traffic patterns

Consider a hypothetical minute containing 40 classified observations: 26 qualifying failures and 14 positive responses. The failure share is 26 divided by 40, or 65%. Both Decision A requirements are met: at least 30 observations and at least 60% failures. The breaker opens for 45 seconds.

Now consider two failures among three observations. The apparent failure share is higher, but the minimum sample is unmet. That alone does not open this breaker. Five consecutive qualifying failures within ten minutes would activate the separate sparse-traffic rule.

The trade-off is deliberate. A minimum sample reduces false trips but can delay containment on quiet routes. A consecutive-failure rule provides another path without pretending that sparse traffic supplies a large statistical sample.

Also inspect distinct jobs. Repeated attempts from one broken job can dominate the observation window. Attempt counts describe resource pressure; distinct-job counts help reveal whether the incident is broad or concentrated. Neither substitutes for correct outcome classification.

5. Make open-state behaviour explicit

Suppressing calls is only half the policy. Interactive callers need a prompt unavailable result, not an empty customer record that looks like a successful lookup. Deferred jobs need bounded storage, expiry, and a visible reason for waiting.

In the fictional policy, the 501st deferred job receives an explicit capacity rejection if the queue remains full. A job reaching its ten-minute expiry is marked expired rather than silently retried. Neither result counts as a dependency-health failure because no outbound attempt occurred.

Existing calls still finish or reach their deadlines after opening. Record their outcomes for incident analysis, but do not let those old in-flight calls establish recovery. Half-open evidence must come from explicitly admitted probes in the new probe period.

For operations that change remote state, resolve uncertain outcomes before retrying. A timeout does not prove that the remote action never happened. A fallback also needs its own capacity limits and authorisation checks; failing over does not make execution automatically safe.

6. Bound probes and ramp with integer limits

After the open timer expires, start a 30-second half-open observation period. Permit one probe in flight, with at most five probe attempts across all instances during that period. Use representative, authorised requests and the five-second deadline. Do not launch a probe after the period ends; cancel any remaining probe then and classify it as inconclusive.

Require three positive health observations to enter the ramp. A completed inconclusive probe may be replaced while time and attempt budget remain. Apply these outcome rules immediately:

  • Qualifying failure: reopen for 90 seconds.
  • Positive health evidence: increment the recovery count, even if a documented request-specific rejection means that particular job failed.
  • Inconclusive or unclassified error: do not increment the count; try another eligible probe within the remaining budget.
  • Credential failure or rate restriction: end probing and apply the corresponding gate.
  • Fewer than three positives when time or attempts run out: reopen for 90 seconds with reason “insufficient recovery evidence.” This is not a claim that the dependency failed.

Safe representative reads may supply probes where authorised. If none are available, low demand can leave recovery unproven; do not create unauthorised actions to escape that state.

Three positives begin the hypothetical ramp:

  1. Stage one: two concurrent calls for at least 30 seconds and until five positive health observations have completed.
  2. Stage two: five concurrent calls for at least 30 seconds and until five new positive health observations have completed.
  3. Normal: restore the 20-call allowance after stage two passes.

The allowances are integer slots: two is 10% of 20; five is 25%. Each stage continues until both its duration and evidence requirements are met. Insufficient traffic holds the stage; inconclusive calls do not satisfy its count. Any qualifying failure reopens for 90 seconds. Other gates still apply.

Archive pre-opening observations when the ramp begins and start a fresh recovery record. During the ramp, its stricter any-qualifying-failure rule replaces the normal rolling threshold. On returning to normal, start a fresh rolling window and consecutive-failure sequence. Keep historical evidence for review, not retripping. Old failures therefore cannot immediately undo successful recovery.

7. Walk through recovery and test the exceptions

Continue the fictional 65% failure example. After the 45-second pause, the first probe is cancelled locally. It supplies no recovery evidence. Three subsequent probes return valid responses within the remaining period. Four attempts were used, so the breaker enters the two-slot ramp.

Stage one has five positive observations after 18 seconds. It remains capped until 30 seconds have elapsed. Stage two reaches 30 seconds with only three positives; it stays at five slots until two more complete. Normal traffic then resumes with fresh failure counters.

If the first five probes instead produced only two positives and three inconclusive outcomes, the breaker would reopen for insufficient evidence. If a probe returned a documented invalid-record rejection, that could count as positive health evidence while the affected job remained rejected.

Test those paths before enforcement, alongside deadline expiry, credential suspension, rate restrictions, queue overflow, and a qualifying failure during each ramp stage. Restart an instance while open to check that the shared probe allowance remains bounded. Observation-only evaluation can expose false trips, but does not prove worker containment works.

8. Measure containment, not just state changes

Capture outbound attempts, suppressed calls, worker occupancy by job type, deferred queue age, expiries, and transitions with their evidence and reason. Separate request completion from dependency-health observations so validation errors do not appear as completed business work.

The useful outcome is observable containment: repeated unsuccessful calls stop occupying workers, unrelated work continues, affected jobs retain truthful statuses, and recovery stays within its allowance. If those outcomes do not improve, inspect worker isolation, queue handling, and retry ownership before merely changing the failure threshold.

Integration reliabilityCircuit breakersAPI operationsRecovery planning

Put the next conversation to work.

Explore Alactic Communications for WhatsApp conversations, campaigns and AI-assisted workflows. Choose the product and plan that fit your team.

Explore Communications