CoursesSystem design from first principles to design review

Puzzle 17 · 18 of 19

When is it worth calling a failing dependency again?

Use failure replies and a quiet interval to control how many calls reach a recovering dependency.

All puzzles in this course

The problem

Request coalescing let two callers share a result, including a failed result. Now suppose storage keeps failing after that shared read. Each new request can occupy a worker until its own deadline. If callers keep arriving, how can the map service give storage a quiet interval and still discover that it has recovered?

See it

Do callers reach storage during and after the recovery probe? Breaker decisions and replies 2 failures Open · 100 ms quiet10 One probe110 115 ms Success · calls resumeFailure · quiet until 215

Scroll sideways to see the full diagram.

  1. The first two responses indicate a dependency failure. The second arrives at time 10.

    Step 1 of 4
  2. The open interval starts at time 10. A caller at 50 is refused locally. Storage sees no call from that caller.

    Step 2 of 4
  3. At 110 one request takes the policy’s single trial slot. A second caller arrives at 111, before the probe replies. Predict its outcome. The axis shows event order; the spacing is schematic.

    A recovery probe entered at 110 and is still unanswered at 111. Should a second caller reach storage?

    Step 3 of 4
  4. The probe succeeds at 115. The next caller at 116 reaches storage.The probe fails at 115. The next caller at 116 is refused. The next probe is eligible at 215.

    Step 4 of 4

The building block

A circuit breaker remembers recent call outcomes. Think of a closed electrical switch allowing current to flow. In the closed state this policy allows calls. Two consecutive qualifying failures open it. While open it refuses new attempts immediately. This policy waits 100 milliseconds after opening before permitting a trial call, known as a probe. During that half-open trial, other requests are refused.

A timeout answers how long one call can wait. A breaker decides whether a new call may start. The probe itself also needs a timeout. A full-lot business refusal can be a healthy response from storage, so define in advance which errors count as dependency failures.

Try a successful and a failed probe

Each time in the code is milliseconds on one local clock. The fixture delivers failure responses at times 5 and 10, opens the breaker at 10, and sends a request at 50. Predict whether storage receives that request. Then compare the row at 111, when the probe is in flight, with the row at 116, when its response has arrived.

if (state === 'open' && time >= nextProbe) state = 'ready';
const allowed = state === 'closed' || state === 'ready';
if (state === 'ready') state = 'probing';

Read the complete runnable code
function testBreaker(probeReply) {
  let state = 'closed';
  let failures = 0;
  let nextProbe = 0;
  const rows = [];
  function attempt(time, name) {
    if (state === 'open' && time >= nextProbe) state = 'ready';
    const allowed = state === 'closed' || state === 'ready';
    if (state === 'ready') state = 'probing';
    rows.push({ time, event: name, state, decision: allowed ? 'call' : 'refuse' });
    return allowed;
  }
  function complete(time, ok) {
    failures = ok ? 0 : failures + 1;
    if (ok) state = 'closed';
    else if (state === 'probing' || failures >= 2) {
      state = 'open';
      nextProbe = time + 100;
    }
    rows.push({ time, event: 'reply', state, decision: ok ? 'success' : 'failure' });
  }

  if (attempt(0, 'first')) complete(5, false);
  if (attempt(6, 'second')) complete(10, false);
  attempt(50, 'early');
  if (attempt(110, 'probe')) {
    attempt(111, 'during probe');
    if (probeReply !== 'missing') complete(115, probeReply === 'success');
  }
  attempt(116, 'next');
  return rows;
}

The caller at 50 is refused locally. The caller at 110 holds the one probe slot. The caller at 111 cannot start a second trial. A successful reply at 115 closes the breaker, allowing the attempt at 116. A failed reply opens it for another 100 milliseconds, until 215. The rows separate attempts from responses so the interval while the probe is unanswered stays visible.

Go deeperWhat does the simple policy leave out?

The fixture orders the first two calls and their replies. In a busy service dozens of older calls might finish after the breaker opens. Associate probes and results with the generation of the breaker state that admitted them. An old success should not close a newer open generation. Each service instance also makes its own decision unless the design explicitly shares state.

If ten instances each admit one probe, how many probes might the dependency see?

Up to ten in that recovery interval. The one-probe rule in this fixture belongs to one instance.

Back to the main puzzle

Try it

Select the missing-reply scenario. The probe begins at 110 and never reports success or failure. Predict the state at 116 and whether the next call reaches storage.

Compare your result

The fixture stays in the probing state and refuses the next call. The experiment has no probe timeout. Describe the deadline and cleanup your real client needs before applying the policy.

Check yourself

What happens to the request at time 50?

Right. The quiet interval lasts until time 110.

Not quite. An open breaker stops new dependency calls.

Why refuse the request at 111?

Not quite. The fixture limits concurrent trials independently of caller deadlines.

Right. The half-open policy reserves one trial slot.

When may new work enter after a successful probe?

Not quite. An unanswered probe has not established recovery.

Right. The reply at 115 changes the state.

Sources

What you now know

A circuit breaker refuses work while open and limits the probes that test recovery.

A missed deadline can release a caller. What if a second driver still needs the underlying read?