CoursesSystem design from first principles to design review

Puzzle 18 · 19 of 19

If one driver gives up, whose read should stop?

Give two callers different deadlines and trace what survives the first timeout.

All puzzles in this course

The problem

The circuit breaker currently allows calls. Ada and Ben ask the same worker for the west map. Request coalescing gives them one pending storage read. Ada can wait 400 milliseconds. Ben can wait six seconds. Storage has not replied when Ada's time runs out. When Ada gets a timeout, is Ben still waiting? What must happen before his six-second deadline for him to receive the map?

See it

Three lifetimes during a shared map read One pending read for west AdaBenStorage StartDeadlineReply 504 200 Result to Ben

Scroll sideways to see the full diagram.

  1. Ada and Ben have joined the same lookup. The three lines show each caller's wait and the underlying storage work.

    Step 1 of 4
  2. Ada's deadline expires before storage replies. Her line ends at the cross. Ben still has time left. Predict what he can receive.

    Ada's wait has expired. Ben's deadline has not. Storage now returns the west map. Which reply can Ben still receive?

    Step 2 of 4
  3. Storage completes once. Ben's wait ends with a successful reading. Ada's earlier timeout stays a timeout.

    Step 3 of 4
  4. One result served the remaining waiter. If the callers had used different service instances, both could still receive these replies even though storage did twice the work.

    Step 4 of 4

The building block

Separate three lifetimes: Ada's wait, Ben's wait and the storage operation. A single storage read can supply both callers. Each caller waits for whichever happens first: its own deadline or the shared result. When Ada's deadline wins, the service can send her a timeout. Ben still holds a reference to the shared work.

Think of two people waiting for the same printed train timetable. Ada leaves to catch a bus. Ben is still at the counter, so the clerk keeps printing. If expensive work should stop when everyone leaves, the clerk needs to know how many people remain. That is a separate policy.

Run the two callers

The browser starts both waits and releases the deadline and storage result in a controlled order. It models the deadline as an explicit event so the result does not depend on how fast your phone is. Before pressing Run scenario, predict Ben's reply if storage succeeds before his deadline and how many storage reads will have started. Then run both callers on one instance. Then change only the routing so each caller reaches a different instance. Predict which column changes. The reads column is cumulative, including the fresh lookup in the last row.

if (!pending.has(lot)) {
  reads++;
  const work = storageReady.then(() => '200 ' + lot);
  pending.set(lot, work.finally(() => pending.delete(lot)));
}
return Promise.race([pending.get(lot), deadline]);

Read the complete runnable code
async function compareWaiters(split) {
  let releaseStorage, expireCaller;
  let reads = 0;
  const storageReady = new Promise(resolve => { releaseStorage = resolve; });
  const shortDeadline = new Promise(resolve => { expireCaller = resolve; });
  const noDeadlineInThisTrace = new Promise(() => {});

  function makeInstance() {
    const pending = new Map();
    return function read(lot, deadline) {
      if (!pending.has(lot)) {
        reads++;
        const work = storageReady.then(() => '200 ' + lot);
        pending.set(lot, work.finally(() => pending.delete(lot)));
      }
      return Promise.race([pending.get(lot), deadline]);
    };
  }

  const first = makeInstance();
  const second = split ? makeInstance() : first;
  const short = first('west', shortDeadline);
  const long = second('west', noDeadlineInThisTrace);
  expireCaller('504 timeout');
  const shortReply = await short;
  const rows = [{ event: 'deadline', short: shortReply, long: 'waiting', reads }];
  releaseStorage();
  const longReply = await long;
  rows.push({ event: 'storage replies', short: shortReply, long: longReply, reads });
  const later = await second('west', noDeadlineInThisTrace);
  rows.push({ event: 'later lookup: ' + later, short: 'finished', long: 'finished', reads });
  return rows;
}

The code first looks up the pending work for the lot. The last line creates a separate wait for each caller. Winning that race does not cancel the other promise. Each instance has its own list of pending operations. A later call after cleanup starts a new read because this fixture has no completed-answer cache.

Try it

Does this outcome also occur across real HTTP connections? Save the local experiment as waiters.py and run the command below with Python 3. The script starts temporary HTTP servers on your own computer. Each service instance has its own pending table. They run in the same Python process and share an executor for the storage calls.

python3 waiters.py

The script holds the storage response until both driver requests have entered their handlers and the short wait has ended. The following lines are the decision in the handler. The timeout belongs to the caller waiting on the future, the object holding the eventual answer. The storage worker continues running. The status is 504 Gateway Timeout because the map service is acting as a gateway: it answers for a storage read it is waiting on.

try:
    return result.result(timeout=0.4 if budget == "short" else 6)
except concurrent.futures.TimeoutError:   # the same class as TimeoutError from Python 3.11
    return 504, "caller deadline"
Compare your result

On one instance, the server returns 504 to Ada and 200 to Ben after one overlapping storage read. Across two instances the driver replies stay the same, but storage receives two reads. Both runs also verify a successful fresh lookup after cleanup. The difference comes from the location of the pending table.

What if both drivers leave? This implementation still lets an already-started storage read finish. To justify canceling it, count remaining waiters under the same ownership rule as registration and check that storage actually stopped the work. A client timeout alone cannot establish that the underlying work stopped.

Check yourself

What expires when Ada's time runs out?

Right. A caller can stop waiting while shared work is pending.

Not quite. Ben has his own deadline and still needs the read.

What changes after routing the callers to separate instances?

Not quite. The second instance can still return the released storage result. Re-read that part ↑

Right. Each instance has an independent pending table.

Why does the later lookup start fresh work?

Not quite. This experiment keeps only unfinished reads. Re-read that part ↑

Right. It deliberately has no completed-answer cache.

Sources

What you now know

One caller can stop waiting while a shared operation still serves another caller.

In the final project, decide which caller may stop waiting, which operation continues and what evidence the next attempt must consult.

Apply what you learned

Continue with worked examples and practice cases. When you reach the final project, make your own decisions before comparing your evidence with the worked review.

Continue to Guarantees under load and partial failure → Start the final project →