CoursesBackend engineering from first request to production

Puzzle 4 · 5 of 18

Alive, ready, or serving the wrong thing?

Diagnose a storage failure and rehearse recovery.

All puzzles in this course

The problem

The process is running and the phone still gets errors. Should the operator restart the program, replace its image or investigate storage first?

See it

Alive, ready, or serving the wrong thing? Live 200 Ready 503 Restore table Write again

Scroll sideways to see the full diagram.

  1. The process still receives requests.

    Step 1 of 4
  2. A query cannot read the expected table.

    Step 2 of 4
  3. The operator restores the original name.

    Step 3 of 4
  4. A real write tests recovery.

    Step 4 of 4

The building block

A readiness probe asks whether an instance can currently serve its expected work. The live route only proves the handler can answer. The ready route reads from the bookmark table. In the disposable incident drill, rename the table while the process runs. Live still returns 200. Ready and bookmark writes return 503. The server log points to storage. Restore the table and retry a real write. A successful read probe alone cannot prove that a disk accepts writes. For a code regression with a compatible database, practice starting the previous image and checking the saved row.

Read the incident as a sequence of observations

At 10:00, a bookmark write succeeds. After a table rename, the same write fails and the readiness response changes from 200 to 503. The live response stays at 200. Those observations point to the work performed by the ready route after it reaches storage. Compare two branches from that failure. Before restarting, restoring the original table name makes river visible again. Restarting instead runs the startup routine: it creates an empty bookmarks table because that name is free. Readiness now returns 200 and a new write can succeed, while river is still in the renamed table. Check both identities. Once new writes exist in the replacement table, preserve both tables and reconcile their contents; a blind rename is no longer a complete recovery.

Try it

Run the browser experiment

Optional local lab

Run the incident drill. Write a short incident report and rehearse the rollback. Include one condition under which reverting the image would be insufficient.

Open the local service source files

Check yourself

The read probe succeeds after recovery. What should you verify next?

Not quite. A running process can still fail to write. Re-read that part ↑

Right. A process can answer a simple probe while its storage path fails.

After restarting, readiness returns 200 but river is missing from the list. Where should you look?

Right. A successful probe checked the replacement table.

Not quite. Startup created a fresh table under the expected name. Re-read that part ↑

Why keep the bookmark written before the incident as a test case?

Not quite. A successful new write can coexist with missing old rows. Re-read that part ↑

Right. A successful new write can coexist with missing old rows.

Sources

What you now know

A process can answer a simple probe while its storage path fails.