CoursesBackend engineering from first request to production
Puzzle 4 · 5 of 18
Alive, ready, or serving the wrong thing?
Diagnose a storage failure and rehearse recovery.
All puzzles in this course
The problem
The process is running and the phone still gets errors. Should the operator restart the program, replace its image or investigate storage first?
See it
Scroll sideways to see the full diagram.
The process still receives requests.
Step 1 of 4A query cannot read the expected table.
Step 2 of 4The operator restores the original name.
Step 3 of 4A real write tests recovery.
Step 4 of 4
The building block
A readiness probe asks whether an instance can currently serve its expected work. The live route only proves the handler can answer. The ready route reads from the bookmark table. In the disposable incident drill, rename the table while the process runs. Live still returns 200. Ready and bookmark writes return 503. The server log points to storage. Restore the table and retry a real write. A successful read probe alone cannot prove that a disk accepts writes. For a code regression with a compatible database, practice starting the previous image and checking the saved row.
Read the incident as a sequence of observations
At 10:00, a bookmark write succeeds. After a table rename, the same write fails and the readiness response changes from 200 to 503. The live response stays at 200. Those observations point to the work performed by the ready route after it reaches storage. Compare two branches from that failure. Before restarting, restoring the original table name makes river visible again. Restarting instead runs the startup routine: it creates an empty bookmarks table because that name is free. Readiness now returns 200 and a new write can succeed, while river is still in the renamed table. Check both identities. Once new writes exist in the replacement table, preserve both tables and reconcile their contents; a blind rename is no longer a complete recovery.
Try it
Optional local lab
Run the incident drill. Write a short incident report and rehearse the rollback. Include one condition under which reverting the image would be insufficient.
Check yourself
Sources
What you now know
A process can answer a simple probe while its storage path fails.
Answer every question above correctly to unlock the next puzzle.
Something wrong or unclear on this page? Tell me: a puzzle that didn't click is a bug in the puzzle.