CoursesSystem design from first principles to design review
Practice chapter 1 of 5
Guarantees under load and partial failure
What did the replica actually acknowledge?
A primary records changes in its write-ahead log. A standby receives log records and replays them into its own readable database. Receiving, durably flushing and replaying a record are distinct milestones. With asynchronous replication a primary can acknowledge a write before the standby reaches any of them. Choose the milestone based on the promise you make to the caller and which failures you intend to survive.
Worked example
Ada confirms seat B12 at log position 820. A standby has received through 820 but applied only through 818. A read routed there can still show B12 as available. The booking service must arbitrate the next purchase at the authoritative writer. For a read-your-write receipt, route that read to the primary or wait for an eligible replica to apply 820, within a deadline.
Try it
The primary stops before the standby receives record 820. The phone has a successful reply. Can the new primary reconstruct the sale from its own data?
Compare your answer
The promoted standby can lack the acknowledged sale. Returning to a primary for the read fixes ordinary lag only while the original record survives. To promise survival of a specified primary failure, configure the required synchronous acknowledgements and an eligible promotion policy, then rehearse that failure. Decide how the system responds when the required acknowledger is unavailable.
Source · PostgreSQL · Standby servers
How many failures can a user journey afford?
A service level indicator is an observed ratio or measurement. Define the eligible requests, what counts as good and the time window before choosing a target. For a reservation journey, a correct full-lot response may be good. A timeout can be bad even if the server later commits. Count from a place that can observe the promised outcome.
Worked example
Assume one million eligible attempts in 30 days and a 99.9% success target. The budget is 1,000 bad attempts. A release fails 240 of 8,000 attempts. It spent 24% of the monthly budget while handling 0.8% of the monthly traffic. Its 3% error rate is 30 times the permitted 0.1% rate.
Try it
The busiest hour improves, but customers on one mobile carrier still time out. Which evidence would justify closing the incident?
Compare your answer
Split eligible attempts by carrier and client version, inspect sample size, then recompute the same journey indicator before and after the fix. Weight the overall result by actual attempt counts. A healthy server-only ratio cannot establish that the affected phones received timely confirmations.
Source · Google SRE · Implementing SLOs
Where should retries spend the request budget?
A timeout limits how long one caller waits. It does not cancel every operation that caller started. A retry can therefore add concurrent work to a dependency that is already slow. Set a deadline for the whole journey, assign one layer responsibility for retries and use stable operation identities for writes with repeatable effects.
Worked example
A phone calls a gateway, which calls a service, which calls a database-facing dependency. If all three calling layers make up to three attempts, one user action can trigger 27 attempts at the deepest boundary. With a 500 ms deadline, 220 ms spent on the first call and 100 ms of backoff, at most 180 ms remain for another call and returning the result.
Try it
The downstream call usually needs 200 ms. Would jitter make another attempt fit? What happens to an unknown payment outcome?
Compare your answer
Jitter spreads arrivals across time. It supplies no extra time or capacity. Stop when the remaining deadline cannot support the chosen call budget. Record the unresolved payment identity and reconcile its status through the documented provider contract. Count physical attempts separately from user actions.
Source · AWS Builders' Library · Timeouts, retries and backoff