CoursesSystem design from first principles to design review

Practice chapter 3 of 5

Rebuild a concert seating map from reservation events

Commit a reservation fact before announcing it

The booking database owns the right to sell. The seating map is a projection: a derived copy optimized for showing availability. In the same transaction as the booking, write an outbox entry containing the show and seat, the event ID, the next seat version and the change. A relay publishes committed entries. The live outbox is a delivery mechanism; a rebuild also requires a retained history or a consistent source snapshot.

Worked example

Show 42 starts with seat B12 free at version 0. One transaction holds it and writes E7, version 1, delta −1 free. A crash before commit leaves neither change. A crash after commit leaves the hold and a recoverable pending publication. The authoritative checkout rejects another hold even while the map still shows B12 as free.

Try it

The relay publishes E7, crashes before marking the row sent and later publishes E7 again. What should be counted once and what may legitimately be counted twice?

Compare your answer

Count one logical event identity and one acquired right. Two delivered messages and two processing attempts are possible. Carry E7 unchanged across publication retries so the consumer can recognize the repetition.

Source · AWS · Transactional outbox

Make a repeated delivery harmless at the receiver

A broker position tells a consumer where to resume. An event ID identifies the business fact. If recording a consumed position and updating a separate database are independent writes, crashing between them creates either replay or loss. Commit the projection and its processed-event marker together; advance the broker position only after durable processing or durable handoff.

Worked example

Starting at 100 free seats, apply E7 at position 400: write 99 and insert processed(E7) in one local transaction. Stop after it commits but before acknowledging position 400. The broker delivers 400 again; the unique marker prevents a second decrement. Even if the relay appended E7 again at position 401, the same marker still protects the count.

Try it

Why is a marker written first in its own transaction unsafe?

Compare your answer

If the process stops after saving the marker but before subtracting one, replay skips E7 forever and the display says 100. Run both crash schedules and inspect the count, marker and checkpoint. A harmless SQL rollback protects only changes made inside that transaction.

Source · Apache Kafka · Message delivery semantics

Detect a version gap without blocking unrelated seats

For each seat, give related events one routing key. Kafka preserves append order within a partition, but an upstream relay can append a later business version first. The consumer compares the event version with the last applied seat version. A delta for version n requires the predecessor n−1. Missing versions need repair or durable staging; blindly applying the latest delta loses information.

Worked example

B12 emits E7 hold v1 (−1 free), E8 release v2 (+1), E9 hold v3 (−1). Arrival order is v1, v3, v2. After v1 the aggregate is 99. Persist v3 as waiting, apply v2 to reach 100, then apply v3 to return to 99. Maintain B12’s watermark at v1 until v2 is applied. An unrelated seat may still progress if its events are independent.

Try it

If you pause the entire partition at the v3 offset and v2 is a later offset in that same partition, how do you escape the wait?

Compare your answer

Persist v3 as pending before moving past its offset, or fetch the missing authoritative seat state through a repair path. Resume with bounded backlog and age checks. Two different IDs claiming the same seat version indicate a producer conflict requiring investigation.

Source · Apache Kafka · Partitions and log order

Keep old events readable after a schema change

An Avro reader uses the writer’s schema to resolve fields against the reader’s schema. Adding a field with a reader default can allow a new reader to decode an old event. Field compatibility is only one test: the application must still interpret the old business meaning correctly. Keep schema version distinct from the per-seat business version.

Worked example

The new event schema adds bookingChannel with a string default of unknown. Old E7 decodes under the new reader with unknown, while a new event can record kiosk. An old reader can ignore this unused extra field when reading the compatible new record. Replaying E7 must still subtract exactly one free seat. Never silently label a historical sale as mobile simply to fill the new field.

Try it

An engineer changes deltaFree to mean “free seats after this change” but leaves its type as integer. Would a schema compatibility test catch the danger?

Compare your answer

A compatible integer type does not distinguish delta from absolute count. Introduce a new event shape or explicit semantic version and conversion rule. Run mixed-history fixtures against both decoders and compare the resulting per-seat states and totals.

Source · Apache Avro · Schema resolution

Repair the map while bookings continue

Rebuild into a separate projection generation so the active display keeps serving. Record a consistent snapshot and its corresponding stream position for every partition, or replay a complete retained history from its beginning. Apply later changes and validate at a common comparison boundary. Keep the rebuild isolated from real email, payment and refund consumers.

Worked example

At the chosen boundary the authoritative seat rows contain 98 free, one held and one sold: 100 seats total. The old map also reports 98 free, but it labels B12 sold and C15 held while the source has the opposite. Count equality hides the error. The rebuilt generation must agree on seat ID, state and version, plus section totals. Catch up to the cutover boundary, switch readers and retain the old generation briefly for rollback.

Try it

Your oldest retained message begins at position 900; the only trusted snapshot for that partition covers through 850. Can replay certify the repaired map?

Compare your answer

Positions 851 through 899 are unaccounted for. Obtain a newer consistent source snapshot with a matching resume position or recover the missing archive. Record what the rebuild can actually verify. A quiet consumer and zero current lag establish catch-up only from the chosen start, not completeness of earlier history.

Source · Azure Architecture Center · Event sourcing and projection rebuilds