CoursesSystem design from first principles to design review
Practice chapter 4 of 5
Follow a record through Apache Kafka
Where does a record live?
A producer sends a record to a named topic. A topic is split into partitions, each an ordered log. A broker stores log data. An offset identifies a position within one partition. Reading a record leaves it available for other readers until the retention policy removes it.
Worked example
A shipment service writes “packed” at partition 0, offset 18 and “label printed” at partition 1, offset 18. A receipt containing only 18 is ambiguous. A receipt containing the topic, partition and offset identifies which entry to inspect.
Try it
A consumer sees offset 51 in partition 0 and offset 47 in partition 1. Which happened first? How would you preserve the order in which a single producer records changes for shipment S?
Run the offset experiment on a disposable local broker. Follow the JVM steps in the Kafka 4.1 quick start to start one broker on localhost:9092. Save the replay script, open another terminal in the extracted Kafka directory and run bash /path/to/replay.sh using your saved path. The script creates a temporary single-partition topic, writes packed and shipped, reads offsets 0, 1 and 0, and deletes the topic. Predict all three values first.
Compare your answer
The two offsets cannot answer the timing question. Route the changes for S to one partition using a stable shipment key and partition mapping, with producer idempotence enabled to preserve order across retries. The partition gives an append order. If independent producers can submit stale business changes, include a shipment revision and validate it.
You should read packed, shipped, packed. The third read visits the first record again. This exercise addresses one partition directly. Group checkpoints are the subject of the next lesson.
Who owns the next offset?
Consumers that share a group divide the work. In a standard consumer group, a partition has one assigned reader at a time. A separate group can read the same log for a different purpose. Each group can checkpoint its next position for each partition so another member can resume.
Worked example
Billing and analytics each need every shipment. Give them separate groups. Billing has three members and two partitions. One member has no partition assignment. If an active member leaves, the group can assign its partition to a remaining member.
Try it
Partition 0 has records 40, 41 and 42. Billing finishes 40 and 41 and commits 42. Analytics commits 40. Both groups restart before reading again. Which record does each fetch first, assuming all three are retained?
Compare your answer
Billing resumes at 42 and analytics at 40. Rewinding analytics does not rewind billing. Adding a fourth billing member cannot divide either of the two partitions into smaller assignments.
What survives a crash between SQL and Kafka?
Imagine a consumer that updates a SQL total and then records its next Kafka offset. These writes belong to different systems. A crash between them can make Kafka deliver a record that already changed the total. Moving the checkpoint earlier opens a window where an unapplied change is skipped.
Worked example
Shipment event E7 adds 3 to a total of 10. SQL commits 13, then the consumer dies before committing its offset. On retry, a plain increment yields 16. Instead, insert E7 into a table with a unique event ID and increment only when the insert succeeds. In PostgreSQL, use INSERT ... ON CONFLICT DO NOTHING RETURNING id and increment only if a row was returned. Perform both steps in one SQL transaction.
Try it
Crash once after inserting E7 but before incrementing, then once after SQL commits but before the offset commits. Predict the eventual total in both runs. Why would two separate SQL transactions break the first case?
Compare your answer
An uncommitted insert rolls back with the increment. Retry can claim E7 and reach 13. After a committed increment the stored ID prevents a second effect and the checkpoint can advance. A separately committed ID could suppress the only increment. Kafka transactions can pair Kafka output with consumed offsets; an external SQL write needs its own cooperation.
Can a compacted topic rebuild yesterday?
Compaction lets Kafka discard older values for a key while preserving the positions of records it keeps. This serves a reader rebuilding current keyed state. An audit asks a different question: which changes happened, including intermediate ones?
Worked example
A price key appears at offsets 20, 24 and 29 with values 12, 15 and 13. After cleaning, a fresh reader might see only offset 29 for that key. It can reconstruct the current price of 13, but cannot infer the peak of 15. Cleaning runs asynchronously, so an earlier scan may still see all three.
Try it
An old price cache was offline through a deletion. A record with a null value marks the deletion, but its retention window has elapsed. Is replaying new records into that cache sufficient? Design a recovery check.
Compare your answer
The cache may keep a stale price after missing the deletion marker. Rebuild into an empty store from a suitable complete source, catch up to a recorded position for every partition and compare keys and values against an authoritative snapshot at a compatible boundary before switching readers.