Mobile system design, from a packet to an app · Puzzle 10 · 11 of 11

Retry without charging twice

The user taps Pay. The request times out. Did the payment go through? Retrying might charge them twice; not retrying might lose the order. Here's how a key makes retries safe, and how to retry without flooding a struggling server.

The problem

The user taps Pay. Your POST /payments times out after 30 seconds. Maybe the request never arrived. Maybe the server charged the card and the answer got lost on the way back. You can't tell which. Retry, and you might charge twice. Don't, and you might lose a paid order.

See it

The app sends a payment with key k1; the server charges and stores the result, but the answer is lost; the app retries with the same key k1 and the server returns the stored result without charging again; many clients retrying with jitter spread out over time app server pay 20 €, key k1 k1: charged answer lost: timeout retry, key k1 same key stored result, no chargeretries with jitter: spread out
  1. The app sends the payment with an idempotency key, k1, generated once for this tap.

    Step 1 of 4
  2. The server charges the card and stores the result under k1. The answer is lost on the way back: a timeout.

    Step 2 of 4
  3. The app retries with the same key. Predict: does the server charge again?

    Step 3 of 4
  4. No: it sees k1 and returns the stored result. Retries use exponential backoff with jitter, so many clients don't return at once.

    Step 4 of 4

The building block

An operation is idempotent when doing it twice has the same effect as doing it once. HTTP says GET, PUT and DELETE are meant to be: setting a profile name to "Anna" twice leaves it "Anna". POST isn't: two POST /payments can be two payments.

The fix is an idempotency key. Before the first try, the app generates a unique ID, a random UUID, and sends it with the request, for example in an Idempotency-Key header. The server remembers each key and its result:

  • First time it sees the key: it does the work and stores the result.
  • It sees the key again: it doesn't charge again; it returns the stored result.

Now a timeout is safe: retry with the same key. Either the first try never arrived and this one does the work, or it did and this one gets the saved answer. Generate the key once per user action and save it with the pending order, so a retry after the app restarts still uses it. (That's also how puzzle 8's pending writes should sync: each with its own key.) Stripe compares a retry's parameters with the original's and errors if they differ, and may forget keys after 24 hours.

When to retry matters too. If the server is struggling and every client retries at once, the retries pile on. Exponential backoff waits longer each time: 1 s, 2 s, 4 s, with a cap on the wait and on the number of tries. Retry only failures that may pass: timeouts, 5xx errors, and 429 Too Many Requests (honouring its Retry-After header). A 400 won't fix itself. Jitter randomises each wait, so clients that failed together spread out instead of returning in waves.

The analogy

Posting a cheque with a reference number written on it. If you're not sure it arrived, you send another with the same number. The bank sees the number, recognises the duplicate, and cashes only one.

Check yourself

POST /payments timed out. You retry with a new idempotency key. What can happen?

Not quite. Servers don't match by amount; only the key marks a retry.

Not quite. The server recognises retries by the key; a new key looks like a new payment.

Right. Same action, same key.

10,000 clients lost the connection at the same second. Why add jitter to their backoff?

Not quite. That's the idempotency key's job; jitter only spreads the timing.

Not quite. Jitter isn't about speed; it breaks up the waves.

Right. Randomness breaks up the stampede.

What's the difference between an idempotency key and exponential backoff?

Right. Whether it's safe vs when to do it.

Not quite. That's the two swapped. Re-read that part ↑

Sources

What you now know

A timeout doesn't tell you whether the server did the work. An idempotency key, a unique ID the client sends with the request, lets the server recognise a retry and return the first result instead of acting twice. Retry with exponential backoff plus jitter, so clients that failed together don't all retry at the same moment.

That's everything published so far

You've reached the last puzzle of Mobile system design, from a packet to an app. Next: Module 2: Designing a feature end to end, about a feed with pagination and cursors; images and their cache; sync with conflicts; real-time updates (polling, long polling, WebSockets, push); rolling out safely with flags.

New puzzles are announced on LinkedIn and in the learning log feed.

All courses →