Mobile system design, from a packet to an app · Puzzle 9 · 10 of 11

When the connection breaks

Puzzle 0 said TCP resends lost packets. Yet requests still fail after a while, and sometimes you can't tell whether the server did the work. Here's where TCP's promise ends, and the question the app then can't answer.

The problem

The user taps Pay in a lift. After ten seconds, OkHttp throws a SocketTimeoutException. TCP was supposed to resend whatever got lost. So why did it fail, and, the question that matters: was the payment made or not?

See it

The phone sends POST orders; in a lift the connection breaks; after ten seconds a read timeout fires; two stories look the same from the phone: the request was lost, or the server created the order and the answer was lost phone server POST /orders lift: connection breaks 10 s: read timeout order created?lost on the way, or answer lost?
  1. The phone sends POST /orders over an open connection.

    Step 1 of 4
  2. In a lift the connection breaks. TCP can't resend into a connection that's gone.

    Step 2 of 4
  3. After ten seconds, OkHttp gives up: a read timeout. Predict: was the order created?

    Step 3 of 4
  4. You can't tell: the request may have been lost, or the server created the order and only the answer was lost.

    Step 4 of 4

The building block

TCP resends lost packets, but only while the connection exists and both ends can still reach each other. On a phone, that ends all the time: a lift, a tunnel, a switch from Wi-Fi to mobile data. Then the packets have nowhere to go, and TCP can't help.

So the app can't wait forever. OkHttp gives up after a timeout, and there are two kinds you'll meet:

  • Connect timeout: no connection could be made at all. The request never left: safe to try again.
  • Read timeout: the connection was made and the request was sent, but no answer arrived in time.

The read timeout is the dangerous one. It has two possible stories, and from the phone they look identical:

  1. The request was lost on the way: the server never saw it.
  2. The server did the work, and the answer was lost on the way back.

That's the question a timeout can't answer: did it happen? For reading data, it doesn't matter: ask again. For a payment or an order, retrying could do it twice. Puzzle 10 is the fix.

The analogy

Posting a cheque and hearing nothing back. Maybe the post lost your letter; maybe the bank cashed it and the receipt got lost. From your sofa, you can't tell which.

Check yourself

A read timeout fires on POST /orders. What do you know?

Not quite. The server may have answered; only the answer went missing.

Right. A read timeout can't say if it happened.

Not quite. TCP resends packets inside one request; it doesn't send the request twice.

A connect timeout fires. Is it safe to retry a payment request?

Not quite. No connection means nothing was sent; there's nothing to check.

Not quite. With no connection, the request couldn't reach the server.

Right. No connection, nothing sent.

What's the difference between a connect timeout and a read timeout?

Not quite. That's the two swapped. Re-read that part ↑

Right. Never left vs no answer back.

Sources

What you now know

TCP is reliable only while the connection lasts; on a phone, connections break in lifts, tunnels and handovers between Wi-Fi and mobile data. OkHttp gives up after timeouts: connect (no connection made) and read (no answer arrived). A read timeout can't tell you whether the server did the work: the request may have arrived and the answer been lost. Puzzle 10 makes that safe to retry.