LLMs for mobile engineers, from tokens to your app · Puzzle 3 · 4 of 8

Why answers stream

Your chat screen shows a spinner for eight seconds, then the whole answer at once. Users think it's broken. The model had the first words after half a second. Here's how to show them, and what changes in your networking code.

The problem

Your app calls the model and waits for the full answer: eight seconds of spinner, then a wall of text. In the model's own playground, words start appearing almost at once. Same model, same prompt. What does the playground do that your app doesn't?

See it

Without streaming the app shows a spinner until the whole answer arrives after eight seconds; with streaming each token arrives as a small event over one open response, so words appear after half a second; leaving the screen cancels the stream chat screen model 8 s of spinner one open response Great first words after 0.5 s user leaves: cancel
  1. The app as written waits for the whole answer: eight seconds of spinner, then all the text at once.

    Step 1 of 4
  2. With streaming, one HTTP response stays open, and each new piece arrives as one of the server-sent events.

    Step 2 of 4
  3. The first words show after half a second: the time to first token. Predict: what if the user leaves mid-answer?

    Step 3 of 4
  4. Cancel the stream: every token still being generated costs money and battery, read or not.

    Step 4 of 4

The building block

Puzzle 1 showed the model produces an answer one token at a time. So the first tokens exist long before the last one. If the API waits for the whole answer, your user waits for all of it.

With streaming, the API sends each new piece as soon as it's generated. Anthropic's API does this when you set "stream": true: one HTTP response stays open, and pieces arrive as server-sent events, small text events on the same connection.

Two numbers now matter:

  • Time to first token: how long until the first words appear. This is what the user feels.
  • Total time: about the same as before. Streaming doesn't make the model faster; it stops hiding its progress.

On Android, the stream is a long-lived request, so it changes your code more than a normal call:

  • Run it on a coroutine that the screen's ViewModel owns, and append each piece to the UI state as it arrives.
  • Cancel it when the user leaves or taps stop: you pay for every token generated, wanted or not. If the stream goes through your own server (puzzle 6), the server must cancel its request to the model too, or generation and billing carry on.
  • OkHttp has an SSE add-on (okhttp-sse), and Anthropic publishes a Java SDK that streams.
  • A dropped connection mid-answer is normal on mobile, in a lift or a tunnel: keep what arrived, show it, and offer to retry.

The analogy

A friend reading you a letter over the phone. You could wait until they've read it all to themselves and then hear a summary, or hear each sentence as they read it. The letter takes the same time; one way you're listening from the first line.

Check yourself

Streaming is turned on. What gets faster?

Not quite. The bytes are about the same; only when they're shown changes.

Not quite. The model writes at the same speed; you just see it sooner.

Right. Sooner, not faster.

The user leaves the chat screen mid-answer. Why cancel the stream?

Right. Unread tokens are still paid for.

Not quite. They can; you cancel because the answer is no longer wanted.

Not quite. It wouldn't crash if the ViewModel owns it; the waste is the point.

What's the difference between time to first token and total time?

Not quite. That's the two swapped. Re-read that part ↑

Right. Start vs end.

Sources

  • Anthropic: Streaming messages ("When creating a Message, you can set \"stream\": true to incrementally stream the response using server-sent events (SSE).")
  • Anthropic: Glossary, TTFT ("Time to first token (TTFT) is a performance metric that measures the time it takes for a language model to generate the first token of its output after receiving a prompt. It is an important indicator of the model's responsiveness")

What you now know

Because the model writes one token at a time, the API can stream: send each piece as it's generated, over one open HTTP response, as server-sent events. Time to first token, not total time, is what makes an app feel fast. On mobile, the stream is a long-lived request: keep it on a background coroutine, append to UI state as events arrive, and cancel it when the user leaves.