LLMs for mobile engineers, from tokens to your app · Puzzle 3 · 4 of 8
Why answers stream
Your chat screen shows a spinner for eight seconds, then the whole answer at once. Users think it's broken. The model had the first words after half a second. Here's how to show them, and what changes in your networking code.
The problem
Your app calls the model and waits for the full answer: eight seconds of spinner, then a wall of text. In the model's own playground, words start appearing almost at once. Same model, same prompt. What does the playground do that your app doesn't?
See it
The app as written waits for the whole answer: eight seconds of spinner, then all the text at once.
Step 1 of 4With streaming, one HTTP response stays open, and each new piece arrives as one of the server-sent events.
Step 2 of 4The first words show after half a second: the time to first token. Predict: what if the user leaves mid-answer?
Step 3 of 4Cancel the stream: every token still being generated costs money and battery, read or not.
Step 4 of 4
The building block
Puzzle 1 showed the model produces an answer one token at a time. So the first tokens exist long before the last one. If the API waits for the whole answer, your user waits for all of it.
With streaming, the API sends each new piece as soon as it's generated. Anthropic's API does this when you set "stream": true: one HTTP response stays open, and pieces arrive as server-sent events, small text events on the same connection.
Two numbers now matter:
- Time to first token: how long until the first words appear. This is what the user feels.
- Total time: about the same as before. Streaming doesn't make the model faster; it stops hiding its progress.
On Android, the stream is a long-lived request, so it changes your code more than a normal call:
- Run it on a coroutine that the screen's ViewModel owns, and append each piece to the UI state as it arrives.
- Cancel it when the user leaves or taps stop: you pay for every token generated, wanted or not. If the stream goes through your own server (puzzle 6), the server must cancel its request to the model too, or generation and billing carry on.
- OkHttp has an SSE add-on (
okhttp-sse), and Anthropic publishes a Java SDK that streams. - A dropped connection mid-answer is normal on mobile, in a lift or a tunnel: keep what arrived, show it, and offer to retry.
The analogy
A friend reading you a letter over the phone. You could wait until they've read it all to themselves and then hear a summary, or hear each sentence as they read it. The letter takes the same time; one way you're listening from the first line.
Check yourself
Sources
- Anthropic: Streaming messages ("When creating a Message, you can set \"stream\": true to incrementally stream the response using server-sent events (SSE).")
- Anthropic: Glossary, TTFT ("Time to first token (TTFT) is a performance metric that measures the time it takes for a language model to generate the first token of its output after receiving a prompt. It is an important indicator of the model's responsiveness")
What you now know
Because the model writes one token at a time, the API can stream: send each piece as it's generated, over one open HTTP response, as server-sent events. Time to first token, not total time, is what makes an app feel fast. On mobile, the stream is a long-lived request: keep it on a background coroutine, append to UI state as events arrive, and cancel it when the user leaves.
Answer every question above correctly to unlock the next puzzle.
Something wrong or unclear on this page? Tell me: a puzzle that didn't click is a bug in the puzzle.