LLMs for mobile engineers, from tokens to your app · Puzzle 1 · 2 of 8

One token at a time

You send a prompt and get a paragraph back. It looks like the model writes the answer the way you would. It doesn't. Here's the loop that produces every word, and the unit everything is measured and billed in.

The problem

Your app sends "Summarise this review" with a 300-word review, and gets a 60-word summary back. The API bill doesn't count words at all. The response time grows mostly with the length of the answer, much less with the question. What is the model actually doing?

See it

The prompt is cut into tokens; the model reads all tokens so far and predicts one next token; that token is appended and the model runs again; each new token is one more step Summarisethisreview model Great appended, then run again phoneone token per step
  1. Your prompt is cut into tokens: whole words, pieces of words. "Summarise" might become two, depending on the model.

    Step 1 of 4
  2. The model reads every token so far and picks one next token: next-token prediction.

    Step 2 of 4
  3. That token is appended to the text, and the model runs again on the longer text. Predict: what does that mean for long answers?

    Step 3 of 4
  4. Each new token is one more step. A 300-token answer is 300 runs, which is why long answers take longer.

    Step 4 of 4

The building block

A language model doesn't see letters or words. Text is first cut into tokens: common words are one token, rarer words are split into pieces, and emoji or unusual characters can take several. On Anthropic's earlier Claude models, one token was about 3.5 English characters; other languages and other models differ. The API's token counting tells you exactly.

Then the model does one thing, over and over: next-token prediction. Given all the tokens so far, the prompt plus whatever it has written, it picks the next token. That token is appended, and the model runs again on the slightly longer text. It stops when it picks an end-of-answer token, or hits the maximum length you set (the next puzzle shows where). (The prompt itself is read in one go; only the answer is written step by step. That's why a long answer costs more time than a long question.)

Three consequences you'll feel in an app:

Because
A long answer takes longer than a short oneeach token is one more run of the model
Limits and prices are in tokensthat's the unit the model reads and writes
The same text costs different amounts in different languagesthe split into tokens differs

The analogy

A typist who can only see what's already on the page, and adds one word at a time, each time rereading the page to decide the next. The page is never planned in advance: every word is chosen when it's time.

Check yourself

Why does a 300-token answer take longer than a 30-token one?

Not quite. It doesn't reread for sending; it generates one token per step.

Right. One token per step.

Not quite. A few kilobytes cost little; the time is in generating each token.

The same text in Polish costs more tokens than in English. Why?

Right. Different languages, different splits.

Not quite. Same meaning, more tokens: the split differs.

Not quite. The price is per token; the text just becomes more tokens.

What's the difference between a token and a word?

Not quite. Common words are one token, but tokens can also be parts of words. Re-read that part ↑

Right. The model's unit, not the dictionary's.

Sources

  • Anthropic: Glossary ("Tokens are the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes"; earlier models: "a token represents approximately 3.5 English characters"; "autoregressive language models… are pretrained to predict the next word, given the previous context of text")

What you now know

A language model reads and writes tokens: words, pieces of words, characters. It produces an answer by next-token prediction: given all the text so far, pick one next token, append it, and run again, until it stops. Limits, latency and price are all counted in tokens, and longer answers take longer because each token is one more step.