LLMs for mobile engineers, from tokens to your app · Puzzle 5 · 6 of 8

Same question, different answers

Your unit test checks the model's summary against an expected string. It passes, then fails, then passes, with no code change. Here's where the randomness comes from, the knob that turns it down, and why it never reaches zero.

The problem

Your test calls the model with the same review and asserts the summary equals a saved string. It flips between passing and failing. Users also notice: "regenerate" gives a different answer each time. Is the model broken?

See it

The model gives each possible next token a score: Great high, Good medium, Fine low; sampling picks one at random by score; low temperature makes the top token win almost always; high temperature gives the others more chances model GreatGoodFine picked at random, by score low temperature: Great almost always high temperature: more variety
  1. At each step the model scores every possible next token: "Great" likely, "Good" less, "Fine" rarely.

    Step 1 of 4
  2. One is chosen by sampling: at random, weighted by the scores. Usually "Great", not always.

    Step 2 of 4
  3. Low temperature tilts the odds further toward the top token. Predict: what does a high one do?

    Step 3 of 4
  4. It evens them out: "Good" and "Fine" get picked more often, so answers vary more.

    Step 4 of 4

The building block

Puzzle 1 said the model picks the next token. More precisely, at each step it gives every possible next token a probability, a share of 100% saying how likely it is: "Great" 60%, "Good" 30%, "Fine" 10%. Then one is chosen by sampling: at random, weighted by those scores. Usually the likely one; sometimes not. Once a different token is picked, everything after it can go differently.

Temperature controls how adventurous the sampling is:

  • Higher: less likely tokens get picked more often. More varied, more creative, sometimes less accurate.
  • Lower: the most likely tokens win more often. More predictable, more repetitive.

For an app feature that must be consistent, like extracting fields or summarising, use a low temperature. For brainstorming names, a higher one.

But Anthropic's documentation is explicit: even at temperature 0, results aren't fully deterministic (the same input always giving the same output), and identical inputs might produce different outputs. Temperature 0 means "pick the top token", but tiny numeric differences on the provider's servers can still change which token is on top. So tests can't compare exact strings. Test what must be true: valid JSON, the required fields present, length under a limit, no forbidden words. For quality, keep a set of example inputs and check the answers in bulk.

The analogy

A jazz musician playing a standard. Ask for a strict take (low temperature) and you get nearly the same performance every night. Ask for a free one (high temperature) and every night is different. Even the strict take isn't a recording: tiny differences still creep in.

Check yourself

You set temperature to 0. Can you now assert the exact output string in a test?

Not quite. Anthropic's docs say even at 0 the results won't be fully deterministic.

Right. Test properties, not strings.

Not quite. Prompt length doesn't make the output deterministic.

Your app extracts a date and an amount from a receipt photo. Which temperature suits it?

Right. Consistency wants low temperature.

Not quite. It does: higher temperature makes less likely, possibly wrong, tokens more frequent.

Not quite. Varied inputs don't need varied outputs; you want consistency.

What's the difference between sampling and temperature?

Not quite. That's the two swapped. Re-read that part ↑

Right. The choice vs the knob on it.

Sources

  • Anthropic: Glossary, Temperature ("Temperature is a parameter that controls the randomness of a model's predictions during text generation. Higher temperatures lead to more creative and diverse outputs… Lower temperatures result in more conservative and deterministic outputs that stick to the most probable phrasing and answers"; "Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.")

What you now know

At each step the model gives every possible next token a probability, and the next token is chosen by sampling from them. Temperature sets how adventurous that choice is: higher gives more varied text, lower sticks to the most likely tokens. Even at temperature 0 the output isn't fully deterministic, so test model features on properties, not exact strings.