LLMs for mobile engineers, from tokens to your app · Puzzle 5 · 6 of 8
Same question, different answers
Your unit test checks the model's summary against an expected string. It passes, then fails, then passes, with no code change. Here's where the randomness comes from, the knob that turns it down, and why it never reaches zero.
The problem
Your test calls the model with the same review and asserts the summary equals a saved string. It flips between passing and failing. Users also notice: "regenerate" gives a different answer each time. Is the model broken?
See it
At each step the model scores every possible next token: "Great" likely, "Good" less, "Fine" rarely.
Step 1 of 4One is chosen by sampling: at random, weighted by the scores. Usually "Great", not always.
Step 2 of 4Low temperature tilts the odds further toward the top token. Predict: what does a high one do?
Step 3 of 4It evens them out: "Good" and "Fine" get picked more often, so answers vary more.
Step 4 of 4
The building block
Puzzle 1 said the model picks the next token. More precisely, at each step it gives every possible next token a probability, a share of 100% saying how likely it is: "Great" 60%, "Good" 30%, "Fine" 10%. Then one is chosen by sampling: at random, weighted by those scores. Usually the likely one; sometimes not. Once a different token is picked, everything after it can go differently.
Temperature controls how adventurous the sampling is:
- Higher: less likely tokens get picked more often. More varied, more creative, sometimes less accurate.
- Lower: the most likely tokens win more often. More predictable, more repetitive.
For an app feature that must be consistent, like extracting fields or summarising, use a low temperature. For brainstorming names, a higher one.
But Anthropic's documentation is explicit: even at temperature 0, results aren't fully deterministic (the same input always giving the same output), and identical inputs might produce different outputs. Temperature 0 means "pick the top token", but tiny numeric differences on the provider's servers can still change which token is on top. So tests can't compare exact strings. Test what must be true: valid JSON, the required fields present, length under a limit, no forbidden words. For quality, keep a set of example inputs and check the answers in bulk.
The analogy
A jazz musician playing a standard. Ask for a strict take (low temperature) and you get nearly the same performance every night. Ask for a free one (high temperature) and every night is different. Even the strict take isn't a recording: tiny differences still creep in.
Check yourself
Sources
- Anthropic: Glossary, Temperature ("Temperature is a parameter that controls the randomness of a model's predictions during text generation. Higher temperatures lead to more creative and diverse outputs… Lower temperatures result in more conservative and deterministic outputs that stick to the most probable phrasing and answers"; "Even with temperature set to 0, the results will not be fully deterministic, and identical inputs might produce different outputs across API calls.")
What you now know
At each step the model gives every possible next token a probability, and the next token is chosen by sampling from them. Temperature sets how adventurous that choice is: higher gives more varied text, lower sticks to the most likely tokens. Even at temperature 0 the output isn't fully deterministic, so test model features on properties, not exact strings.
Answer every question above correctly to unlock the next puzzle.
Something wrong or unclear on this page? Tell me: a puzzle that didn't click is a bug in the puzzle.