CoursesLLMs for mobile engineers, from tokens to your app
Puzzle 9 · 10 of 10
Which step failed the answer?
Keep the same help question and find whether the problem happened during search or writing.
All puzzles in this course
The problem
A customer asks when a card refund will appear. The answer says “one day.” The article says “five business days.” Which system component deserves the bug report?
See it
Scroll sideways to see the full diagram.
The labeled case points to the refund article.
Step 1 of 4The application records which articles the search returns.
Step 2 of 4A model can still make an unsupported claim after seeing the right article.
Step 3 of 4Score selection and evidence support separately.
Step 4 of 4
The building block
A labeled test set lists questions and the evidence or outcome expected for each. Use a small set with literal questions, paraphrases, and questions your documents cannot answer. Save it before changing the application.
Hit at 3 counts how many questions have the expected document among the first three search results. In the previous lab, TF-IDF put the expected article first for seven out of twelve and among the first three for nine. MiniLM put it first for all twelve on this collection.
An evidence check asks whether the answer's claims are supported by the text it received. Run it separately. In this example:
| Evidence provided | Answer observed | Diagnosis |
|---|---|---|
| An article about ending a subscription | One day | Retrieval missed the refund article. |
| The refund article saying five business days | One day | The answer contradicts available evidence. |
| No article that states a refund window | I couldn't find a refund window in the help articles. | The answer acknowledges missing evidence. |
The lab measured the first stage only. The three rows above are worked examples of the second stage, not results from a model run. If you compare two chat tools later, give them the same allowed articles and score their claims against those articles.
Go deeperDesign a fair agent comparison
One agent can search all articles and another sees only five. Can you attribute an answer difference to the model?
Not from this comparison. Freeze the corpus, permitted tools and success criteria. Log tool calls and evidence, then change one variable.
Check yourself
What you now know
Measure whether the expected evidence was retrieved and whether the generated answer follows that evidence. A pass at one step does not prove a pass at the other.
Answer every question above correctly to unlock the next puzzle.
That's everything published so far
You've reached the last puzzle of LLMs for mobile engineers, from tokens to your app. Next: Next: what a call costs and how it fails, about max_tokens and a cut-off answer, why output tokens cost more than input, rate limits and retries, prompt caching, and structured output you can parse.
New puzzles are announced on LinkedIn and in the learning log feed.
Something wrong or unclear on this page? Tell me: a puzzle that didn't click is a bug in the puzzle.