CoursesLLMs for mobile engineers, from tokens to your app

Puzzle 9 · 10 of 10

Which step failed the answer?

Keep the same help question and find whether the problem happened during search or writing.

All puzzles in this course

The problem

A customer asks when a card refund will appear. The answer says “one day.” The article says “five business days.” Which system component deserves the bug report?

See it

Which step failed the answer? Known answer Search results Generated claim Compare evidence

Scroll sideways to see the full diagram.

  1. The labeled case points to the refund article.

    Step 1 of 4
  2. The application records which articles the search returns.

    Step 2 of 4
  3. A model can still make an unsupported claim after seeing the right article.

    Step 3 of 4
  4. Score selection and evidence support separately.

    Step 4 of 4

The building block

A labeled test set lists questions and the evidence or outcome expected for each. Use a small set with literal questions, paraphrases, and questions your documents cannot answer. Save it before changing the application.

Hit at 3 counts how many questions have the expected document among the first three search results. In the previous lab, TF-IDF put the expected article first for seven out of twelve and among the first three for nine. MiniLM put it first for all twelve on this collection.

An evidence check asks whether the answer's claims are supported by the text it received. Run it separately. In this example:

Evidence providedAnswer observedDiagnosis
An article about ending a subscriptionOne dayRetrieval missed the refund article.
The refund article saying five business daysOne dayThe answer contradicts available evidence.
No article that states a refund windowI couldn't find a refund window in the help articles.The answer acknowledges missing evidence.

The lab measured the first stage only. The three rows above are worked examples of the second stage, not results from a model run. If you compare two chat tools later, give them the same allowed articles and score their claims against those articles.

Go deeperDesign a fair agent comparison
One agent can search all articles and another sees only five. Can you attribute an answer difference to the model?

Not from this comparison. Freeze the corpus, permitted tools and success criteria. Log tool calls and evidence, then change one variable.

Back to the main puzzle

Check yourself

The expected article is second of three results. Does this question count as a hit at 3?

Not quite. First place is the cutoff for hit at 1, not hit at 3. Re-read that part ↑

Right. Pick the cutoff before measuring.

Not quite. This retrieval metric is computed independently of the answer. Re-read that part ↑

The model saw the correct refund article but said one day. What failed in this case?

Right. Compare the answer with the text that was actually provided.

Not quite. The discrepancy is between the claim and the provided evidence. Re-read that part ↑

Not quite. The relevant article was available to the model. Re-read that part ↑

Two chat setups differ in both model and article access. What can you conclude from a score difference?

Not quite. Article access changed at the same time. Re-read that part ↑

Right. Freeze other variables to attribute a difference.

Not quite. The model changed too, so the effect cannot be isolated. Re-read that part ↑

Sources

What you now know

Measure whether the expected evidence was retrieved and whether the generated answer follows that evidence. A pass at one step does not prove a pass at the other.

That's everything published so far

You've reached the last puzzle of LLMs for mobile engineers, from tokens to your app. Next: Next: what a call costs and how it fails, about max_tokens and a cut-off answer, why output tokens cost more than input, rate limits and retries, prompt caching, and structured output you can parse.

New puzzles are announced on LinkedIn and in the learning log feed.

All courses →