CoursesLLMs for mobile engineers, from tokens to your app

Puzzle 8 · 9 of 10

Find evidence before asking the model

A support assistant needs the correct help article. Compare two ways to select it for the same question.

All puzzles in this course

The problem

The help article says “register an authenticator app.” A reader asks “Can I require an extra proof of who I am when I log in?” What text should the app add to the model call?

See it

Find evidence before asking the model Question Candidate text Rank documents Keep evidence

Scroll sideways to see the full diagram.

  1. The reader describes an extra proof of identity.

    Step 1 of 4
  2. The app has a small collection of help articles.

    Step 2 of 4
  3. A lexical or learned score orders the same collection.

    Step 3 of 4
  4. The app keeps the highest-ranked evidence for the next step.

    Step 4 of 4

The building block

Retrieval selects candidate text from a collection before the application asks a model to answer. The query and every article get a score. Only the highest-ranked articles are passed along.

One approach, TF-IDF, measures how strongly words characterize each document. Common words receive less weight than words that are unusual in the collection. A short question and article can rank well if they share distinctive wording.

An embedding is a list of numbers produced by a model trained to represent text. The application can compare the direction of query and article vectors using cosine similarity. Related phrasing can end up close together even when it uses different words.

I compared both methods for twelve English questions against the same eight synthetic help articles. The expected document was assigned before running the comparisons.

Query setTF-IDF first resultMiniLM first result
Six questions with similar wording6/66/6
Six paraphrases1/66/6
All twelve7/1212/12

This run measures selection in a tiny collection. Neither method here generated an answer. The result is an example of why to measure paraphrases on your own documents.

Go deeperWhat happens with an exact reference number?
A user quotes a rare error identifier exactly. Is there a reason to keep lexical matching in the comparison?

A rare, exact identifier is a strong lexical signal. The paraphrase results tell us nothing about how either method ranks unseen identifiers, so add exact-identifier cases to the test set.

Back to the main puzzle

Check yourself

The query and article express the same idea with different words. What should you measure?

Right. Hold the corpus fixed and compare the rankings.

Not quite. Neither method in this run generates answers. Re-read that part ↑

Not quite. The problem is selecting evidence from the collection. Re-read that part ↑

The model says something plausible after receiving the wrong article. Which stage already failed?

Not quite. The wrong article was selected before the answer reached the interface. Re-read that part ↑

Right. Locate the first broken step.

Not quite. Answer length cannot repair choosing unrelated evidence. Re-read that part ↑

Both methods score 6/6 on similar wording. What can you infer for this collection?

Not quite. The experiment tests a small retrieval set, not all questions or generated answers. Re-read that part ↑

Not quite. The table separates the paraphrases and shows a difference. Re-read that part ↑

Right. Limit the conclusion to the measured cases.

Sources

What you now know

Retrieval selects evidence before generation. Word overlap and learned vectors can rank the same documents differently, especially when a query changes its wording.