Applied AI

Course map · S2 Retrieval and RAG · Episode 3 of 16

Measuring it

S2 E3 Explainer · 21 slides · 19 October 2026

  1. Slide 1 of 21, Cover.
  2. Slide 2 of 21, Why it matters.
  3. Slide 3 of 21, Mental model.
  4. Slide 4 of 21, Key terms.
  5. Slide 5 of 21, Key terms.
  6. Slide 6 of 21, Part 1: Which half failed.
  7. Slide 7 of 21, Part 1: Which half failed.
  8. Slide 8 of 21, Part 2: Can the judge be trusted.
  9. Slide 9 of 21, Part 2: Can the judge be trusted.
  10. Slide 10 of 21, Part 2: Can the judge be trusted.
  11. Slide 11 of 21, Part 2: Can the judge be trusted.
  12. Slide 12 of 21, Finding.
  13. Slide 13 of 21, Finding.
  14. Slide 14 of 21, Finding.
  15. Slide 15 of 21, Finding.
  16. Slide 16 of 21, Where it breaks.
  17. Slide 17 of 21, For engineers.
  18. Slide 18 of 21, For leaders.
  19. Slide 19 of 21, The scoreboard.
  20. Slide 20 of 21, Decision rule.
  21. Slide 21 of 21, Series map.

Lesson notes

Split by stage, E2's 217 of 400 shows the model answers 85% right with the right passage and 8% without it, and the judge holds up except on faithfulness.

Decision rule

Score search and answer separately, include questions your documents cannot answer, and check your judge before you trust its score.

Ideas in this lesson

  • Evaluation
  • Measuring, not quoting

Take it further