The measurements that did not exist five years ago, and that most AI products still do not take. What a pass rate means, how to measure whether an answer is supported by its sources, why task success is graded on the end state, and what the copilot metrics actually tell you.
| # | Lesson | The question it answers |
|---|---|---|
| 01 | Eval Pass Rate and Coverage | What does a pass rate mean, and what share of real behaviour does your set cover? |
| 02 | Groundedness and Citation Accuracy | How do you measure whether an answer is supported by its sources? |
| 03 | Task Success on End State | Why grade the outcome rather than the path, and how? |
| 04 | Acceptance Rate | The copilot metric: what fraction is kept, and what does keeping mean? |
| 05 | Edit Distance and Survival | How much of what you generated is still there an hour later? |
| 06 | Intervention and Escalation Rate | How often does a human step in, and what does that cost? |
| 07 | Containment and Deflection | The support metric everyone quotes. What does it hide? |
| 08 | Retry, Regeneration and Abandonment | What do users do when the answer is bad, and how do you count it? |
| 09 | Refusals and Over Refusal | When is a model saying no a defect rather than a safeguard? |
| 10 | Measuring Trust | Can you measure it at all, and what are the honest proxies? |
| 11 | Segmenting Quality | Why does one average pass rate hide the segment that is failing? |
| 12 | Worked: A Quality Dashboard | One real set of metrics, and what each one is for |