Weight: 16%. About 10 of the 63 exam questions.
This domain is about proving quality, finding the cause of failures and tuning cost and latency.
| Objective | What it tests |
|---|---|
| 4.1 Metrics | Accuracy, latency, cost, safety, security; per-component metrics |
| 4.2 Datasets and frameworks | Representative and adversarial data, labels, mixed methods, CI gates |
| 4.3 A/B testing and iteration | Randomisation, guardrail metrics, sample size, one change at a time |
| 4.4 Diagnosis | Prompt failure vs hallucination vs model mismatch vs pipeline and data bugs |
| 4.5 Optimisation | Tokens, latency, routing, caching, batch |
| 4.6 Monitoring | Logging, versions, alerts, feedback linked to traces |
Mental model
- Evaluation is the system's test suite. Every prompt or model change runs it.
- Diagnose by asking "what changed?" and by looking at what the model actually received.
Exam heuristics
- Only the prompt changed → suspect the prompt. Nothing changed but an upstream system → suspect the inputs.
- Failures cluster in a complexity slice and prompts don't help → model mismatch.
- Invented details despite correct sources → hallucination; ground with quotes or citations and allow "I don't know".