Evaluating AI agents: offline evals, pass@k and production metrics
Analytics case · 8 min read
How to talk about measuring an AI agent in an interview: task success, graders, pass@k versus pass^k, and the online metrics that confirm it.
What an eval is
An eval gives the system an input, then applies grading logic to the output. For agents, the input is a task plus tools and an environment, and the grader often checks the final state, not just the text.
pass@k versus consistency
pass@k asks whether at least one of k attempts succeeds, which suits tools where a person picks the best result. When every run must work, such as a customer-facing agent, measure the chance that all k attempts succeed instead. Say which one matches the product.
Offline then online
Use a fixed golden task set to catch regressions before release. After launch, track resolution rate, escalation to a human, cost per task and user-reported errors, and compare against a holdout where possible.
Common mistakes
- Grading only the final message and missing a wrong action taken by a tool.
- Using pass@k for a product where every run must succeed.
- Reporting accuracy without cost or latency.
Further reading
Try it now, free
Task success and pass@k
No account needed. You get graded feedback right away.
Answer the question