Traditional software is tested with assertions: given input A, expect output B. AI agents break that model. They are non-deterministic, take different paths to the same goal, call tools, and can be "mostly right" in ways that are hard to score. Yet without evaluation you are shipping on vibes: you cannot tell whether a prompt tweak, a new model or a new tool made things better or quietly worse.
Agent evaluation (often called evals) is the practice of measuring agent quality systematically, with repeatable tests that run on every change.
What to measure
- Task success: did the agent reach the correct end state (ticket resolved, record updated, answer correct)?
- Trajectory quality: did it take sensible steps, use the right tools with the right arguments, and avoid wasted or risky actions?
- Safety and policy compliance: did it stay within permissions, respect approval gates and refuse what it should?
- Efficiency: steps, tokens, latency and cost per task.
- Reliability: does it succeed consistently, not just once?
Three kinds of graders
| Grader | How it works | Best for |
|---|---|---|
| Code-based | Deterministic checks: database state, test results, schema validity, exact fields. | Anything objectively verifiable. Fast, cheap, reliable. |
| Model-based (LLM judge) | A model scores output against a rubric. | Open-ended quality: tone, completeness, helpfulness. Calibrate against humans. |
| Human review | Experts grade samples. | Ground truth, subjective quality, and calibrating the other graders. |
Building an eval suite
1. Start from real failures
Collect actual tasks and, especially, actual mistakes from pilots and production logs. Twenty to fifty well-chosen cases drawn from reality beat hundreds of invented ones. Every bug you fix should become a permanent test.
2. Cover the edges
Include easy wins, ambiguous requests, missing data, tool errors and timeouts, hostile or manipulative inputs, and cases where the correct behaviour is to refuse or escalate.
3. Make environments reproducible
Run tasks in a sandbox with controlled data and mocked or snapshot-restored systems, so a score changes because the agent changed, not because the world did.
4. Measure reliability, not luck
Because runs vary, run each task several times. A metric like "succeeds at least once in k tries" flatters an agent; a business process usually needs it to succeed every time. Track both, and watch the consistent-success rate.
5. Evaluate offline and online
Offline suites gate every change before release. Online monitoring of production traces, with sampling, user feedback and automated checks, finds the problems no test set anticipated, and feeds new cases back in.
Common mistakes
- Testing only the happy path.
- Trusting an LLM judge without ever comparing it to human grades.
- Changing prompt, model and tools at once, so you cannot tell what helped.
- Letting the eval set go stale as the product changes.
- Reading only the aggregate score and never the failing transcripts.
Evaluation is a core part of the agent harness: it is how an agent loop earns the right to run unattended. Our enterprise agent engagements ship with an eval suite from the first pilot.