Posts tagged Evals.
A scripted multi-turn eval replays fixed user turns regardless of what the agent said. The moment the agent asks a question the case author did not anticipate, the script answers something else and the case fails for the wrong reason - rewarding agents that guess and punishing agents that ask.
An eval harness is four components: the case, the runner, the capture, and the graders. A guide to what each one has to do for an AI agent, built around two real harnesses, so you can decide whether to build one, adopt a framework, or combine both.
Where the first cases come from, which ones earn a place in the suite, why every case that pushes a behavior needs one that bounds it, how to grade without an LLM judge, and what the first honest run surfaced.
Write AI agent evals that catch what you missed: source cases from production traces, spot tests that pass for the wrong reason, and know when to overrule an LLM judge.
You designed the tool well. Nothing verifies it still does what its description promises. An audit of eight tools found four contract bugs, including one that made a finance agent report $2,812 for a $2,752 bill, and the cheap test layer that catches them.