Posts tagged AI Agents.
A scripted multi-turn eval replays fixed user turns regardless of what the agent said. The moment the agent asks a question the case author did not anticipate, the script answers something else and the case fails for the wrong reason - rewarding agents that guess and punishing agents that ask.
An eval harness is four components: the case, the runner, the capture, and the graders. A guide to what each one has to do for an AI agent, built around two real harnesses, so you can decide whether to build one, adopt a framework, or combine both.
Where the first cases come from, which ones earn a place in the suite, why every case that pushes a behavior needs one that bounds it, how to grade without an LLM judge, and what the first honest run surfaced.
Anthropic deleted over 80% of Claude Code system prompt for Claude 5 with no measurable loss on their coding evals. The instructions did not become wrong. The model outgrew needing them. Here is the maintenance pass that finds which of yours have expired, and why you need evals before you delete a single line.
Agent Plugins is an open standard for packaging Agent Skills and MCP servers into one portable folder any client can load. What plugin.json and mcp.json actually define, which clients support it, and whether to package one yet.