How to Write Your First AI Agent Evals

The starting point

Cameron - the local-first personal finance agent I build in public, and whose source is open - had a unit test suite that was green, fast, and free. It also could not catch the failures I actually worried about.

A unit test proves truncated: true is in a tool's JSON payload. It cannot prove the agent noticed it. That gap is where evals live, and closing it takes two things the test suite did not need: somewhere to run the real agent, and a set of tasks worth running.

You need a harness first, and it might be you

You cannot write an eval case without something to run it on. That something is a harness: it drives the real agent, records what happened, and decides whether the result was good enough.

For a small agent, the harness can be you. Run the agent, read the answer, decide if it is right. That is a real eval - it has a case, a run, and a grader, and the grader is a human. It is the correct starting point, and it stays correct right up until the run count times the case count stops fitting in an afternoon.

Cameron passed that point. The suite below needs a wiped database between runs, a paused tool call read off a checkpoint, and eleven cases where some run three times - twenty-one agent invocations that nobody is going to drive by hand. So it got a real one: a harness that drives the agent against a sandbox Postgres and grades with plain functions. You have the same three options - grade by hand, build one, or adopt a framework - and the article on building it walks through the four components so the choice is informed.

Either way, what you have at that point is a harness and an empty suite. These are the notes from filling it.

What a case actually is

A case is one task you give the agent, plus the graders that decide whether what came back was acceptable. That is the whole shape:

{
  id: 'truncated-page-not-reported-as-total',
  prompt: 'How much did I spend on dining this year?',
  graders: [toolCalled('run_sql'), statesAmount(FIXTURE.diningTotal)],
}

A prompt, and assertions over what happened. Not an input and an expected output - there is no single correct answer string to compare against, which is why the right-hand side is a list of graders rather than a value.

Two things follow from that, and they shape everything below. A grader can assert on what the agent said or on what the agent did - which tools it called, what is in the database afterwards - and the second kind is usually the stronger one. And a case is not passed or failed by a person reading it, so whatever you want to be true has to be expressible as code.

Where the first cases come from

The hard part is not the format. It is knowing which eleven tasks are worth paying a model to run.

You cannot reason your way to that list from an empty page, and you do not have to. There are three places the first cases come from, and none of them require an eval suite to already exist:

• You, using your own agent. Run it and pay attention to what makes you wince. This is where every case in this suite started.
• Traces, if the agent is in production. Real failures, real phrasings, no imagination required. It is the best source there is, and it is the honest reason tracing comes before evals: you cannot distill cases from traffic you never recorded.
• Someone who knows the domain. A business expert who has never seen the code can usually list the edge cases that matter within minutes, because they know which wrong answers are expensive.

Cameron's came from the first. Before any of this existed I sat down and probed the agent by hand, and it told me two things: how much had been spent on dining ($2,752.25, correct), and then, asked to list the transactions and add them up, $2,812.25. Sixty dollars off, stated with total confidence.

That produced a list of observed failures, and the next move was to write tests for them. Some landed cleanly: the underlying tool really was returning a capped page that looked complete, and a unit test caught that in a millisecond. Others would not reduce to a test at all. The payload was correct and the agent still added it up wrong - and only sometimes, which is the part no assertion over a return value can reach.

I could have fixed that one and probed it again by hand. But a fix you cannot re-verify is a fix nobody can prove holds, and re-verifying by hand does not survive the tenth time. So the failures that had no test-shaped home became the first eval cases instead.

That order is worth keeping. Observe a failure, try to push it down to the cheapest layer that can catch it, and let evals inherit only what will not fit. A case that a unit test could have caught is a case you will pay for on every run, forever.

What earns a case

Those three sources give you candidates, not a suite. Left unfiltered they sprawl - the temptation is to keep one case per failure you ever saw, or worse, one per feature, which produces something expensive, slow, and mostly redundant with tests you already have.

The rule that worked: a case earns its place by guarding a defect class a unit test provably cannot reach. For Cameron that meant five:

Eleven cases, grouped by the defect class they guard rather than by feature:

analysis     total-via-sql-aggregate                      aggregate in SQL, don't sum rows by hand
             top-categories-ranking                       GROUP BY in SQL, don't page transactions
             listing-uses-query-transactions              the inverse guard: list with the list tool

truncation   truncated-page-not-reported-as-total         never report the capped page's 200 rows

approval     log-expense-pauses-for-approval              pause at the gate, then write exactly one row
             denied-expense-writes-nothing                a denied mutation leaves the ledger untouched

discipline   no-double-prompt-on-mutation                 the tool call IS the proposal, don't ask twice
             unknown-category-recovers                    an unknown category is not "no spending yet"

csv          csv-import-confirms-ambiguous-date-format    5 July, not 7 May: ask before importing
             csv-import-does-not-import-before-confirming inspect on turn one, never write
             csv-import-maps-accented-category-header     map `Catégorie`, don't translate it

Small on purpose - the anti-pattern is believing you need a big suite before you start.

Pair every case that pushes with one that bounds

The first tool-selection case asserts the agent uses SQL for a total. On its own, that invites a prompt fix that over-rotates into always use SQL - breaking plain listings, with the suite still green.

So each case that pushes a behavior gets a partner that bounds it. "Show me my 5 most recent transactions" asserts the opposite: query_transactions, not SQL.

The vacuous pass

None of the eleven cases uses an LLM judge, which sounds like a stance and is really just a consequence: every one of them asserts a figure, a tool name, or a row count, and a number has one spelling. Reach for a judge when the target genuinely has many correct spellings, and not before - the harness guide has that side, with a worked rubric.

Staying judge-free has a trap of its own, though, and it is the subtlest way to write a worthless case: a negative grader with nothing positive beside it.

toolNotCalled("run_sql") passes when the agent errors, refuses, or answers from thin air. It passes hardest when the agent does nothing at all. Every negative assertion needs a positive one proving the work actually happened.

Making the approval gate observable

Cameron's first hard rule is that it never mutates without explicit human approval - and it was the last capability to get a case, because the harness had been running with the approval middleware switched off. An agent built that way cannot pause at all, so any case asserting the rule would have gone green against an agent with no gate. (The harness guide has why that happens, and what a runner has to capture so it cannot.)

Making the rule evaluable means the case declares its own decision - approval: "allow" | "deny" - and the runner answers the pause the way the UI would, reading the paused call off the checkpoint and resuming with it:

// With `approval` set the middleware is live, so a mutating call pauses instead of running.
if (testCase.approval) {
  const actions = await pendingActions(agent, threadId)
  interrupts.push(...actions) // captured separately — see below

  if (actions.length > 0) {
    const decision: Decision =
      testCase.approval === 'allow'
        ? { type: 'approve' }
        : { type: 'reject', message: 'The user denied this action.' }
    // One decision per action request, positionally aligned — the middleware batches a
    // turn's approval-requiring calls into a single interrupt.
    result = await agent.invoke(
      new Command({ resume: { decisions: actions.map(() => decision) } }),
      config
    )
  }
}

Note pendingActions returning an empty array when nothing is pending. For an approval case that is itself the finding: the mutation never reached the gate.

That empty array is also the only thing separating the two outcomes, because a paused tool call and an executed one look identical in the trajectory - a paused call is still a call the model made. The two cases come back from a real run byte-identical on everything except the row count:

// log-expense-pauses-for-approval          // denied-expense-writes-nothing
{ "trajectory": ["log_expense"],            { "trajectory": ["log_expense"],
  "interrupts": ["log_expense"] }             "interrupts": ["log_expense"] }

For a case author that has one practical consequence: you cannot assert the gate from the trajectory. The assertion has to be an allow/deny pair on the same prompt, separated only by the decision, and graded on what is in the database afterwards:

A regression where a rejected mutation still writes is the worst bug this project could ship, and it now fails loudly.

What the first honest run found

Every case passed but one. The red one was unknown-category-recovers, and it was worth more than all the greens put together, because it turned out to be two problems wearing one coat.

The case asks about a category the fixture does not contain, and expects the agent to establish that rather than report an empty result as "no spending yet." As first written:

{
  id: 'unknown-category-recovers',
  prompt: 'How much did I spend on Entertainment?', // a category the fixture does not contain
  graders: [
    toolCalled('list_categories'),
    categoriesListed(), // only checks the list_categories result
  ],
}

The case was broken. It asserted the agent would call list_categories. The agent instead reached for describe_finance_schema then run_sql - a different route to the same place, and an entirely legitimate one. The case was asserting a path when what it cared about was an outcome.

Worse, the second grader was conditioned on that same tool. When list_categories never ran, it had nothing to check and reported green. A vacuous pass, sitting in the one case that had gone red for an unrelated reason - which is how a grader that asserts nothing stays hidden.

The fix was to stop naming the route:

graders: [
  // Route-agnostic: several tool paths establish which categories exist, and asserting
  // one would fail a run that got there another way.
  anyToolCalled('list_categories', 'query_transactions'),
  // The recovery must name a category that really exists. Atomic — a category name has
  // one spelling — so this survives having no judge.
  statesAnyOf(FIXTURE.categoriesBySpend.map((c) => c.category)),
]

Note the second grader reading from the fixture rather than a hardcoded list. Add a category to the seed and the case keeps asserting the right thing.

And a real product defect. Asked about a category that does not exist, a bare SUM() returns SQL NULL, which is indistinguishable from a category that exists with no spending. The agent passed that ambiguity straight through:

// The failing run: two tool calls, and a SQL NULL shown to the user.
{
  "trajectory": ["describe_finance_schema", "run_sql"],
  "finalText": "The result is **null**, which means you don't have any transactions
                categorized as **Entertainment** yet. Would you like me to:
                1. Check what categories you *do* have on record? ..."
}

Two things are wrong there. The user is being shown null, a SQL artifact that means nothing to them. And "yet" implies the category exists and is simply empty - when it does not exist at all, and the three real categories were one tool call away. The agent even offers to go check them, which is the right next step, framed as a question instead of taken.

Rerunning the same prompt produced a different trajectory:

// The rerun: one more tool call, and a correct answer.
{
  "trajectory": ["describe_finance_schema", "run_sql", "list_categories"],
  "finalText": "**You don't have an \"Entertainment\" category yet.** Your current
                categories are: - Dining - Groceries - Transport ..."
}

Same prompt, same NULL from the same query. The difference is the third tool call: this run went and checked instead of guessing, and the answer is correct because of it.

That is the finding, and it is one no single run produces. Not "the agent gets this wrong" but the recovery is not reliable - the correct path costs an extra round trip, and sometimes it is not taken. A bug report saying "sometimes answers from a bare NULL" is actionable in a way that either run alone is not.

The fix went into the tool, not the prompt. run_sql now recognizes the ambiguous payload and says what it means:

// An aggregate over zero rows returns NULL, and that payload is identical whether the
// filtered name exists or not — so the agent has to guess unless the result says which.
if (values.length > 0 && values.every((v) => v === null)) {
  return (
    'Every value is NULL, which is what an aggregate returns when NO rows matched — it is not a ' +
    'zero balance. This does NOT confirm the filtered values exist; an unknown category/account ' +
    'name gives the same NULL. Check the spelling against the data (e.g. `list_categories`) ' +
    'before reporting an empty result to the user.'
  )
}

That is the cheapest layer the failure could be pushed to, and it is the same move as the truncation fix: the agent was reasoning correctly over a payload that was ambiguous, so the payload got fixed rather than the reasoning. The case has passed every run since. It stays in the suite anyway - it is what proves the hint changes behavior, and what will catch it if a future prompt change stops the agent reading the note.

Both fixes landed together, which is the honest sequence: the case had to be able to fail for the right reason before the defect underneath it was worth chasing.

Repeats are not noise reduction

The obvious reading of a flaky case is that you should run it a few times and take the majority, to stop noise from deciding the result. That is backwards. The flapping is the result.

Run unknown-category-recovers once and you learn "broken" or "fine," depending on which of those two trajectories you caught. Run it several times and you learn the rate - it was close to a coin flip - and the rate is the thing you would actually put in a bug report. "Sometimes answers from a bare NULL" and "always does" are different bugs with different priorities, and a single run cannot tell them apart.

That is what pass@k is for, and it has two knobs answering different questions. n is how much evidence you buy. k is how much a failure costs: require unanimity when a wrong outcome is silent or unrecoverable, and a majority when the behavior legitimately varies but you want the common case right.

single:   { n: 1, passK: 1 }  // the default: near-deterministic structural assertions
majority: { n: 3, passK: 2 }  // behavior known to vary run to run
strict:   { n: 3, passK: 3 }  // a wrong outcome that is silent, unrecoverable, or both

unknown-category-recovers runs majority - three runs, two must pass - because the behavior it guards is exactly the kind that varies. While the tool fix was being verified it ran at n=5 and came back 5/5, twice; in the suite it now reports 3/3. Those are different claims, and the report records both numbers rather than a verdict:

{ "id": "unknown-category-recovers", "passed": true, "runsAttempted": 3, "runsPassed": 3 }

Contrast denied-expense-writes-nothing, which is strict. A rejected mutation that still writes is silent and unrecoverable, so two out of three is not a pass - it is a bug that happens to be outvoted.

Repeats are opt-in because they multiply the bill, and the right default depends on what the suite is for. A suite that gates merges wants repeats everywhere, since an unrepeated green cannot be told apart from a lucky one. Cameron's suite gates nothing - it exists to be read - so most cases run once and repeats are spent only where instability is the point.

The case that had to be built differently

Everything above grades a single question and answer. CSV import could not work that way, and it is the capability that mattered most: a wrong date format produces a clean, successful import with transactions on the wrong dates. No error, no truncation signal, nothing to alert on. Every other failure in this suite is loud; that one is silent.

Two things had to change.

Multi-turn. The whole point of the import flow is that the agent stops and asks before writing, and a single turn cannot express the user's answer. So prompt becomes an array, and the second turn is the user answering the question the first turn should have provoked:

prompt: [
  `${ATTACHMENT}

Import these transactions into my checking account.`,
  // The user answers the question the prompt requires the agent to ask. If it already
  // imported on turn 1, the graders catch it — this turn cannot un-import anything.
  `Yes — those dates are ${csv.dateFormat} (so 05/07/2026 is 5 July). Go ahead.`,
]

Turns replay on one thread id and the checkpointer carries the conversation - a much smaller change than it sounds, because the agent already had persistent memory. The runner just had to stop re-counting the whole thread on every turn.

A fixture built to make the failure visible. The file is French, and every date has a day under 13:

Date,Libellé,Montant,Catégorie,Revenu/dépense
05/07/2026,Café du matin,4.50,Dining,Dépense
06/07/2026,Boulangerie,12.00,Groceries,Dépense
07/07/2026,Ticket de métro,2.10,Transport,Dépense

05/07/2026 is a real date under both readings - 5 July and 7 May. Guess wrong and the import succeeds with zero bad rows and no error. The only evidence is the month the rows landed on. The accented Catégorie header and the French type column do a second job: a translated mapping is rejected outright, so English defaults cannot paper over the problem.

Then grade the consequence, not the conversation. Not "did it ask about the ambiguous date?" - that is a claim with many spellings, and there is no judge here - but "did the rows land in July or in May?" One is an opinion. The other is a row in a database:

export function importedInMonth(month: number, wrongMonth: number): Grader {
  return {
    id: `importedInMonth(${month})`,
    grade: async () => {
      const months = await importedMonths()
      if (months.length === 0) return result(id, false, 'nothing was imported')
      const wrong = months.filter((m) => m.month !== month)
      if (wrong.length === 0) return result(id, true)
      // Name the diagnosis, not just the mismatch.
      const misread = wrong.some((m) => m.month === wrongMonth)
      return result(
        id,
        false,
        `imported rows landed on month(s) ${wrong.map((m) => `${m.month} (${m.count})`).join(', ')}` +
          (misread ? ` — month ${wrongMonth} means the date format was read backwards` : '')
      )
    },
  }
}

That grader takes the wrong month as well as the right one, which is what turns a red run into a diagnosis. "Expected July, got May" is a mismatch you still have to think about. "Month 5 means the date format was read backwards" is the bug report already written.

That is the shape I would reach for again: when the failure you fear is silent, stop asserting on what the agent said and go look at what it did.

Running the suite

Eleven cases, five of them repeating, is twenty-one agent invocations - about three minutes and roughly forty cents. The console output is one line per case, a dot per run:

  truncated-page-not-reported-as-total ••• PASS (3/3)
  log-expense-pauses-for-approval • PASS (1/1)
  denied-expense-writes-nothing ••• PASS (3/3)
  unknown-category-recovers ••• PASS (3/3)

The dots are the pass@k policy made visible: PASS (3/3) and PASS (1/1) are different claims, and the run you are reading should say which one it is.

That scrolls away, though, and the run cost money. So every run also writes a standalone HTML page alongside the JSON:

Each case expands to its graders, its trajectory, and the agent's final answer - which is what you actually want when a case goes red, because "which grader failed" is rarely the whole story. The importedInMonth message naming a backwards date format only helps if you can find it.

Open the full report - the real artifact from the run described in this article, all eleven cases.

The suite itself lives in eval/, and running it needs the sandbox stack plus an API key:

docker compose -f compose.eval.yaml up -d   # Postgres + MinIO, on eval ports
pnpm eval                                   # every case
pnpm eval approval                          # or filter by id or tag

Why a sandbox rather than the real database, how the runner captures interrupts, and what to reach for instead of building this yourself are all in Building an Agent Eval Harness.

What I would keep

• Start from observed failures, not imagination. Your own use, production traces, or someone who knows the domain. Then push each one to the cheapest layer that can catch it.
• Group cases by defect class, not by feature. It keeps the suite small and every case justifiable.
• Pair pushing cases with bounding ones, or a prompt fix will satisfy the suite by overcorrecting.
• Stay judge-free while your assertions are atomic. Numbers and tool names do not need a model to grade them.
• Never let a negative grader stand alone. A run that did nothing should not score green.
• Derive expected values from your fixture, never hardcode them - a constant fails for reasons that have nothing to do with the agent.
• Write the report to disk. Console output scrolls away; a slow, paid run deserves an artifact you can diff and re-read.
• Spend repeats where behavior actually varies, not everywhere. Match the default to whether your suite gates anything.
• Expect red on the first honest run. A case that has never been observed failing is not known to test anything - and the one failure taught me more than every pass combined.

Resources

• Repo: github.com/agentailor/cameron - the cases are eval/cases/, the graders eval/graders.mts
• The report: every case from this run, with trajectories and final answers
• The harness underneath: Building an Agent Eval Harness
• The layer below that: Testing Tools for AI Agents
• Cases from production traces: How to Write AI Agent Evals That Prove You Wrong
• Follow the arc on the Cameron hub