Building an Agent Eval Harness: The Complete Guide

Your tools can be correct and your agent still wrong. A unit test proves a tool returns what its contract promises: the right fields, the right shape, the right error when you hand it nonsense. It says nothing about whether the agent picked that tool over a worse one, read the answer properly, or stopped when it should have asked.

Answering that means running the real agent, with a real model, against real data, and then deciding whether what came back was good enough. That is a harness, and you have a genuine choice about where it comes from. There are hundreds of eval tools out there, a handful of them mature enough to hand you a harness on day one. You can also write the thing yourself in an afternoon, which is more tempting than it used to be now that a coding agent will draft it for you.

Both are defensible. What makes the decision hard is that "eval harness" is not one thing. It names four components plus a set of choices around them, and a framework replaces some of those better than others.

So this article walks through the components by building them: how Cameron, a local-first finance agent I build in public, assembles its harness, what each piece has to do, and which properties of agents make it different from the test runner you already use. The point is not that you should build your own. It is that once you can see the parts, "build, buy, or both" stops being a matter of taste.

You already know four of the parts

Start with the familiar case. Every test framework hands you the same four things:

JUnit, Vitest, pytest, and Jest converged on those four because the shape holds across every project, which is why the maintenance is worth sharing and almost nobody writes their own.

An agent eval harness needs those same four things. But five properties of agents change what each one has to do, and none of them are properties the existing libraries were designed around:

• Non-determinism. One run passes, the next fails, with no code change in between. A single green result is not evidence of anything.
• Cost. Every run is a paid API call. "Run everything on every commit" stops being free advice.
• Multi-turn. Often the behavior you want to test is the agent stopping to ask a question, which a single input-output pair cannot express.
• No oracle. There is no expected value to compare against. A correct answer has many valid phrasings, and a correct trajectory has many valid paths.
• Real data over mocks. This one reverses a habit. Unit tests mock the world to stay fast and deterministic; evals want the real corpus, because the failures worth catching are the ones real inputs produce.

Those five are the actual subject here. Everything below is a component, plus what these five did to it.

The components

Cameron's eval harness is the worked example throughout: about 1,500 lines across nine modules, small enough to hold in your head and real enough to have caught a genuine bug. The second example is this blog's own agent, which made the opposite choice at almost every turn. Nothing below is specific to either, to TypeScript, or to LangGraph.

Every harness converges on the same four parts, whatever it is built with:

• The case is a task plus the graders that judge the result. Non-determinism adds a repeat policy; a human-in-the-loop gate adds a decision to play back.
• The runner drives the real agent against whatever corpus you point it at, and records what happened. Multi-turn means it replays several turns on one thread.
• The capture is everything the runner recorded, and the only thing graders are allowed to see. It is the seam between running and judging.
• The graders turn a capture into pass or fail. With no expected value to compare against, they assert structural facts, atomic values, or a judgement from a second model.

Those four are the harness. Everything else is personalization: what corpus you run against, how many times you repeat a case, what the run writes to disk when it finishes. Frameworks differ mostly in how much of that they decide for you, which is the whole build-versus-buy question in one sentence.

Three type definitions carry most of the design:

interface EvalCase {
  id
  prompt: string | string[]
  graders: Grader[]
  runs?
  approval?
}
interface RunCapture {
  finalText
  trajectory
  toolResults
  interrupts
  error?
}
interface Grader {
  id
  grade: (capture: RunCapture) => GradeResult | Promise<GradeResult>
}

prompt is a string or an array of strings, which is multi-turn. runs is the repeat policy, and it is optional because repeats cost money. approval exists because this agent has a human-in-the-loop gate that a case may want to answer.

Those three definitions are the whole of Cameron's harness contract, and they depend on no eval library at all — which is what keeps the build-versus-buy decision reversible, and comes back at the end.

The case

A case is one task plus the graders that judge how the agent handled it. Cameron's shortest one carries every moving part:

{
  id: 'denied-expense-writes-nothing',
  description:
    'A DENIED expense must leave the ledger untouched. The worst regression this repo ' +
    'could ship is a rejected mutation that still writes.',
  prompt: 'Log a $12.50 coffee on my checking account.',
  approval: 'deny',
  graders: [
    // Positive half: the gate actually fired. Without it, an agent that never called the
    // tool would satisfy the row-count assertion for the wrong reason.
    pausedForApproval('log_expense'),
    rowCountInStore('transaction', FIXTURE.transactionCount),
  ],
  runs: RUN_POLICY.strict,
  tags: ['approval', 'mutation'],
}

The description is not decoration. Six months from now it is the only thing that explains why a red run matters, and it is what tells you whether to fix the agent or delete the case.

Note the two graders. One proves the approval gate fired; the other proves the ledger is unchanged.

Cases also travel in pairs. Here is one from this blog's agent, which asserts that a recent article gets answered without a staleness caveat:

{
  id: 'freshness-recent-no-caveat',
  description:
    'OVER-TRIGGERING GUARD: a question routed to the July 2026 MCP v2 post must be ' +
    'answered without any staleness hedge.',
  turns: [{ content: "What's actually changing in MCP v2, and what's being deprecated?" }],
  graders: [noStalenessCaveat(), keywordPresent('MCP')],
  tags: ['quality', 'freshness', 'over-triggering'],
}

Its siblings assert the opposite: that old articles are flagged as dated. Without this one, a prompt fix that made the agent hedge on absolutely everything would turn the whole suite green while making the agent worse. Any case that pushes a behavior wants a partner that bounds it.

It also pairs a judge with a plain string check, which is the common shape: use the cheap grader wherever it will do, and spend the model call only on the part that needs judgement.

The runner

The runner is the part that actually costs money. It takes a case, drives the real agent through it, and hands back everything that happened.

Three things make it more than a function call.

It replays turns on one thread. When a case has several turns, they belong to the same conversation, so the runner keeps the thread alive rather than sending each turn cold. That is what lets a case test an agent stopping to ask a question, and it is the reason prompt is a string or an array.

It may have to answer a pause. When the agent stops for human approval, the runner has to reply on the human's behalf, or the case never finishes. That changes what has to be recorded, which is the next component.

It can run the same case more than once. Non-determinism means one green run is weak evidence, so a case can ask for n runs and require k of them to pass. Repeats are opt-in because they multiply the bill, and choosing n and k is a case-authoring decision rather than a harness one — How to Write Your First AI Agent Evals covers how to set them.

Where the harness runs

The runner has to point at something. The tempting answer is production. It is already provisioned, it is definitionally realistic, and pointing an eval at it costs nothing to set up. It is also the wrong default, for a reason that has little to do with realism: evals are destructive by nature. A case that tests an import has to write. A case that tests a deletion has to delete. And repeatability means resetting between runs, which means wiping whatever the last run left behind.

So the better shape is a separate environment that mirrors production rather than being it: same database engine, same migrations, same object storage, same versions, different instance. Cameron runs its evals against a parallel stack on different ports.

# compose.eval.yaml — the same stack as compose.yaml on its own ports and volumes. Ports are
# hardcoded, not `${VAR:-default}`: inheriting the variables would let a stray POSTGRES_PORT
# point evals at the dev database.
name: cameron-eval

services:
  db:
    image: postgres:17-alpine
    environment:
      POSTGRES_DB: cameron_eval
    ports:
      - '5545:5432' # dev runs on 5544
    volumes:
      - eval_db_data:/var/lib/postgresql/data

  minio:
    image: quay.io/minio/minio:latest
    ports:
      - '9110:9000' # dev runs on 9100
    volumes:
      - eval_minio_data:/data

volumes:
  eval_db_data:
  eval_minio_data:

That parity matters more than it looks. An eval running against SQLite while production runs Postgres is not testing the same agent: date handling, collation, and concurrency all differ, and those differences surface as agent behavior. Mirror the infrastructure, isolate the instance.

The capture

Here is the detail I did not anticipate before building Cameron's harness, and the one that generalizes furthest.

Take the denied-expense case above. The problem it runs into is that a paused tool call and an executed one look identical in the trajectory. A call that got stopped is still a call the model made, recorded the same way as one that ran. If all you capture is the sequence of tool calls, you cannot tell a gate that worked from a gate that was never there.

So what got paused has to be captured separately from what got called. It is the only evidence the gate did its job, and no amount of reading the transcript afterwards will reconstruct it.

There is a trap on the other side of this. In LangGraph the gate is middleware, so an eval that wants to avoid the pause can end up removing it:

// "Approve everything" isn't a mode — it's the middleware being absent.
const middleware = cfg?.approveAllTools
  ? []
  : [
      humanInTheLoopMiddleware({
        interruptOn,
        descriptionPrefix: 'This action needs your approval',
      }),
    ]

An agent built that way cannot pause at all, so the suite reports green on the one rule it was meant to check.

The general rule: a harness can only assert what it captured, so what you capture is a design decision, not a logging detail. Cameron captures four things, and each one buys a class of assertion that would otherwise be impossible.

Decide this early. Adding a capture field later means re-running every case that needed it, and re-running is the expensive operation.

The corpus

In unit testing, the default is to mock. You replace the database, stub the network, freeze the clock, and you do it so the test is fast, free, and deterministic. That instinct is correct and deeply trained.

In evals it runs the other way. You want real data, and ideally production data, because the failures worth catching are the ones real inputs produce. A suite built on invented data tests the failures you were able to imagine.

Two agents I work on sit at opposite ends of this axis, which is what makes the trade concrete rather than theoretical:

Cameron's fixture is a constraint of the project, not a recommendation — built in public, the only realistic corpus would be my own bank statements. What is worth copying is not the fixture but the discipline applied to it. One of its tools shows why.

query_transactions lists transactions matching a filter. It caps how many rows come back, so its contract has to tell the agent when it is looking at a partial set:

name: 'query_transactions',
description:
  "Search the user's transactions with optional filters (date range, type, account, " +
  'category, text). Read-only. Use this to LIST matching transactions, never to compute ' +
  'totals or rankings — for those use `run_sql`. Returns `{ returned, matched, truncated, ' +
  'transactions }`. When `truncated` is true you are seeing a PARTIAL set — never sum or ' +
  'count these rows as if they were complete.',

The contract is correct, and it is not self-enforcing: nothing stops the agent reading truncated: true and summing the rows anyway. That is what the fixture has to expose, which gives the rule: engineer the fixture so a wrong answer is still visible.

• The Dining category holds 262 transactions against that 200-row cap. Any attempt to total it by listing rows must come back truncated. The trap is the point.
• The CSV fixture uses dates that are valid under two different readings, so an agent that guesses the format instead of asking imports cleanly onto the wrong months. The CSV case has the full construction.

That second one is the shape to internalize. A fixture that only catches loud failures tests the easy half of your agent.

The right-hand column costs nothing to maintain and catches what you would never have invented: its cases come from traces, including real user questions replayed verbatim, typos intact, because the exact phrasing is what reproduced the bug. I wrote about that suite in How to Write AI Agent Evals That Prove You Wrong.

Live data is not free either. A case pinned to a specific article silently stops testing what it claims when that article is renamed, and both its graders can end up passing vacuously. Dates have to be computed at run time or the case rots.

Mocking buys stability and costs realism; production data buys realism and costs stability. Use production data unless something stops you.

This is also the honest reason tracing comes before evals. You cannot distill cases from traffic you never recorded.

Graders

A grader takes the capture and returns pass or fail. That is the whole interface, and there are two kinds.

Deterministic graders are plain code. They assert a structural fact — which tool ran, what is in the database now, whether the gate fired — or an atomic value like a figure or a tool name. They are free, instant, and incapable of nuance:

export function statesAmount(expected: number): Grader {
  return {
    id: `statesAmount(${expected})`,
    grade: (c: RunCapture) => {
      const normalized = c.finalText.replace(/,/g, '')
      if (normalized.includes(expected.toFixed(2))) return result(id, true)
      // Surface what it DID say — for a wrong total that number is the whole finding.
      const numbers = [...normalized.matchAll(/\d+\.\d{2}/g)].map((m) => m[0])
      return result(
        id,
        false,
        `expected ${expected.toFixed(2)}; amounts in answer: ${numbers.join(', ')}`
      )
    },
  }
}

LLM-as-judge graders ask another model for a verdict, which is what you need when the thing you are asserting has no single correct spelling. The judge gets a rubric, the request, and the response, and answers true or false:

const FLAGS_STALENESS_PROMPT = `You are evaluating whether an AI assistant correctly SIGNALLED
THE AGE of a dated source it answered from.

Context: the assistant answers from a blog whose posts span roughly two years. Older posts
still carry real value, but their version-specific detail reflects the ecosystem as it was
when written. The wanted behavior is to USE the old article AND briefly note that it is dated.

Return TRUE if the response both (a) actually answers from the material, and (b) somewhere
signals the source's age or that specifics may have moved on.

Return FALSE if it presents dated specifics as current with no age signal, or refuses to use
the article, or hedges so heavily the answer is useless.

Do not require an exact date or a specific phrasing — any clear age signal counts.

<request>{inputs}</request>
<response>{outputs}</response>`

Notice what that rubric spends most of its words on: not the pass condition, but the ways a response could be wrong in both directions. A judge that only knows what good looks like will pass anything confident.

The two agents split cleanly here. Cameron is judge-free, because its assertion targets are numbers, and a number has one spelling. 2733.00 is atomic; a claim like "acknowledged the denial" is not, so matching one phrasing of it passes by luck and fails on harmless rewording. The blog agent needs judges for exactly that reason: whether an answer flagged its source as dated is a judgement, and no string match will settle it.

Reach for a judge when your assertion target genuinely has many correct spellings, and not before. Every judge is a second model call, a second thing that can be wrong, and a second thing to debug when a case goes red for no reason.

Which cases earn a place in a suite, why a negative assertion standing alone is worthless, and what the first honest run actually found are the subject of the companion piece, How to Write Your First AI Agent Evals.

Cameron's harness, end to end

Those four components, assembled. Cameron's suite is eleven cases, grouped by the defect class they guard rather than by feature:

analysis     total-via-sql-aggregate                      aggregate with SQL, don't page rows
             top-categories-ranking                       rank in SQL, not in prose
             listing-uses-query-transactions              the inverse guard: list with the list tool

truncation   truncated-page-not-reported-as-total         a capped page is not a complete answer

approval     log-expense-pauses-for-approval              the gate fires on a mutation
             denied-expense-writes-nothing                a denied mutation leaves no trace

discipline   no-double-prompt-on-mutation                 ask once, not twice
             unknown-category-recovers                    an unknown category is not "no spending"

csv          csv-import-confirms-ambiguous-date-format    ask which date format, then import
             csv-import-does-not-import-before-confirming don't write before the answer
             csv-import-maps-accented-category-header     map `Catégorie`, don't translate it

Eleven is small on purpose. A case earns its place by guarding a failure a unit test provably cannot reach, which is a much shorter list than "things the agent does."

Running them needs the sandbox stack and an Anthropic API key:

docker compose -f compose.eval.yaml up -d   # Postgres + MinIO, on eval ports
pnpm eval                                   # every case
pnpm eval approval                          # or filter by id or tag

The output is deliberately plain: one line per case, a dot per run, and a verdict.

Cameron evals — 11 case(s)
  model: anthropic / claude-haiku-4-5
  db: cameron_eval (sandbox, wiped)
  seeded

  total-via-sql-aggregate • PASS (1/1)
  top-categories-ranking • PASS (1/1)
  listing-uses-query-transactions • PASS (1/1)
  truncated-page-not-reported-as-total ••• PASS (3/3)
  log-expense-pauses-for-approval • PASS (1/1)
  denied-expense-writes-nothing ••• PASS (3/3)
  no-double-prompt-on-mutation • PASS (1/1)
  unknown-category-recovers ••• PASS (3/3)
  csv-import-confirms-ambiguous-date-format ••• PASS (3/3)
  csv-import-does-not-import-before-confirming • PASS (1/1)
  csv-import-maps-accented-category-header ••• PASS (3/3)

11/11 cases passed
  report: eval/results/latest.json

Note the dots. Three of them means the case opted into repeats, so PASS (3/3) is a different claim from PASS (1/1), and the console says which one you got.

The console scrolls away, though, and the run cost money. So each run also writes latest.json, a timestamped copy for history, and a standalone HTML page:

That is the top of the page; the full report shows every case with its graders, trajectory, and final answer.

What it costs

Four cents a case, and note that is twenty-one agent invocations rather than eleven, because five cases run three times. Cheap enough that the reason to keep a suite small is that small suites stay meaningful, not that you cannot afford a large one.

That number holds because Cameron's graders are plain functions. Add an LLM judge and it inverts: the judge is a second model call per grader, and it can outspend the thing being tested. This blog's agent is that case, and it is the article I am writing next.

Build, buy, or both

Cameron's harness was built from scratch, for two reasons worth separating.

The first is positional. The Agentailor line is independence: lock-in by choice, not by consequence. Building the minimal version first means deciding whether to adopt a framework with real information about what it would replace, instead of guessing up front.

The second is pedagogical, and more broadly useful: you cannot reason well about a component you have only ever consumed. The trajectory problem two sections up is the example — it surfaced from writing the runner, and would have stayed invisible behind a library's abstraction until it bit.

The counterweight is real, and coding agents have changed its shape. Getting a prototype harness standing is now cheap. Maintenance is the cost, and maintenance is recurring. Cameron's harness stays small because it is scoped to one agent with one shape of assertion; that is not the general case. Reach for a framework when the plumbing becomes the bottleneck rather than the point: trajectory matchers, trace parsing, batching, retries, CI reporting, dashboards.

For most projects the answer is both, and it is worth designing for that on day one. The Grader interface is the seam. Because graders are plain functions over a capture, adopting a library later means writing adapters that satisfy the interface, not rewriting the suite. Build the seam even if you never use it, and the decision stays reversible.

For the current landscape of what you would be adopting, Top 7 AI Agent Evaluation Frameworks groups the options by the job they actually do.

Offline and online

One distinction that gets muddled: offline versus online is about when you run, and it is independent of what you run against.

Offline means a curated set of cases and a verdict you can gate on. Online means scoring real production traffic as it arrives, which is why it usually lives inside a tracing platform.

The blog agent shows the two axes are genuinely independent: it runs offline evals against live production data, which is neither of the usual stereotypes.

Two practical notes. Both harnesses keep tracing off during eval runs, because the persisted report, not a trace, is the artifact of an eval. And neither runs evals in CI, because they cost money and need a model. Cameron gates merges on the free unit tests instead, and keeps the eval tree honest with a typecheck that costs nothing.

What you actually need on day one

• A case type. A prompt and a list of graders. Add repeat policies and approval decisions when a specific failure demands them, not before.
• A runner that captures more than the answer. The trajectory, the tool results, and anything your agent's guardrails do. Capture decisions are expensive to revisit.
• The most realistic corpus you can safely point it at. Production data unless something stops you; if something does, engineer the fixture so a wrong answer stays visible.
• Plain-function graders. Assert structural facts and atomic values. Reach for a judge when your assertion target genuinely has many correct spellings, not by default.
• A report on disk. Console output scrolls away, and a slow paid run deserves an artifact you can diff, re-read, and attach to a bug report.
• An isolated place to run. Mirror production, but never point a destructive suite at it.

The harness is not the goal. It is the thing that makes the next question answerable, and the next question is which cases are worth paying for.

• Repo: github.com/agentailor/cameron — the harness is eval/, with the design notes in its README
• The companion: How to Write Your First AI Agent Evals
• The layer underneath: Testing Tools for AI Agents
• Cases from traces: How to Write AI Agent Evals That Prove You Wrong
• The landscape: Top 7 AI Agent Evaluation Frameworks
• Follow the arc on the Cameron hub