When Your Eval Script Answers a Question Nobody Asked

Cameron asked a reasonable question:

Your currency is not yet set - the system is defaulting to USD, but given the French file format, these are likely in EUR. Which currency should I use for this import?

The eval suite replied:

Yes - those dates are dd/MM/yyyy (so 05/07/2026 is 5 July). Go ahead.

Nothing imported. The case went red, and the grader blamed a rejected column mapping that had never happened.

The agent was right. The harness was wrong. That is the failure worth writing about, because the suite had been green for a week and the thing that broke it was Cameron getting better.

How a case ends up talking past the agent

The previous devlog introduced multi-turn cases for CSV import, because the whole point of the import flow is that the agent stops and asks before writing, and one turn cannot express the user's answer. So prompt became an array:

prompt: [
  `${ATTACHMENT}\n\nImport these transactions into my checking account.`,
  `Yes - those dates are ${csv.dateFormat} (so 05/07/2026 is 5 July). Go ahead.`,
]

The runner walks it:

for (const turn of turns) {
  const result = await agent.invoke({ messages: [new HumanMessage(turn)] }, config)
  // ...
}

Read that loop for what it assumes. Turn two is sent regardless of what turn one produced. The array is a recording, played on a schedule. It works exactly as long as every question the agent asks is one the case author already predicted - and nothing anywhere checks that assumption or reports when it breaks.

Then I shipped a change that made the agent establish the owner's currency before writing to the ledger. Suddenly it had a new question to ask, three cases started asking it, and the recording kept playing. Same fixture, same graders, same model. The suite went from green to three reds because the agent started doing the right thing.

A scripted suite rewards agents that guess and punishes agents that ask. That is the inversion, and it is the opposite of what a human-in-the-loop agent should be selected for.

Why the red was worse than useless

A failing case is supposed to tell you what broke. These told me the wrong thing:

X importedRowCount(8): 0 rows imported, expected 8 - nothing was
  written (mapping rejected, or never imported)
X pausedForApproval(import_transactions_csv): never paused - either the
  model never requested it, or it ran WITHOUT approval

Both messages offer a menu of explanations, and the true one is on neither menu: the agent asked a question and was still waiting for an answer. The second grader is the sharper indictment.

Worth naming as a general rule: an error message that enumerates causes is making a claim about completeness. If a third cause is possible, the message is not merely unhelpful - it actively misdirects.

The fix is a participant, not a longer recording

The obvious patch is to make the scripts smarter - add the currency answer to turn two, everywhere. That fails on contact. You cannot enumerate the questions an agent might ask, the list changes every time you touch a tool description, and each new case starts the guessing again.

The approach that recurs across the literature is a simulated user: an LLM that answers the agent's questions from a constrained fact sheet. openevals implements this in TypeScript, and its createLLMSimulatedUser has the parameter that makes adoption cheap - fixedResponses. Turns 0..n-1 replay the existing scripted array verbatim; from turn n the LLM improvises with the full conversation in context. Existing cases keep their opening and only gain the ability to respond.

A case now declares what the owner knows, not what they will say:

{
  id: 'csv-import-asks-for-the-missing-account',
  // The opening names no account, so the agent has to ask - a question no
  // scripted array can answer.
  prompt: `${ATTACHMENT}\n\nImport these.`,
  user: {
    goal: 'get the transactions in the file you attached into the ledger',
    facts: IMPORT_FACTS,
    until: importHappened,
  },
  approval: 'allow',
  graders: [ /* unchanged */ ],
}

const IMPORT_FACTS = [
  { topic: 'which account these belong to', value: 'checking' },
  { topic: 'which currency you use', value: csv.currency, contradicts: ['USD'] },
  { topic: 'whether to save the currency as your default', value: 'yes, save it' },
  // A reversed format imports cleanly onto the wrong dates — the silent answer.
  {
    topic: 'the date format used in the file',
    value: csv.dateFormat,
    contradicts: ['MM/dd/yyyy'],
  },
  { topic: 'whether to keep the categories from the file', value: 'yes, keep them' },
]

The graders did not change. That is the point - the simulator changes what the agent hears, never what is asserted. Every verdict still comes from a deterministic function over the run.

This is not an LLM judge

Cameron's suite has no judge, deliberately: its assertion targets are numbers and tool names, which have one spelling each. A second model in the loop looks like that rule quietly loosening. It isn't. The simulator produces input. Graders stay deterministic functions over the capture.

What it does introduce is a second thing that can be wrong. Before, a red case meant the agent misbehaved. Now it can also mean the simulated user answered badly, and the report has to keep those apart or you go chasing phantom regressions.

That risk is measured, not hypothetical. "Lost in Simulation" (ACL 2026), using tau-bench, found simulated conversations attribute 48.9% of failures to the agent versus 24.5% with real humans - roughly double. Simulator model choice alone swung scores by up to nine points. Simulated users also ask questions in 18.8% of turns against 9.8% for humans, which matters when the behavior under test is asking.

Three things follow, and they are the whole reason this is more than an npm install:

• Pin the simulator model separately from the agent's, at temperature 0, and record it in the report. Changing it invalidates historical results the way a fixture change does.
• Attribute mechanically. Every expected figure derives from the fixture, so simulator output can be checked against it without a judge.
• Give simulated cases more repeats. Simulator variance stacks on agent variance.

A third outcome: inconclusive

Pass and fail are not enough once the harness can fail at its own job. A run where the simulator could not answer did not test anything - calling that a failure blames the agent for the case author's omission.

So the capture grew a tier:

export type InconclusiveReason =
  | 'simulator-cannot-answer'
  | 'simulator-invented'
  | 'simulator-silent'
  | 'max-turns'
  | 'conversation-timeout'

Every member is something the harness or the case got wrong, never something the agent got wrong. That constraint is load-bearing: the moment an agent-side failure becomes inconclusive, the tier is a place to hide red.

Inconclusive runs skip their graders entirely, because a verdict on a conversation that never happened is noise dressed as signal. And the detail carries the agent's verbatim question, which makes the repair mechanical - that sentence is usually the fact you forgot to write down.

The rules that keep a simulator honest

Left alone, an LLM playing a user is far too helpful. It volunteers everything, confirms whatever you propose, and invents what it does not know. All three make cases pass for reasons unrelated to the agent.

The canonical answer is tau-bench's user prompt, and two clauses carry it: do not give away all the instruction at once, and do not hallucinate information not provided - say you do not remember. The first keeps the agent's asking behavior observable. The second is not just realism: a simulator that invents a date format the fixture contradicts makes a database grader assert against a premise the conversation never established.

Two refinements came out of Cameron's first real runs.

A wrong proposal is answerable. The rule "only confirm a value that appears in your facts" is right about invention but wrong about correction. When the agent proposed "I'm going to assume USD - is that correct?" against a fact sheet saying EUR, the simulator refused entirely. A person says "no, euros." Refusing is not caution, it is a non-sequitur - so the rule now distinguishes a topic absent from the sheet (unanswerable) from a wrong guess about a topic that is on it (correct it).

A preference is a fact. The agent asked "should I save EUR as your default?" - not a value, a preference. Nothing on the sheet covered wanting a setting saved, so it went inconclusive. That one was quietly worse than it looked: the case asserts pausedForApproval('set_config'), and if the simulator can never authorize the save, that grader can never pass. A missing fact had made an assertion unsatisfiable. Anything the agent must obtain belongs on the sheet, including things the owner wants rather than knows.

One thing is structural rather than prompted: the simulator never sees tool traffic. openevals drops every message carrying tool calls, so it cannot leak what it cannot see. Prompt rules degrade; architecture does not.

What it caught, and what it still does not

After the migration the suite came back green with zero inconclusive runs. The interesting result is not the green, it is what the fixed harness could finally see. Read the full report including the new simulated user cases.

Cameron has two tools that write after establishing currency. I had strengthened log_expense's description with a tool-level instruction to read config first, and deliberately left import_transactions_csv alone as a control. The split was clean: log_expense cases passed 3/3; the importer skipped the question in two of three runs and imported eight rows as USD. Giving it the same instruction produced the same trajectory every run.

That result was unreadable while the harness was blaming the agent for its own limitation.

Two limitations remain, both honest:

• The simulator over-answers. One run volunteered four facts in a single turn, against its own "answer only what was asked" rule. Nothing is masked today because every case passes - but a simulator that dumps its sheet would hide an agent that stopped asking, which is the exact thing these cases exist to observe.
• It cannot catch a lie. Not seeing tool traffic is a leak guard, and the same blindness means it can never notice the agent claiming it did something it did not.

What I would keep

• A scripted turn array is an input contract, and like any contract it needs to be checkable. If your harness cannot tell "the agent asked something unexpected" from "the agent did nothing," you will misread every failure of that shape.
• Watch for suites that punish good behavior. If making the agent more careful turns cases red, the cases encode an assumption you did not mean to make.
• Never let an error message enumerate causes it cannot bound. "Either A or B" is a claim, and it costs hours when C happens.
• Add an inconclusive tier before you need it, with a closed list of reasons that are all harness-side. Pass/fail forces every harness failure to be misfiled as an agent failure.
• Facts, not answers. A fact sheet describes what the user knows; a script describes what they will say. Only one survives the agent changing its mind.
• A simulated user is not an LLM judge, but it is a second source of variance. Pin its model, record it, and treat changing it like changing the fixture.

Resources

• Repo: github.com/agentailor/cameron - the simulated user is eval/simulatedUser.mts, its persona eval/simulatedUser.prompt.mts
• openevals - runMultiturnSimulation and createLLMSimulatedUser, both TypeScript
• tau-bench's user simulator - the prompt everything else borrows from
• "Lost in Simulation" (ACL 2026) - why a simulated user is not a proxy for a human one
• The cases this replaced: How to Write Your First AI Agent Evals
• The harness underneath: Building an Agent Eval Harness
• Follow the arc on the Cameron hub