Testing Tools for AI Agents: catching the bugs your agent can't tell you about

Asked how much had been spent on dining, a personal finance agent answered $2,812.25. The real figure was $2,752.25. The model wasn't hallucinating and the database wasn't wrong: the tool in between handed over 50 rows out of 262 and gave no indication that the other 212 existed.

That is a return-shape bug, and it is the kind an audit finds in tools that otherwise look well designed.

A tool is a contract between a deterministic system and a non-deterministic caller. The five principles for writing tools AI agents can actually use cover how to design that contract: consolidate around workflows, name things precisely, return context rather than IDs, spend tokens carefully, treat descriptions as prompts.

Those principles are now packaged as a skill, a reusable instruction set an agent loads on demand, published in agentailor/skills. Which raises a question worth asking of any checklist: what happens when you point it at tools that were already written with those same ideas in mind?

Cameron, a local-first personal finance agent, has eight tools written by hand, all of them predating the skill. They were built against the five principles. The audit asked whether built against actually means conforms to.

It mostly did. But the misses clustered around one thing, and it wasn't a principle anybody forgets. It was the return value.

What this article covers

This is a worked audit rather than a testing tutorial, so it's worth saying up front what you get and in what order.

1. The defect class, with four real before/after examples. What a "contract bug" actually looks like in a tool that passed design review, why each one is invisible in code review, and the one question that catches all four.
2. The deterministic tests that catch them: what to assert, why to assert it against the returned payload, and how the design checklist maps onto mechanical tests. This is the transferable method, and it's framework-agnostic.
3. The ceiling, demonstrated rather than asserted. A failure that passes every deterministic test, why it needs evals instead, and how to tell which layer a given bug belongs to.

Cameron is the example throughout, but nothing here is specific to it, to TypeScript, or to LangGraph. A tool is a tool. If you want the design side first, it's in Writing Effective Tools for AI Agents; this article is what happens after those tools ship.

What the audit produced

Most of the checklist passed. Names are action-oriented and specific. Parameters carry format and constraints. The CSV import returns a summary instead of dumping rows into the context window. Of the eight tools, most needed nothing at all.

Worth stating plainly: that result proves less than it looks like. The skill was extracted from the same thinking that produced the tools, so this is work being checked against its own externalized standard rather than an independent one. A checklist still earns its keep that way, because it catches what you knew and forgot. It just isn't a test.

What it did catch was a cluster. Four tools, four different symptoms, one underlying defect:

Every one of those return values is technically true. And every one of them is read, by a model with no other source of information, as something false.

The money bug: a capped page that claims to be complete

query_transactions lists transactions matching a filter. The repository behind it caps results at 50 by default, 200 maximum. Here is what the agent received when asked about a Dining category holding 262 transactions.

Before:

{ "count": 50, "transactions": [ ... ] }

Fifty rows, and nothing anywhere indicating that 212 more exist. count was the number of rows returned, not the number that matched. From the model's seat that payload is indistinguishable from "you have exactly 50 Dining transactions", and Cameron's system prompt actively encourages reasoning over the rows it receives.

In a finance agent that produces a wrong number stated with confidence. The worst failure mode available: not an error, just a plausible figure that happens to be false.

After:

{
  "returned": 50,
  "matched": 262,
  "truncated": true,
  "hint": "Showing the 50 most recent of 262 matching transactions. Narrow the date range, add a category/account filter, or raise `limit` (max 200). For a total or a ranking across ALL matches, use run_sql instead. Do not add up these rows."
}

The tool-side change is small, because the interesting part is one comparison:

// Truncation must be visible, or the agent mistakes a capped page for the whole set.
const truncated = total > items.length
return JSON.stringify({
  returned: items.length,
  matched: total,
  truncated,
  transactions: items,
  ...(truncated ? { hint: `Showing the ${items.length} most recent of ${total} matching…` } : {}),
})

That total has to come from somewhere, and this is where a truncation fix usually gets abandoned: nobody wants a second COUNT query on every list call. It isn't needed on every call. A short page cannot have been truncated, so the count is only worth paying for when the page came back full:

// Only pay for the COUNT when the page came back full — a short page cannot be truncated.
// Not in one transaction with the select: a concurrent insert can make `total` over-report,
// which is benign (it prompts the agent to narrow, never to over-claim).
let total = rows.length
if (rows.length === limit) {
  const [counted] = await db.select({ value: count() }).from(transactions).where(where)
  total = counted?.value ?? rows.length
}

The race condition is worth designing deliberately rather than locking away. An over-reported total makes the agent narrow a query it didn't need to narrow. An under-reported one would make it claim completeness it doesn't have. When the two error directions aren't symmetric, pick which one you want.

The subtle one: an empty result that means something else

This is the defect most worth internalizing, because it produces no wrong number and no visible failure.

When query_transactions received a category name that doesn't exist, it declined to silently drop the filter. Good instinct, wrong execution:

Before:

{ "count": 0, "transactions": [] }

That's the identical payload the tool returns for a category that exists and genuinely has no transactions. Two very different facts, one representation. Ask about "dining" when the category is stored as "Dining", and the agent is told, in effect, that you have never spent money on food.

After:

{
  "returned": 0,
  "matched": 0,
  "truncated": false,
  "transactions": [],
  "note": "No category named \"dining\" exists (the match is exact and case-sensitive). Call list_categories to see the valid names, then retry."
}

Same empty array. The difference is that the payload now distinguishes no data from bad query, and names the next move. An agent given the first payload confidently reports a false conclusion. An agent given the second one fixes its own call.

The same defect outside the read path

inspect_csv previews an uploaded file before import. Handed a bad file key, it threw.

An exception reaches the model as a generic failure with no structure, and the model's most likely next action is to retry the same call. Worse, two genuinely different situations both surfaced as "something went wrong": a file that couldn't be read at all, and a file that parsed fine but has no rows in it.

After, three distinct outcomes, each naming its own remedy:

{
  "error": "file_unreadable",
  "message": "Could not read the file for fileKey \"abc\": … Use the exact fileKey from the attachment reference. Do not invent or guess one, and ask the user to re-upload if there is no such reference."
}

{
  "error": "no_columns_found",
  "message": "No header row could be parsed, so there are no columns to map. Confirm the upload is a CSV with a header row.",
  "totalRows": 0
}

{
  "error": "no_data_rows",
  "message": "The file has a valid header row but no data rows, so there is nothing to import. Tell the user the file is empty and ask for one containing transactions.",
  "headers": ["date", "amount", "description"]
}

run_sql got the same treatment: it already reported truncated: true, which flags the problem without solving it. It now explains that a capped result is partial and that the way to a complete answer is to aggregate in SQL rather than page through rows.

And log_expense, which writes, now echoes back what was written rather than what was passed in, including a category it had to create as a side effect. An agent that coins "Grocery" when "Groceries" already exists should find that out immediately, not three conversations later.

The pattern worth taking away

Four fixes, one rule:

A tool's return shape is part of its contract, and "technically correct" is not the bar. The bar is unambiguous to a non-deterministic reader.

Every defect above passes code review. count: 50 is accurate. An empty array for a missing category is accurate. A thrown exception is a legitimate way for a function to fail. They break because the consumer is a language model that treats the payload as the complete truth about the world, has no way to ask a follow-up question, and cannot tell the difference between absent and withheld.

The practical test is a single question, applied to every field a tool returns: could this value be read as something false by a reader who sees nothing else? Ambiguity that a human developer would resolve by checking the docs or the database is ambiguity an agent resolves by guessing confidently.

The test layer that catches all four

The truncation bug wasn't subtle. A test that stubs a full page against a larger total catches it in about a millisecond. It survived because there was no test layer at all: Cameron shipped v1 with zero tests.

The general shape of that gap matters more than the specific bug. A tool description is a promise about behavior, and nothing in a typical project verifies that the implementation keeps it. The description promised a bounded list. The implementation delivered a bounded list that lied about its own bounds. Those two drift apart silently, because the only consumer positioned to notice is a model with no way to complain.

So the work stopped being "fix the bug" and became "build the thing that would have caught it." The truncation test was written first and watched fail before the fix landed:

// The regression this suite exists for: a capped page must not read as a complete result.
it('tells the agent when results were truncated, and how to narrow them', async () => {
  vi.mocked(transactionRepo.list).mockResolvedValue({ rows: makeTransactions(200), total: 847 })

  const result = await callTool(queryTransactions, { from: '2026-01-01' })

  expect(result.returned).toBe(200)
  expect(result.matched).toBe(847)
  expect(result.truncated).toBe(true)
  // "truncated" alone isn't enough — the hint must give the next move.
  expect(result.hint).toEqual(expect.stringContaining('847'))
  expect(result.hint).toEqual(expect.stringMatching(/narrow|filter|run_sql/i))
})

No database, no network, no API key, no model. Repositories stubbed, tools called directly, the whole suite running in about two seconds. That matters because a test you can run on every commit is a test that actually runs.

One detail there is worth copying deliberately: the assertions are on the parsed payload the tool returns, not on the repository call or on internal state. That's the same surface an eval grades against later, so these assertions survive the move to a model-in-the-loop harness instead of being rewritten. Test the contract at the boundary the agent actually sees.

Unit tests were sufficient here because Cameron's tools are thin wrappers over repositories: nearly all the behavior worth checking lives in the payload they assemble, so stubbing the repository leaves little that matters untested. That's the thin case, not the universal one. A tool whose correctness depends on real I/O — a multi-step write, a transaction boundary, a third-party API — needs integration tests in this same layer. The layer isn't defined by the kind of test; it's defined by there being no model involved, and therefore a stable verdict.

Most of the design checklist turns out to be mechanically testable this way:

Where deterministic tests stop, and evals start

With the suite green, the real agent ran against a seeded database of 262 Dining transactions, true total $2,752.25. Three probes:

"Show me my Dining transactions." The agent replied "Your Dining category has 262 transactions in total. Here are the 50 most recent," and flagged the partial view unprompted. ✅

"How much did I spend on Dining?" It reached for SQL, ran a SUM(), and answered $2,752.25. Exact. ✅

"List my Dining transactions and then add them up for me." It pulled the raw rows and summed them in prose: $2,812.25. Off by sixty dollars. ❌

Every test passed. The payload was correct. The agent added it up wrong, and only sometimes: re-running the same prompt produced a correct SUM() query instead.

That is the honest ceiling of the deterministic layer, and no amount of it would have helped. A test proves truncated: true is present. It cannot prove the agent noticed it, or that the model didn't quietly do arithmetic over three hundred rows of generated text. Those failures need a model in the loop, which is a different and more expensive instrument.

The two layers split on whether a model is in the loop, not on what kind of test you write. That distinction is worth holding onto, because it tells you which side a bug falls on before you go looking for it:

• If the failure is in the payload, it's deterministic. Wrong field, missing signal, an error that throws instead of explaining, a default that doesn't match the description. Cheap, repeatable, belongs in CI.
• If the payload is correct and the behavior is still wrong, it's an eval. The agent picked the wrong tool between two plausible siblings, ignored a flag that was right there, or reasoned badly over correct data. Non-deterministic by nature: the same prompt gets it right on a re-run.

Both of the bugs in this article are useful precisely because they sit on opposite sides of that line. The truncation bug was a payload defect that a millisecond of CI now catches forever. The prose-arithmetic bug is a behavior defect that no assertion over a return value will ever detect.

Which is why that second bug is logged and not yet fixed. Fixing a non-deterministic failure without a way to detect it means shipping a fix nobody can prove holds. That needs evals.

Where the principle needed a caveat

One audit finding was a place where following the skill would have made things worse.

Principle 1 says consolidate: don't wrap every endpoint as its own tool, or the agent struggles to choose among near-duplicates. Cameron has two read tools that look like textbook merge candidates. query_transactions lists rows matching a filter; run_sql answers aggregate questions with guarded, SELECT-only SQL. Same data, same domain.

Merging them would satisfy the principle's letter and make the agent's job harder. One tool with a mode flag forces a decision about which mode on every single call, and the two have genuinely different return shapes and safety properties. The separation is what lets the description of one point at the other as the escape hatch, which is exactly the mechanism that fixes the truncation problem.

The principle needed a qualifier it didn't have: consolidate operations on the same workflow, not operations that merely touch the same data.

What went back to the skill

Dogfooding is supposed to surface what you can't see from inside a project. The bug was real but small. The gap in the skill was the larger finding.

tool-design explained how to design a tool and never how to verify one. Its entire testing guidance was a single clause, "give the agent 3–5 realistic requests and watch", which describes an eval informally while omitting the cheap deterministic layer underneath it. The expensive instrument, with the affordable one missing.

That's the same gap most agent projects have, and it's not an accident: testing advice for agents tends to start at evals, because that's where the novel problem is. The unglamorous layer beneath gets skipped, and it's the one that would have caught every bug in this article.

So the skill now carries a testing section covering both layers: deterministic tests for the contract, evals for the behavior. Written together, deliberately, because a section teaching only the cheap layer would imply it suffices, and the $60 error is standing proof it doesn't. It also gained a Tested checklist item, an "unverified contract" failure mode, and the consolidation caveat above. If you'd rather have the method as something your coding agent applies than as an article you remember, that's the form to take:

npx skills add agentailor/skills --skill tool-design

The change came from the audit issue, which has the full reasoning and the evidence it was drawn from.

What shipped

• query_transactions reports returned / matched / truncated plus a hint that names the next move
• run_sql explains how to turn a capped result into a complete answer
• An unknown category reports as unknown rather than as an empty ledger
• inspect_csv returns structured errors instead of throwing
• log_expense echoes what was written, including any category it created
• 68 unit tests, offline and free, with CI gating merges and release tags
• A documented testing approach, including why unit tests aren't the whole story

Next: evals, so the prose-arithmetic bug can be fixed with proof that it stays fixed.

• Repo: github.com/agentailor/cameron
• The skill: agentailor/skills
• The principles: Writing Effective Tools for AI Agents
• Follow the arc on the Cameron hub