· 23 min read
When Does Your Agent Need Progressive Tool Disclosure? We Measured It
Every MCP server you connect is paid for on every model call. We ran 1,342 agent runs on two models with 3 to 20 servers, binding every tool up front and loading tools on demand, and measured tokens, cache hits, cost, accuracy and latency. A decision guide for when progressive tool disclosure is worth its extra model call, and when it is not.
Agents are easier to build with a spec. Talk yours through in a few questions, then hand the brief to your coding agent.
Plan it with AgentailorConnecting an MCP server to an agent takes a minute. A name, a URL, maybe a sign-in, and the agent has thirty new tools.
What it costs is less visible. Tool definitions are not sent once per conversation. They are sent on every model call, including the ones that have nothing to do with that server. Ask an agent with twenty servers for your account balance and the request carries the schemas for creating Stripe invoices, querying Grafana dashboards and driving a browser.
The usual answer is progressive tool disclosure: show the model a short list of what exists, and send a tool's full definition only once the model asks for it. Cameron, the personal finance agent I build in public, moves to this design in v5.
But "usual answer" is not a reason. Disclosure has a cost of its own, an extra model call to find the tool, and most writing on the subject skips the question that matters: at how many tools does it start paying for itself?
So I measured it. Same agent, same tasks, 3 to 20 MCP servers, with every tool bound up front and with disclosure, on two models. 1,342 runs, 97 million input tokens. This guide is the result, and the short version is:
- Disclosure is a cost decision, not an accuracy one. On both models I tested, binding 336 tools did not make tool choice worse.
- It has a fixed price of about one extra model call per task, so with a few small servers it is the more expensive option.
- Counted in tokens, it pays for itself at one heavy server, roughly 3,500 tokens of tool schemas.
- Counted in money, it can pay off much later. On a provider that caches well, a warm cache kept binding everything cheaper up to five servers, and disclosure only won at ten.
- At twenty servers it saved 2.5 to 8.5 times, depending on what your provider charges for cached tokens and how warm your cache is.
The code is LangChain/LangGraph in TypeScript, but the numbers and the decision rule carry to any stack.
The bill you pay on every call
Here is the input size of the first model call of a conversation, before the user's question has been answered, as servers are added. The baseline is Cameron with no MCP servers at all: its system prompt and its own built-in tools.
| Servers connected | MCP tools | First-call input tokens |
|---|---|---|
| None | 0 | about 9,400 |
| 3 small ones | 9 | 9,632 |
| 4 | 33 | 31,319 |
| 5 | 46 | 42,176 |
| 10 | 142 | 71,503 |
| 20 | 336 | 118,153 |
Two things stand out.
Server count is the wrong unit. The first three servers are small mocks, nine tools with one or two arguments each, and together they add a few hundred tokens. The fourth is the real Notion server: 24 tools and about 21,700 tokens. One production-grade server weighs more than twice the agent's entire base prompt, system prompt and built-in tools included.
The number is per call, not per conversation. A simple question takes two model calls: one to pick the tool, one to read its result and answer. At twenty servers that is 236,000 input tokens to return one account balance.
What progressive disclosure changes
The idea is the one agent skills use for instructions: keep names and one-line descriptions in context, and load the full thing on demand.
In Cameron, the model sees three things instead of every tool:
- its built-in tools, as before;
- one extra tool,
load_tools; - a short list appended to the system prompt, one line per server:
Additional capabilities are available. Call load_tools with a short description of what you need before using them.
- bank: Live data from the owner's bank: accounts, balances and recent card activity.
- calendar: The owner's calendar: upcoming and past events.
- notes: The owner's personal notes.
When the model needs something from that list, it searches:
load_tools({ query: string; server?: string; limit?: number }); // limit defaults to 5
{
"ok": true,
"loaded": [
{
"name": "bank_get_account_balance",
"description": "Current available balance of one account.",
},
],
"alreadyLoaded": [],
"matched": 3,
}
The search is plain BM25 over a cached copy of each server's tools/list. No embeddings, no model call, no connection to the server. The matched tools are added to the model's tool list from the next step on, and the server is only connected when one of its tools is actually called.
Three design choices matter for the numbers that follow:
- Definitions arrive through
tools, not through the tool result. A schema pasted into a tool result is re-sent with the history forever and can never be removed, so it saves nothing. - Nothing is ever unloaded. Changing the tool list invalidates the provider's prompt cache, so a loaded tool stays for the rest of the thread. A conversation only grows by tools the model asked for.
- It is a middleware, not a mode. The agent registers its core tools and the middleware does the rest:
// Simplified from src/lib/agent/index.ts
const disclosure = mcpServerDisclosureMiddleware({ servers, resolveTool })
const agent = createAgent({
model,
tools: coreTools, // no MCP tools here
systemPrompt,
checkpointer,
middleware: [disclosure, invalidToolCallMiddleware(), ...approval],
})
The full design is in Cameron's MCP_DISCLOSURE.md. The rest of this guide is about whether it earns its place.
The benchmark
The setup is small enough to describe in full.
Two arms, one agent. bind-all passes every MCP tool straight to the agent, the way most agents work today. disclosure runs the agent as it ships. Nothing else differs: same system prompt, same built-in tools, same model.
// eval/bench/run.mts, trimmed
const agent = await getAgent({
provider,
model,
...(arm === 'bind-all'
? { tools: world.tools, mcpForEval: { servers: [], resolveTool: never } }
: { mcpForEval: { servers: world.servers, resolveTool: world.resolveTool } }),
})
Three servers hold every answer. A bank, a notes app and a calendar, nine tools between them, each returning fixed data. Every task can be solved with these three, or with Cameron's own ledger.
Everything else is a distractor. Scales above three add the real tools/list of real servers, in a fixed order: Notion, Google Workspace, GitHub, Slack, a filesystem server, Brave, Playwright, Stripe, Airtable, Grafana, Neon, Docker Hub and five more. Their schemas are exactly what those servers publish. If the model calls one, it gets a valid, empty result. Because a larger scale only adds servers no task needs, any change in the results comes from the extra tools and nothing else.
Twelve tasks in five groups, plus one five-turn conversation:
| Group | Example | What it probes |
|---|---|---|
| Plain | "What's the IBAN of my savings account?" | The obvious tool |
| Paraphrase | "What have I been swiping my card on lately?" | No word shared with the tool name |
| Confusable | "Search for what I wrote about the boiler." | Many servers have a search tool |
| Two-server | "What's my next calendar event, and which accounts do I have?" | Two capabilities in one turn |
| Ledger | "What are my biggest spending categories?" | Needs no MCP tool at all |
| Thread | Five questions in one conversation, alternating servers and ledger | Cost as a conversation grows |
Graded by code. Each task checks which tools were called and whether the final answer states the right value, such as 4217.35. There is no model judging another model, the same rule as Cameron's eval harness.
What was recorded. Input, output and cached tokens for every model call, the number of model calls, the tools called, and wall-clock time.
The runs. Scales of 3, 4, 5, 10 and 20 servers, five repeats per task with a warm cache, then one more pass with a cold cache. The model is muse-spark-1.3-contributor, with a separate API key per arm so the provider's dashboard could confirm the totals independently. They matched to within rounding.
A second model. That provider does not charge for cached input tokens, which makes its bill a special case. So the warm run was repeated in full on openai/gpt-6-luna through OpenRouter, pinned to OpenAI as the provider, which bills cached tokens and reports the cost of every call. Results 1 to 4 below are the first model. The second has its own section, a second model, with the bill measured, and is mentioned wherever it disagrees.
Result 1: disclosure is flat, binding everything is not
Average input tokens to complete one task, warm cache, 60 runs per cell:
| Servers | Tools | Bind-all | Disclosure | Bind-all costs |
|---|---|---|---|---|
| 3 | 9 | 23,708 | 32,308 | 0.7 times |
| 4 | 33 | 77,073 | 31,758 | 2.4 times |
| 5 | 46 | 104,871 | 33,302 | 3.1 times |
| 10 | 142 | 182,125 | 33,040 | 5.5 times |
| 20 | 336 | 294,300 | 34,656 | 8.5 times |
Disclosure costs about 33,000 tokens per task whether three servers are connected or twenty. Adding seventeen servers and 327 tools moved it by 7%. That is the property you are buying: the cost of a task stops depending on what else is installed.
The same table shows the price. At three small servers, disclosure is 36% more expensive. The reason is the number of model calls:
| Arm | Model calls per task |
|---|---|
| Bind-all | 2.4 to 2.5 |
| Disclosure | 3.2 to 3.3 |
Disclosure adds a little under one call per task, the load_tools search, and each call re-sends the base prompt of about 9,400 tokens. Ledger tasks need no MCP tool, so they pay nothing extra; every other task pays for one search, or two when it spans two servers.
Result 2: accuracy did not move
This is the result I did not expect.
| Servers | Bind-all correct | Disclosure correct |
|---|---|---|
| 3 | 60 of 60 | 60 of 60 |
| 4 | 60 of 60 | 60 of 60 |
| 5 | 60 of 60 | 60 of 60 |
| 10 | 60 of 60 | 60 of 60 |
| 20 | 60 of 60 | 59 of 60 |
With 336 tools bound, including 24 distractor tools with "search" in their name, the model picked the right one every time. Across the 732 runs on the first model, only 6 called a distractor tool at all, 2 in the bind-all arm and 4 with disclosure, and every one of them still reached the right answer.
The second model agreed. All 610 of its runs called the right tools and stated the right value, in both arms at every scale, and none called a distractor. (The grader marked 12 of them wrong for writing $2,733 where it expects two decimals. I counted those as correct after reading them.)
The common claim that a long tool list degrades tool selection did not show up here. That is two current models on tasks with a clear right answer, and I would not generalize it to a smaller model or to servers that overlap, two CRMs or three search providers. But it means this guide cannot honestly sell disclosure as an accuracy feature. On these numbers, it is a cost feature.
The single failure is worth a look, because it belongs to disclosure. At twenty servers, "What have I been swiping my card on lately?" was once answered from Cameron's own ledger. The model never called load_tools. It decided its built-in tools were enough and reported ledger spending, a confident and wrong answer.
That is the failure mode disclosure adds. With every tool bound, the bank's card-activity tool is in front of the model. With disclosure, the model first has to decide to look, guided only by one line in the capability list. The quality of that one line per server is now load-bearing, in the same way a tool description is.
Result 3: latency depends on the provider
On the first model, median time per task stayed between 9.4 and 12.1 seconds in every cell, with no trend in either arm. A first call carrying 118,000 input tokens was not measurably slower than one carrying 9,600, and the extra model call in the disclosure arm did not show up as extra time.
The second model behaved differently:
| Servers | Bind-all, median | Disclosure, median |
|---|---|---|
| 3 | 3.0 s | 5.4 s |
| 4 | 5.0 s | 5.5 s |
| 5 | 5.3 s | 6.5 s |
| 10 | 6.8 s | 5.6 s |
| 20 | 8.7 s | 6.4 s |
There, binding everything got slower with every server added, almost three times slower at twenty servers than at three, while disclosure stayed around six seconds. With a few servers the extra model call makes disclosure the slower arm; from ten servers it is the faster one.
So whether input size costs you time is a property of the provider, not of the design. Measure it on yours.
Result 4: a long conversation
The single-task numbers undersell the difference, because real use is a conversation. One thread at ten servers, five questions alternating between the bank, the ledger, the notes and the calendar:
| Bind-all | Disclosure | |
|---|---|---|
| Input tokens, whole conversation | about 854,000 | about 168,000 |
| Model calls | 12 | 15 |
load_tools calls | 0 | 3 |
| MCP tools in each request, end | 142 | 9 |
| Input per call, first to last | 71.5k → 73.1k | 9.6k → 12.1k |
| Correct | 5 of 5 | 5 of 5 |
Disclosure used a fifth of the tokens. Three searches loaded nine tools, and after five turns and fifteen model calls each request had grown by only 2,500 tokens, loaded tools and conversation history included.
The part caching changes
Everything above counts tokens. Your bill does not, because most providers charge less for input tokens they have seen before. This is where the answer stops being a single number.
Prompt caching works on prefixes: if the start of a request matches a recent one, that part is billed at a discount. A bind-all agent is the best case for it. Its tool list never changes, so the huge, expensive prefix is identical on every call.
Disclosure caches worse, by design. Each load_tools changes the tool list, and a changed tool list is a changed prefix.
Share of input tokens served from cache, warm run:
| Servers | Bind-all | Disclosure |
|---|---|---|
| 3 | 81% | 67% |
| 4 | 81% | 68% |
| 5 | 86% | 61% |
| 10 | 88% | 65% |
| 20 | 89% | 64% |
By the fourth and fifth repeat, bind-all at ten and twenty servers was 100% cached. So the fair comparison is not total tokens. It is uncached tokens, the ones billed at full price:
| Servers | Bind-all, uncached | Disclosure, uncached |
|---|---|---|
| 3 | 4,610 | 10,552 |
| 4 | 14,437 | 10,067 |
| 5 | 14,214 | 12,860 |
| 10 | 22,633 | 11,559 |
| 20 | 32,288 | 12,334 |
At twenty servers the gap shrinks from 8.5 times to 2.6 times.
What this costs depends on your provider
Here is the caution that matters most when reading these numbers.
The first model I tested does not charge for cached input tokens. On my bill, the 8.5-times gap in tokens really was closer to 2.6 times in money, and for the whole main run the two API keys came to €0.50 for bind-all against €0.32 for disclosure, despite bind-all sending 4.6 times as many tokens.
That is one provider's pricing. Others bill cached tokens at a fraction of the input price, the fraction varies, and some only cache when you mark the cacheable part of the request yourself. So instead of one answer, here is how much more bind-all costs than disclosure, for input tokens only, at different cached-token prices:
| Servers | Cached tokens free | 10% of input price | 25% | 50% | Full price (no caching) |
|---|---|---|---|---|---|
| 3 | 0.4 times | 0.5 | 0.6 | 0.7 | 0.7 |
| 4 | 1.4 times | 1.7 | 1.9 | 2.2 | 2.4 |
| 5 | 1.1 times | 1.6 | 2.1 | 2.6 | 3.1 |
| 10 | 2.0 times | 2.8 | 3.7 | 4.6 | 5.5 |
| 20 | 2.6 times | 4.0 | 5.5 | 7.0 | 8.5 |
To place yourself in that table, compute it from your own traces:
cost per task = uncached input × input price
+ cached input × cached price
+ output × output price
Find your provider's cached-input price on its pricing page, and check whether caching is automatic. Two more things belong in the estimate. Disclosure produced 10% to 20% more output tokens, since it makes an extra tool call, and output tokens usually cost several times more than input. And the first column of the table moves around more than the others (5 servers reads lower than 4) because it depends entirely on which requests happened to hit the cache.
A warm cache is the best case, so I measured the worst
The warm run repeats the same prefix hundreds of times, back to back. That flatters bind-all. A personal agent that gets a few messages an hour starts most conversations with nothing in the cache, since cached prefixes typically expire within minutes.
So I ran every task once more with a unique line at the top of the system prompt, which guarantees the first call of each conversation is never served from cache. Later calls in the same conversation still can be.
| Servers | Bind-all, uncached | Disclosure, uncached | Bind-all costs |
|---|---|---|---|
| 3 | 17,472 | 23,867 | 0.7 times |
| 4 | 50,065 | 23,181 | 2.2 times |
| 5 | 70,905 | 26,705 | 2.7 times |
| 10 | 96,065 | 28,188 | 3.4 times |
| 20 | 187,680 | 25,059 | 7.5 times |
With a cold start the cached-token price barely matters, because little is cached: at twenty servers the gap is between 7.5 and 8.2 times whatever your provider charges.
A second model, with the bill measured
The table of cached-token prices above is arithmetic on one model's token counts. To check it against a real bill, I repeated the warm run on a model that charges for cached tokens and returns the cost of each call. Per task, 60 runs per cell:
| Servers | Arm | Input tokens | From cache | Measured cost | Bind-all costs |
|---|---|---|---|---|---|
| 3 | Bind-all | 15,552 | 99% | $0.00021 | 0.4 times |
| 3 | Disclosure | 21,691 | 91% | $0.00052 | |
| 4 | Bind-all | 24,277 | 99% | $0.00034 | 0.5 times |
| 4 | Disclosure | 22,530 | 87% | $0.00062 | |
| 5 | Bind-all | 42,721 | 100% | $0.00048 | 0.8 times |
| 5 | Disclosure | 22,530 | 87% | $0.00064 | |
| 10 | Bind-all | 94,823 | 99% | $0.00111 | 1.8 times |
| 10 | Disclosure | 23,066 | 88% | $0.00061 | |
| 20 | Bind-all | 164,099 | 99% | $0.00185 | 2.5 times |
| 20 | Disclosure | 24,474 | 85% | $0.00073 |
The token columns repeat Result 1. Disclosure is flat at about 22,000 tokens, binding everything grows tenfold, and the lines cross at the fourth server. This model counts the same tools as fewer tokens than the first one, so compare the ratios between the two models, not the absolute numbers.
The cost column does not follow the tokens. At five servers, binding everything sent 1.9 times the tokens and was still the cheaper arm. Disclosure only won on the bill at ten servers. Three things explain it:
- Bind-all's prefix cached almost perfectly. 99% of its input was billed at the cached rate, against 85% to 91% for disclosure, whose tool list changes after each
load_tools. - Disclosure makes one more model call, 3.1 per task against 2.3.
- It writes more. Disclosure produced 38% more output tokens over the run, and output is the expensive kind.
This is also why the earlier table could not have predicted this bill: it assumed the first model's cache-hit rates, and this provider cached far more. Your cache-hit rate matters as much as your cached-token price.
The five-turn conversation at ten servers showed the same shape: 464,000 input tokens and $0.0050 with everything bound, 116,000 and $0.0027 with disclosure. The whole run cost $0.27 for bind-all and $0.20 for disclosure, for three times the tokens.
I did not run the cold-cache pass on this model. Given the first model's result, a cold start should move the break-even back toward the fourth server, but that is an expectation, not a measurement.
So the honest range at twenty servers is 2.5 times to 8.5 times. You are near the low end if your provider discounts cached tokens heavily and your traffic is steady enough to keep the cache warm. You are near the high end if cached tokens are billed close to full price, or if your agent is used in short, spaced-out conversations.
The decision rule
The benchmark reduces to one comparison.
Binding a server costs its schema tokens, multiplied by every model call the agent makes, whether or not the call uses that server.
Disclosure costs about one extra model call per task that needs an MCP tool, and that call re-sends your base prompt.
They break even when:
schema tokens ≈ base prompt tokens × extra calls per task ÷ calls per task
For Cameron that is 9,400 × 0.9 ÷ 2.4, about 3,500 tokens of tool schemas. The measured points agree: at a few hundred tokens of schemas bind-all wins, and at 21,700 it loses by more than two times. I have no real server small enough to test the middle, so treat 3,500 as an estimate. All three inputs are numbers you can read off your own traces.
That rule counts tokens, which is the cold-cache bill. With a warm cache on a provider that caches well, the tokens you bind are billed at a fraction of their price, and the break-even moves out. On the second model it fell between five and ten servers, somewhere between 12,000 and 33,000 tokens of schemas. Treat 3,500 as the point where disclosure can start to pay, and your own bill as the judge of whether it does.
In practice:
| Your situation | What to do |
|---|---|
| One provider, and its API has tool search | Try that before building anything. See the next section. |
| One or two small servers, a few thousand schema tokens in total | Bind everything. Disclosure costs you a call. |
| One heavy server (20+ tools with rich schemas), or four to five | Wins on tokens. On a warm, well-cached bill it may not yet. |
| Ten or more servers | Disclosure, clearly: 1.8 to 5.5 times. |
| Users can add their own servers | Disclosure. You do not control the total. |
| Long conversations, or short spaced-out ones with a cold cache | Disclosure. Both widen the gap. |
| Cached tokens free or nearly free, and steady traffic | The gap halves or more. Measure before building. |
And what you take on in exchange, beyond tokens:
- A search that has to be good. A missed search means a missing tool. BM25 was enough here, but it needs tool names and descriptions worth indexing.
- A capability line per server that has to be good. It is the only thing telling the model to look.
- State. Which tools are loaded is part of the conversation and has to be saved with it, including what happens when a loaded tool's server is later removed.
- A cached catalog. Listing tools means connecting, so the list has to be stored and refreshed.
That is real engineering. Cameron's implementation is a package of its own with a design document, and it took several rounds against real servers to get right. If you are under the threshold, not building it is the correct decision. And if you are over it, building it yourself is still not the only option.
Before you build it: your provider may already do this
Everything above measures a version of disclosure I built. If your agent talks to a single provider, check its API first, because the two largest ones now ship the same idea as a feature.
Anthropic has a tool search tool, in a regex and a BM25 variant. You still send every tool definition, but mark the ones to hold back with defer_loading: true. They stay out of the model's context until a search returns them. You can also run the search yourself, with embeddings for example, and hand back references to the tools you found.
OpenAI has tool_search in the Responses API, on gpt-5.4 and later. The same defer_loading: true goes on a function, on a namespace of functions or on a whole MCP server. For a deferred namespace or server, the model sees only its name and description up front, which is Cameron's one-line capability list built into the API. The search runs on OpenAI's side or in your code.
Side by side with what Cameron does:
| Cameron's middleware | Anthropic tool search | OpenAI tool search | |
|---|---|---|---|
| What the model sees first | One line per server, plus load_tools | The search tool and any tools not deferred | Name and description of each namespace or server |
| Who runs the search | Your code, BM25 | The provider (regex or BM25), or your code | The provider, or your code |
| Where a loaded tool goes | Into the tool list | Appended in the conversation | At the end of the context |
| Effect on the cached prefix | Changes it | Leaves it untouched | Designed to leave it untouched |
| Works across providers | Yes | No | No |
The fourth row is the one that matters after this guide's results. Cameron adds a loaded tool to the tool list, which sits at the start of the request, so every load_tools is a cache miss. That miss is a large part of why binding everything stayed cheaper up to five servers on the second model. Both providers document the opposite behavior: a discovered tool is added after the existing context, so the cached prefix survives.
On paper, that keeps disclosure's flat token curve without the cache penalty, which would move the break-even on a warm bill back toward the fourth server. I did not measure it. Like the gateway below, this is a comparison of designs, not of results, and OpenAI's own guide tells you to compare token usage and latency on your own requests before making it the default.
Two more things from Anthropic's documentation are worth setting against this benchmark. It recommends tool search from about ten tools or 10,000 tokens of definitions, the same order of magnitude as the break-even here. And it states that tool selection gets worse past 30 to 50 available tools, which this benchmark did not see at 336. My distractors came from unrelated domains, so I would trust their warning over my result for servers that overlap.
Building your own still makes sense when:
- The agent is not tied to one provider. Cameron's owner picks the model, including local ones behind an OpenAI-compatible endpoint, so the feature cannot depend on one API.
- You need to own the state. Which tools are loaded, what happens when a loaded tool's server is removed, and where the catalog is cached are all decisions the provider otherwise makes for you.
- You want one behavior to test. A middleware behaves the same on every model. Two native features do not.
Where this stops scaling: Uber's gateway
Cameron has one owner and, at the high end, a couple of dozen servers. Its capability list is one line per server in the system prompt, and that is where the design runs out: at some number of servers the list itself is the problem.
Uber has published how they handle that end of the scale: more than 800 MCP servers and over 5,000 tools. Their answer is the same idea, pushed up a level and out of the agent:
- A registry and a gateway. Servers are registered centrally, and agents connect to one proxy rather than to servers directly.
- Discovery as tools. A single MCP server in front of everything exposes a handful of meta-tools: discover a server, discover its tools, fetch one tool's schema, invoke it. Not even the server list is in the prompt.
- Trimming responses. The gateway can project a tool's response down to the fields the agent needs. Disclosure only addresses definitions; at that scale, results are the other half of the context bill.
- Governance by default. Tools are disabled until someone enables them, which is an organizational control as much as a technical one.
Side by side:
| Cameron (one owner) | Uber (an organization) | |
|---|---|---|
| Scale | Up to a few dozen servers | 800+ servers, 5,000+ tools |
| Where discovery lives | Middleware inside the agent | A gateway in front of every agent |
| What the model sees first | One line per server, plus load_tools | A few discovery tools |
| Steps to reach a tool | Search, then call | Find server, find tool, get schema, call |
| How a tool is called | As a real tool, with its own name | Through a generic invoke tool |
| Who decides a tool is allowed | The owner, per call, at the approval gate | Central policy, off by default |
| Connections | Pooled in the agent, opened on first call | Held by the gateway |
The trade is the one this whole guide is about, at a larger size. Every level of indirection makes the context smaller and the path to a tool longer. Cameron pays one extra call. A four-step discovery pays more per task and is only rational when the alternative is thousands of tools.
Uber's post does not publish token or latency figures, so this is a comparison of designs, not of results. But the direction is clear: you do not start with a gateway. You start by binding tools, move to disclosure when one heavy server tips the balance, and reach for a gateway when the number of teams, not the number of tokens, becomes the problem.
Limits of these numbers
- Two models. Accuracy held at 336 tools on both. A smaller or older model may not, and that would add an accuracy argument this guide does not make.
- Two pricing models. Cached input tokens were free on the first provider and discounted on the second. A third will land somewhere else, which is why most tables are in tokens.
- Easy tasks with one right answer. Distractors came from unrelated domains. Servers that genuinely overlap are a harder test for both arms.
- Back-to-back repeats. The warm numbers are the best case for bind-all. The cold pass has one repeat, twelve tasks per cell, on the first model only.
- One base prompt. Cameron's is about 9,400 tokens. A leaner agent has a cheaper extra call and a lower break-even; a heavier one, the opposite.
- Provider tool search was not measured. Anthropic and OpenAI both offer it, and both document a cache behavior that should beat the middleware tested here. That section compares designs only.
- Distractors never fail. Real servers time out, need sign-in and return large results. None of that is measured here.
The benchmark is on the feat/mcp-disclosure-bench branch of Cameron's repository, under eval/bench/. It is not merged into main, since it exists to produce these numbers. It runs against any provider the agent supports, so the most useful number is the one you produce with your own model and your own servers.
Key Takeaways
- Tool definitions are paid on every model call. One real server can weigh more than your whole system prompt; Notion alone added about 21,700 tokens.
- Disclosure makes the cost of a task independent of what is installed. About 33,000 tokens per task from 3 servers to 20, while binding everything went from 24,000 to 294,000.
- It has a fixed price. Roughly one extra model call per task. Under about 3,500 tokens of schemas, binding everything is cheaper.
- It did not improve accuracy. Both arms were right essentially every time on both models, even with 336 tools bound. On a current model this is a cost decision.
- It adds one failure mode. The model can decide not to look. The one-line description of each server is what prevents it.
- Tokens are not the bill. On a model that bills cached tokens, a warm cache kept binding everything cheaper at five servers despite 1.9 times the tokens. Disclosure won at ten.
- Count uncached tokens, then apply your provider's prices. At twenty servers the saving ran from 2.5 times on a warm, well-cached bill to 8.5 times with no caching.
- Check your provider before building. Anthropic and OpenAI both ship tool search, and both load a discovered tool without touching the cached prefix. I did not benchmark either.
- A gateway is the next level, not the first. The same trade, for when you have hundreds of servers and many teams.
What to Read Next
- Writing Tools for AI Agents — the descriptions the search and the model both depend on.
- How to Add Skills to a Custom Agent — the same load-on-demand idea, applied to instructions.
- Building an Agent Eval Harness: The Complete Guide — the harness this benchmark is built on.
- Meet Cameron — the agent behind the numbers, built in public.
Resources
AGENT BRIEFINGS
Stay measured as the field moves.
What actually matters for building and scaling AI agents in production — and what's just hype. Straight from the work, no filler.
$ subscribe agent-briefings
→ what works in production, what doesn't.
→ frameworks, MCP, evals, managed services.
→ signal over hype.