k-onfusion: 200,000 tools, and the agent picks the wrong one
Imagine walking into a restaurant and the waiter handing you a menu the size of an encyclopedia. I’d be overwhelmed, and a little worried too, because why does one kitchen need 27 kinds of chicken? Agents have the same problem with MCP tools: give one 10 tools and it can read all of them and pick the right one (and if it picks wrong, it just tries another), but that stops working once the menu has 200,000 things on it.
I wanted to find out where it stops working and why, so I built k-onfused, a small test bench that hands an agent a task and a catalog of up to 200,000 tools and checks whether it called the right tool with the right arguments. Groceries are expensive, so the whole thing ran on headless Claude Code with my existing subscription, which cost me some evenings and a big bite of my usage window, and nothing on an API bill.
Results up front
60%
tasks solved at 200k tools, 10 per search
from 85% at 100 tools
30%
tasks solved at 200k tools, 3 per search
from 85% at 100 tools
67%
right tool picked once it was in the results, 200k, 10 per search
from 85% at 100 tools
Up to 10,000 tools, the agent barely notices how big the catalog is, and
somewhere between there and 200,000 it falls off a cliff. When the search
hands back only 3 tools, the right one often never reaches the agent at all, and
when it hands back 10, the right one is usually there but the agent picks its
lookalike, a v2 of the same tool or the by-id version when it was given a URL.
I tried the 2 obvious fixes, a second agent that does the searching for it and a small fast model that re-sorts the search results, and neither moved the number by more than noise, because at that size the search is mostly doing its job and what’s left is the agent choosing between near-identical tools it was already handed.
Building a 200,000-tool catalog
Deploying 200,000 real tools was never going to happen, but I did need 200,000 of them, so I mocked them. They’re generated from a seed1 so every run sees the same catalog:
- apps in several editions (Slack, Slack Enterprise, Slack EU)
- the things you’d expect inside them like issues, invoices and job postings
- the usual actions on each
Playing devil’s advocate, I also made the catalog confusing on purpose, the way
real ones get once a few vendors ship a v2 and never retire the v1, so
there’s a daybay_create_event next to a daybay_v2_create_event, and a by-id
and a by-url version of most actions.
The tasks never reuse a tool’s wording either, so where the tool says “Permanently removes a job posting in Staffwise, identified by its numeric ID”, the task says “Nuke vacancy number 28286 in Staffwise”, because “nuke” is what a person would actually type and I wanted the agent to make that jump instead of matching words.
How the agent searches
Pushing 200,000 tool definitions straight into the model’s context isn’t really an option, unless you enjoy watching your credits disappear, so the agent needs a way to ask for tools instead. The idea came from the encyclopedia again: when I want to read about an animal I don’t flip through the whole thing; I go to the animals section.
I put every tool into Qdrant, a database that searches by meaning rather than exact words,2 so when the agent asks for something about GitHub it gets back the tools closest to what it asked for. How many tools come back is called k (hence the name), and it’s the knob this whole post turns, since a bigger k gives the agent more to choose from and a smaller one gives it less.
Every trial starts the agent with a single tool,
request_tools,
where it describes what it needs in its own words, gets k tools back and has to
call the right one:
- task
- request_tools"delete a vacancy in Staffwise"
- search200,000 tools
- top khanded to the agent
- call
- graderight tool, right args
A trial only passes if the right tool got called with the right arguments.3 I ran 20 tasks (15 that need 1 call and 5 that need 2) against catalogs of 100, 10,000 and 200,000 tools, with either 3 or 10 tools per search, all on Claude Haiku 4.5.
The agent runs inside Claude Code with everything else switched
off,
because the last thing I wanted was my own CLAUDE.md quietly helping it in
every trial, and every trial records what it did so I could open any one of them
in Arize Phoenix and see where it went wrong, which
is how I can tell a tool that never showed up apart from a tool the agent walked
right past.
Where accuracy falls off
Going from 100 tools to 10,000 changed almost nothing, and going to 200,000 cost a quarter of the tasks with 10 per search and more than half of them with 3:
| 100 | 10k | 200k | |
|---|---|---|---|
| 10 per search | 85% | 85% | 60% |
| 3 per search | 85% | 80% | 30% |
With 3 tools per search, 11 of the 17 tasks that passed at 100 tools failed at 200,000, and not a single one went the other way, which, with only 20 tasks, makes it the one result in this post I’m confident isn’t noise.
Why lookalikes win
It breaks in 2 different ways, depending on how many tools come back.
With 3 per search, the right tool mostly never shows up. It reached the agent in only 45% of trials at 200,000 tools, and although the agent noticed and searched again with different words, there were simply too many near-matches ahead of it in line.
With 10 per search, it shows up and the agent picks its twin. The right tool
was handed over 90% of the time, but when it was, the agent called it only 2
times out of 3. My favourite example is a task that asked it to “Set up an
appointment onto Team Calendar on Daybay”, where the right tool,
daybay_create_event, came back first out of 10, and the agent went down the
list to number 6, called daybay_v2_create_event with perfectly valid
arguments and then told me it had successfully set up the appointment.
It made the same choice every single time at 10,000 tools and above, which I honestly find hard to blame it for, and the other misses look the same, either another edition of the right app or the by-id version of a tool when the prompt gave a URL.
Set up an appointment onto Team Calendar on Daybay called "Rotate API keys quarterly" from 2026-10-07T09:30:00Z.
request_tools("create a calendar event in Daybay on Team Calendar")
1daybay_create_eventright tool0.834
2daybay_list_event0.826
3daybay_us_create_event0.825
4daybay_enterprise_list_event0.823
5daybay_eu_create_event0.820
6daybay_v2_create_eventcalled0.820
7daybay_cloud_list_event0.817
8daybay_create_event_invite0.810
9daybay_v2_list_event0.809
10daybay_enterprise_create_event0.807
“Successfully created appointment "Rotate API keys quarterly" on Team Calendar in Daybay.”
Two fixes that didn’t help
If the agent gets k tools back and still has to decide by itself, the natural move is to put something smarter in between, and I tried one slow version of that and one fast one, both only at 200,000 tools since that’s where the problem was.
The librarian
Now think about another thing: what if the agent didn’t search at all? The librarian is a second Claude session that the main agent (the worker) goes to and says “hey, this is the task I need to do”, and the librarian runs its own searches, with filters for app, resource and action, and decides which tools the worker should get.
It cost about 2.5× the model calls and time per trial and didn’t clearly help, solving 45% of tasks where plain search solved 30% with 3 per search, and 55% where plain search solved 60% with 10. The traces showed why: it used a filter in only 19 of its 219 searches even though filtering was the whole point of having it, and it usually handed back fewer tools than it was allowed to, so a bigger k was mostly wasted on it.
The re-ranker
Picking one tool out of a handful is more of a quick, instinctive call than a deep think (a System 1 thing rather than System 2), so the other idea was to skip the second LLM and re-sort the search results with a small, fast model built for that job (a re-ranker). It looked promising, because the right tool was almost always somewhere in the top 50 results but only made the top 3 about 60% of the time, so the search was finding it and just ranking it too low.
I tried 3 re-rankers, and the smallest one, MiniLM, was nearly as good as the biggest while running about 20 times faster on my laptop, pushing the right tool into the top 3 about 76% of the time instead of 60% (both measured offline, by replaying the same 50 searches through each ranker).
I put it in front of the agent, and while it did get the right tool to the agent more often with 3 per search (45% to 60%), the number of tasks solved stayed at 30% either way, and with 10 per search it came out a little lower than plain search, losing on the same twin picks as before.
▲ better
I also wanted to try Jev by TypeSafe, a System 1 model built for exactly this kind of pick-one-of-these decision, which might well have changed the result, but I had already burned through too many Claude Code tokens to start another round.
Where this leaves agents
Both fixes went after the search, while the traces kept pointing at the moment
right after it, when the agent is looking at a daybay_create_event and a
daybay_v2_create_event and has nothing to tell them apart but a guess.
Next I’d try a stronger agent, or tool names and descriptions that make twins
easier to tell apart, or Jev, once my usage window forgives me.
lesson
A retrieval number can go up while the thing you care about stays flat, so measure the task end to end before you trust a better-looking search.
Caveats
This is one model (Haiku, inside Claude Code’s own agent loop), 20 tasks, one run of each and synthetic tools, so trust the shape of the curves more than the exact percentages. The drop as the catalog grows holds up statistically, but the differences between plain search, the librarian and the re-ranker are small enough to be luck. My setup also had a bug that failed a few trials at 100 tools for reasons unrelated to the agent, so if anything the cliff is steeper than the chart shows.
Every trial, every failure with a one-line reason and the exact commands to rerun it are in the write-up for this run, so if you think I’ve read something wrong, the evidence is right there.
The code is at jaeaeich/k-onfused, and if you have a tool catalog this big in real life, or you get Jev running on it before I do, I’d love to hear how it went.
Footnotes
-
The smaller catalogs are just the first 100 or 10,000 tools of the big one, so every size is a fair slice of the same menu. ↩
-
Tools are turned into vectors with bge-small, and that model and the re-rankers further down all run on my laptop through Transformers.js, so no search ever left it. ↩
-
None of the tools do anything real, so every call goes to a fake that just checks the arguments with Ajv, and the right tool with a wrong argument still fails. ↩
MCPprotocol
The Model Context Protocol: a standard way for an AI agent to discover and call tools, each described by a name, a description and an input schema.
re-rankerretrieval
A small model that reads the query and each search result together and scores them again, slower than vector search but better at putting the right match first.