The 100-Tool Agent Is a Trap: Scaling with Semantic Routers and JIT Context

Talk transcript

1. 0:01

   Hi, everyone. Thanks for being here. Uh, I am Sohail here, and along with me is Ankush. So, today we'll be talking about a mistake that looks harmless at first, giving-- which is basically giving an AI agent every tool access it might ever need all at once.

2. 0:20

   So, basically that approach works well in a demo. It might even work with, uh, a small number of tools, like, like say for example, ten tools. But once the catalog grows, uh, the agent gets slower, uh, it might become more expensive and less accurate as well.

3. 0:40

   That is why we are calling it "the hundred tool agent trap". In the next half an hour or so, we'll show h-why it breaks, what the numbers look like, and how semantic routing with just-in-time context can help us fix this problem.

4. 0:57

   So, a quick introduction about myself. I am Sohail Shaikh. Um, I'm currently working as a data scientist with Prosodica. My background spans across AI, NLP, marketing, analytics, and even engineering.

5. 1:13

   My current focus is on applied AI, NLP, and conversational intelligence, along with RAG systems. I'm especially interested in making AI systems more reliable, measurable, and even scalable beyond the demos in production.

6. 1:31

   And I'm Ankush Rastogi. I work as a senior data solutions engineer at Prosodica. I have spent more than a decade in AI, data engineering, and production systems. My focus is the engineering side, so it's not about what's gonna work in notebook, but whether it's going to survive in with real load,

7. 1:56

   real user, and real failures. So that is the angle we are taking today.

8. 2:04

   Sohail will focus more on the model and routing behavior, and I will focus more on system design, implementation, and production trade-offs.

9. 2:15

   Awesome. Thank you, Ankush. Um, so let's get into it.

10. 2:20

    So let's imagine a common design. Um, you tend to build a system, and it can do many things. Say, for example, querying a database, sending an email, um, even checking a calendar or looking up an order, calling an API, and so on and so forth.

11. 2:39

    The simplest approach over here would be to give a model every tool definition on every request. Every function name, um, every description, and even every JSON schema will go into the prompt, whether the user might need it or not.

12. 2:55

    So that, we are calling that as a FAT agent. At small scale, it feels fine. With ten tools as well, uh, the model might usually pick the right one.

13. 3:05

    The demo looks good. Then the product grows. Um, ten tools might become thirty, or it will keep on increasing, and eventually the model starts calling the wrong function, con-starts confusing similar tools, may invent tool names, and even take longer to respond. [clears throat]

14. 3:28

    The important point is basically the design does not fail because one tool is badly written. It fails because every request is forced to carry the entire catalog. So, let's look here.

15. 3:42

    There are, say for example, uh, seven hundred and forty-one tools in your, uh, in your entire schema, but-- and it will lo-basically take up to hundred and twenty-seven thousand tokens just to have all those tool descriptions in it.

16. 4:00

    And this is even before the user-user's actual question is even considered. So, basically this will lead to context overload, and, uh, we need to manage that properly.

17. 4:15

    So, on this slide, we see why it is failing and why the accuracy collapses beyond a point. So, when you look at the accuracy curve, with the ten tools, FAT agent will get the tools right almost seventy-eight percent of the times.

18. 4:32

    That is not perfect, but it's usable. At almost a hundred tools, the accuracy drops to around forty percent. Less than half of the tools that are called are the correct tools.

19. 4:45

    And if it grows beyond that, like say for example, in, uh, over here, at seven hundred and forty-one tools, the accuracy will be m-a mere thirteen point six percent.

20. 4:55

    So, in short, it's roughly one correct tool out of eight tools. So, when we compare it with the semantic router, semantic router behaves very differently. It stays above eighty-three percent across the same catalog sizes.

21. 5:11

    That is because the model is not choosing from hundreds of tools. It's choosing from a small and relevant set. One reason the FAT agent fails is because it's lost in the middle problem.

22. 5:24

    Model pays stronger attention to the beginning and end of the long context. When hundreds of tool schemas are packed into the middle, the model does not reliably use them.

23. 5:35

    So we end up pay, uh, paying for a huge prompt, and that prompt makes the decisions even harder.

24. 5:45

    So, two more reasons over here. [clears throat] First is the latency, and the second is the cost. So we saw, uh, in the earlier slide that accuracy was a big problem.

25. 5:56

    Another issue is with latency and cost over here, as we can see. So, like say for example, if we have seven hundred and forty-one tools, we saw that it requires almost hundred and twenty-seven thousand tokens.

26. 6:11

    Uh, that will include the tool description and the schema text. So that cost is basically being paid on every request. At 100,000 requests a day, if you push this into production, uh, you are sending in billions of tokens just to describe those tools.

27. 6:30

    And with just-in-time routing, the prompt may include only three to five relevant schemas, closer to about 1,000 tokens. That is roughly a 99% reduction in tool context tokens. Another issue is latency. [clears throat]

28. 6:46

    With FAT agent approach, time to first token grows as the tool catalog grows, because the model has to process a larger prompt before it can answer the user question.

29. 6:57

    So, say for example, if you have 500 tools in your agent, the FAT agent path can push first token latency past five seconds. So this is specifically important if your product is real-time, because this will take a longer time to respond, so, which will make the design feel slow and unpredictable.

30. 7:22

    So, and with, uh, semantic routing, it might feel more responsive, and it will be more real-time in a way.

31. 7:33

    Here is the clean comparison. So on the left side, we have a FAT agent design, and on the right side, we have a semantic router design. So

32. 7:45

    in the FAT agent design, every schema is loaded for every request, no matter what the user asked.

33. 7:54

    And because of that, the catalog will grow, the prompt will grow,

34. 8:00

    due to which the latency will rise and accuracy will drop.

35. 8:04

    The agent also becomes one large monolithic, which makes it hard to test, risky to update, and even painful to debug. But in the case of semantic routing design, the agent does not start with every tool.

36. 8:24

    So in this case, the router first look at the user's query. It retrieves the three to five most relevant tool, and then it injects only those into the model call.

37. 8:39

    So the context stays small, latency stays stable, and the accuracy holds because the model is choosing from a focused list instead of giant catalog. This is-- Okay, so there is, uh, one important caveat here.

38. 8:56

    If you have fewer than 20 tools, a router may be unnecessary. Just load the tools directly. But once you passed 50 tools in the production system, then just defining router-based, uh, schema make more sense.

39. 9:19

    Thanks, Ankush. So let's now look into how semantic routing works. If you have built a RAG system previously, this will feel pretty similar to that. The difference is that we will retrieve tools instead of the documents.

40. 9:36

    So at the first stage, say for example over here, each tool will have a clear description. For example, a search flights tool, uh, a check calendar availability tool, or retrieve customer order status, say for example.

41. 9:51

    The second step would be those descriptions needs to get embedded and stored in a vector index. This is usually done offline when the tool catalog is created or updated.

42. 10:06

    At the runtime, the user will ask a question. We embed the user query with the same embedding model. Then we search that vector index for the tools with the description that are closest to the query.

43. 10:23

    So the router will then return the top K tools. Often K would be three or five tools that matches the user query, and only those selected schemas will get injected into the model call.

44. 10:38

    So this is the whole pattern. The-- At the first step, you need to index tool descriptions offline,

45. 10:45

    then retrieve relevant tools at runtime, and keep the model's context focused. So in short, the idea is pretty simple. Semantic routing is basically a RAG for tools. Um, if you already have an embedding model and vector database in your stack, most of the infrastructure is already familiar to you.

46. 11:10

    Thank you, Sohail. Now we will deep dive into just-in-time context injection. So semantic routing is the retrieval layer,

47. 11:22

    and just-in-time context injection is the context m-management strategy. So in the case of FAT agent approach, it loads everything before the query is understood, due to which the model receives a huge list of tools first, and then it tries to reason through it.

48. 11:45

    Just-in-time context does the reverse. It will wait until the query is known, then it will inject only the context needed for that request.

49. 11:58

    I mean, this is not a new software idea. We have used lazy loading, just-in-time compilation, and on-demand resource loading from years. We are just applying the simple and same principle to the LLM context.

50. 12:16

    Anthropic has written about this pattern with on-demand tooling loading through MCP. Their report token usage went from one fifty K tokens down to two thousand, which is ninety-eight point seven percent token reduction.

51. 12:36

    That is the signal. Large tool catalog should not be dumped into the, into every prompt. They should be retrieved when needed.

52. 12:47

    So moving to the next slide, let's talk about how to evaluate this fairly.

53. 12:56

    We measure four things here: tool-selection accuracy, time to first token, input tokens per request, and the estimated cost per thousand call.

54. 13:09

    So here, for datasets, we used Berkeley function-calling leaderboard, skills bench-style scenarios, and synthetic tool pools that allow us to scale the number of tools, so we can test at ten, fifty, hundred, two hundred, or even seven forty-one tools.

55. 13:31

    So we ran the same queries in two modes, the FAT agent one and the semantic routing. Same model, same answer key, same tool catalog. The only difference was whether the model saw every tool or only the routed tool.

56. 13:49

    We also swept K at three, five, ten to understand the trade-off. So the smaller K is the faster and cheaper. Larger K can be re-

57. 14:02

    can recover more edge cases. In practice, K equals five is a strong default starting point. So if you are--

58. 14:11

    want to try this, you can default it to five. Here, K means the number of tools the semantic router retrieves and gives to the LLM for each user queries.

59. 14:26

    Here are the benchmark result that we got. So here is the re-result. On the left side,

60. 14:35

    we see the FAT agent accuracy line drop sharply as the tool count grows. It started around seventy-eight percent at ten tool, and then falls to around thirteen percent at seven forty-one tools.

61. 14:51

    But in the case of router, the router line stays above eighty-three percent because the model stays stable. From the model points of view,

62. 15:04

    it always choosing the handful of tool, even if the real catalog contains hundred.

63. 15:13

    On the right side, time to first token shows the same story. The FAT agent path gets slower as more tool schemas are added. At large catalog sizes, the model spend noticeable time just processing the prompt.

64. 15:34

    But in case of router, it stays almost flat because of the prompt size is under control.

65. 15:43

    This is the core lesson from the benchmark. The catalog can grow, but the model's working set should stay small.

66. 15:54

    Now, in the next slide, we'll be discussing about the implementation pattern.

67. 16:00

    There are three steps in it. Step one is offline. So build a catalog of tool. For each tool, store the name, description, schema, then embed the description and store

68. 16:18

    it into a vector database. You can use ChromaDB, you can use Pinecone, you can use Qdrant, whichever vector database you are using. Then step two will happen on every request.

69. 16:33

    Embed the user's query, then run the nearest neighbor search and return the top K tools.

70. 16:42

    In the step three, it will also happen on every request.

71. 16:48

    Fetch the schema for those selected tools, and then pass only those schemas into the model call and log which tool was selected.

72. 16:59

    So here, logging really matters. So let's say if you miss something, and if you want to improve the description or you wanna tune the K, you need l- a good logging system that you can use it.

73. 17:16

    So the runtime overhead is small, one embedding call and one vector search.

74. 17:24

    The payoff is much smaller prompt and more stable tool-selection problem.

75. 17:33

    Awesome. So now that you have learned about, uh, the implementation pattern, I'll take you through the code version of it, which is s- pretty straightforward. So initially, you need to loop through the tool catalog, and for each tool, as Ankush mentioned, embed the description, store that in the vector database with the tool name and other details.

76. 17:57

    At runtime, embed that user query, compare that query vector against the stored tool vectors using cosine similarity or vector search, and then retrieve the top re- most relevant tools, and then call the model only with those schemas in the function or tool parameter.

77. 18:19

    That final line is the most important part Because the model does not receive all the tools, it only receives the routed tools. This works with any embedding model and at-- and with any vector database.

78. 18:31

    You can start, as Ankush mentioned, you can start with, uh, Qdrant or any other mo-- uh, database, vector database locally, and then move to a ma-more managed vector store later if needed.

79. 18:44

    If you already have an existing RAG infrastructure, this is not a new infrastructure, as we mentioned earlier. It is the same retrieval pattern applied to the tool selection.

80. 18:56

    Thank you, Sohail. Uh, now we will discuss semantic routing in action. So imagine you have a two hundred tool agent.

81. 19:07

    A user asks, "Find me a flight to New York next Wednesday." The router embeds that query, returns tool like search flights, book flight, calendar check. The model sees only those relevant schema, so it is much more likely to call the correct tool.

82. 19:29

    But if we load all two hundred tools, portals tool, weather tool, emails tool, sequels tool, or any unrelated workflow tool,

83. 19:40

    it would all be competing for attention. That is when model starts choosing the wrong function.

84. 19:49

    Now, let's take a second query. "What is the weather in Paris right now?" The router returns get weather and get forecast.

85. 20:00

    Flights tools are not injected. Portals tools are not injected. They are simply not available at this request.

86. 20:10

    This is an underrated benefit, but the router does not only add the right tool, it also removes the wrong tool from the model's choice set.

87. 20:22

    Now, in the next slide, we will discuss implementation checklist. So here is the production checklist if somebody wants to, uh, follow this along. So first, catalog the tool in one place.

88. 20:39

    So add name, description, schema, owner, and the version.

89. 20:45

    Second is to build the index. Embed each description and store the vectors. The third one is write the router. So in this, embed the query, then search the index and take top K and fetch schema.

90. 21:06

    The next one is the fourth one, in which

91. 21:10

    a user need to wire it up into the agent loop. So the model tool list should come from the router, not from a hard-coded full catalog.

92. 21:22

    Fifth is to evaluate. Run your test set at K equals to three, five, and ten, and then pick the smallest K that meets your accuracy target.

93. 21:36

    Sixth is to monitor production. Log selected tools, final tool cause, failure,

94. 21:45

    and fallback usage. Then re-embed tools when descriptions or schema change. This is not the six-month platform rewrite. For most team, it is a focused sprint.

95. 22:00

    So, in the next slide, we will discuss what community already has confirmed.

96. 22:09

    This is not just observation. So Anthropic has published on-demand tool loading with MCP and reported a drop of one fifty K tokens to two K, which is a huge reduction in tokens.

97. 22:25

    Developers have raised similar issues in mainstream agent and SDK projects, pointing out that sending all tools definition on every request can increase latency, token usage, and tool confusion.

98. 22:44

    Open proj-- uh, open source projects like MCP Zero have also explored routing at very large scale, including thousands of tool across many servers.

99. 22:59

    And you can find form-- forums post from builder who hit tool confusion with only few tools, long before they reach hundred. So if you, if your agent start failing as tool are added, it doesn't automatically means your prompts are bad.

100. 23:20

     It may means the architecture is asking the model to solve the wrong problem.

101. 23:30

     So here we are just trying to say about the op-- we are going to talk about the open source projects that we came across. So one of them is, uh, from Aurelio Labs.

102. 23:43

     They have an open source semantic router project that you can easily use to test locally, uh, for benchmarking. Tool Bench and the Berkeley Function Calling Loadup-- Leaderboard are also useful starting points.

103. 23:57

     And, uh, for practical guidance, you could read the Anthropic MCP writeup. It explains why on-demand tool loading matters and gives a strong real-world signal for this architecture. The point is that you don't need to invent the ecosystem from scratch.

104. 24:14

     Those pieces already exist. Next slide.

105. 24:24

     Sorry, here it is.

106. 24:27

     Thank you. So there are a few trade-offs with this. The first risk could be a router miss. So if the rou- so the router may fail to retrieve the tool the models need and handle that with-- you could handle that with a feed fallback.

107. 24:45

     If the model cannot complete the task, you could widen K, uh, run a second retrieval pass or even route it to a broader tool group. The second risk would be weak tool descriptions.

108. 24:59

     If your descriptions are vague, embeddings will end up being weak. Write descriptions in the words users actually use and include intent, action, and key entities along with it. The third risk would, over here, would be rare tools.

109. 25:17

     So some tools may never score high unless their descriptions include the right language. So you need to monitor misses and rewrite those descriptions accordingly. And, uh, finally, just to conclude, do not over-engineer small systems.

110. 25:33

     If you have ten or fifteen tools, static loading may still be fine. Routing generally pays off when the catalog is large enough that the prompt size, latency, or tool confusion becomes a real production problem.

111. 25:51

     So a few takeaways. First thing is, as we discussed, tool overload hurts accuracy. Loading every schema into every prompt makes the model's decision harder as the catalog grows on.

112. 26:07

     The second one would be to... about tokens. It's like tokens are both cost and latency. Larger tool catalogs can, uh, add tens of thousands of tokens before the user's actual request is handled.

113. 26:24

     Semantic routing can help fix that problem. The, when the catalog can be, is large, the model only sees the tools relevant to the current query. Um, this is basically a RAG for tools.

114. 26:40

     Index those descriptions properly, retrieve at runtime, and inject only what is needed. And finally, start simple. Use K as five as a starting point, log every decision, and evaluate against a real test set, and improve the tools' descriptions over time in case you get

115. 27:04

     incorrect tools being, uh, picked up. And the goal is not to make these agents more complicated. It's, it is basically to stop forcing the model to reas-reason over irrelevant tools.

116. 27:20

     So there are some references on this slide. So if you're building something, you can use these references,

117. 27:29

     like semantic routing libraries, Tool Bench, Berkeley Function Calling Leaderboard, Anthropic MCP write-up, and vector stores like Pinecone, Qdrant.

118. 27:43

     Thank you so much for watching this. If this was useful, please share it with someone building agents. They have either already hit this wall or they're about to.

119. 27:55

     You can scan our LinkedIn QR codes on this slide to connect with us. We are happy to continue the conversation, share resources, or hear how others are approaching this problem in their own system.

120. 28:11

     Thank you so much for watching this. I'm Ankush.

121. 28:16

     This is Sohail here. Thank you so much.

122. 28:18

     Yeah. Thanks, everyone.