Tool Calling Is Not Just Plumbing for AI Agents

Talk transcript

1. 0:04

   Tool calling is not just plumbing for AI agents. In this talk, I'm going to show you why tool calling is more important than some people think and how you can use tool calling with your agentic frameworks.

2. 0:14

   We'll be looking at some of the techniques for tool calling, such as regular tool calling or meta tool calling, how this works with agent frameworks, and also some other considerations like do you need a separate tool platform or can you look at different types of tools rather than the tools we see today.

3. 0:32

   If you think about plumbing, uh, some people say you should do your own plumbing, some other people say you should always hire a professional. I hope at the end of this talk you'll be feeling like I can build my own tools.

4. 0:43

   I know how to create tools separate from my agentic frameworks and bring them in, and thereby get a lot of flexibility like changing agentic frameworks or whatsoever. If you think about agents in general, I feel we spend a lot of time on improving the agents, but we don't seem to spend a lot of time on building the

5. 1:00

   tools that are reusable, robust, and can be put into different frameworks. I feel it's changing slightly. There are tons of tool platforms popping up. There are more libraries to build tools, and I also feel the, the pressure is sort of off the agents.

6. 1:14

   People know agents can do certain things, and now they're all trying to improve the tools these agents have.

7. 1:21

   So my name is Roy. I always worked for startups or founded my own startups. Uh, a few years ago, I ended up at IBM after an acquisition. I've been doing tons of talks on GraphQL and web development and, um, also React.

8. 1:37

   So you can find some of my books or previous talks on the Internet. Um,

9. 1:42

   also feel free to connect with me on social media in case you, you want to know more.

10. 1:49

    As I mentioned in the beginning, everyone is talking about agents, but not so many people are talking about the tools these agents need. If you look at a typical agent loop, there's always a user asking a question or sending a prompt.

11. 2:01

    There's the agent itself, which needs large language models, which needs a bit of memory. And then of course, there are tools.

12. 2:12

    But way less people are talking about these tools. So everyone is talking about the agents. What agent framework should I use? They're probably using the tools that are provided by the agent framework, or they're writing the tools directly inside of the agent framework by extending some of the functions they have or interfaces they have to create tools.

13. 2:31

    Why are people not really talking about the tools? I always feel like every time I'm building an agent, first thing that breaks is the tools. Either the large language model is not able to call the tool correctly, the large language model uses the wrong tool, uh, maybe something inside my tool breaks because I didn't set up something

14. 2:48

    properly. The agent feels like, like a closed circuit where the tools are more dynamic.

15. 2:55

    So at the end of this talk, I hope you, you feel like the agent is only as good as their tools. Frameworks do so much for you, but they don't do that much for you in terms of building the tools.

16. 3:06

    Someone once said, "A man is only as good as their tools." Uh, so let's make that the agent is only as good as their tools. Make sure you have tools that are reusable and robust, and you can bring these into any other agent framework.

17. 3:21

    The LLMs advanced a lot over the last few years, and that's why I also feel the application layer, especially the part where you build your tools, deserves more attention.

18. 3:30

    If you look at some of the things people are saying on the Internet, uh, it's actually saying like the wrappers around the GPTs or wrappers around models, wrappers around the chat interfaces, they're way more, um, they provide way more options for improvement later on.

19. 3:47

    And I also feel that's where we can do a lot more by writing software, by coming up with smart solutions and doing things on top of the models. At some point, the models are gonna...

20. 3:56

    Well, they are gonna becoming increasingly better, but at some point, we need to have the application layer catch up as well.

21. 4:04

    So let's quickly recap tool calling to make sure we're all on the same level here. Um,

22. 4:11

    for example, imagine you're asking a question to an agent like, "How far does the Moon orbit express as the number of trips between Amsterdam and San Francisco?" So half a year ago, so I moved from Amsterdam to San Francisco, so I was actually kind of interested in the answer to the question.

23. 4:27

    If you look it up on your favorite chat application, uh, you will find the Moon's orbit is about forty-four round trips between Amsterdam and San Francisco. So sometimes the model might be able to come up with the answer itself because there's training data which includes the distance between, uh, the Moon and the Earth, and then a distance

24. 4:45

    between Amsterdam and San Francisco, and it's able to find the answer for you. More probably, it's going to do a set of tool calls. So it's going to search for the distance between the Earth and the Moon.

25. 4:55

    It's going to search for the distance between those two airports. And then it's going to do some calculations. Calculations, maybe it needs a tool, maybe it can do it on its own.

26. 5:04

    This question is fairly simple, but maybe you can make it more complex by asking, "How long does it take me in number of days?" Or, "Is it quicker to have a Moon orb-orbit, um, or do those forty-four round trips?"

27. 5:18

    One trip is like ten to twelve hours. You need to wait at the airport. You need to, uh, fly back every time. So you probably need a bit more than this simplified example.

28. 5:30

    Of course, you can come up with the, the answers for these by doing tool calling or looking at training data. If it's doing tool calling, it's most probably looking at external APIs.

29. 5:39

    So it's going to do a web search, or it's going to go to some database with geographical data. If you do the calculate, maybe it's using JavaScript functions or Python functions, or maybe it's going to an external API like Wolfram Alpha.

30. 5:53

    All of this needs to be implemented, but how you define the tools matters a lot as well. So before I'm going to show you implementation, uh, let me tell you why defining the tools matters a lot.

31. 6:04

    If you look at a typical tool definition, you have your tool name. I always advise to keep this simple. You have your tool description, and this to me almost feels like a system prompt for the large language model.

32. 6:14

    So your tool description typically doesn't just say calculate something based on two incoming variables. If you look at some of the bigger tools, almost like agentic tools, you can see that the tool descriptions are fairly long.

33. 6:27

    They're almost like system prompts. Sometimes you see people duplicating system prompts, uh, duplicating tool descriptions prompts. So there's a lot of things you can do there, but the tool description really is important.

34. 6:39

    Then you have the input parameters, of course, because the model needs to know what is needed in order to call this tool. And then something I really like, and I see this more and more in agentic frameworks, is the output schema has become an input variable as well.

35. 6:54

    This will make you help-- This will help you to make tools type safe. As I mentioned, I'm a big GraphQL guy, or at least I was before I started to work a lot with agents.

36. 7:06

    If you don't have an output schema, how is the model going to know what data is being returned? Um, it's always a string, though. But if you're building sequence of tool callings, if you're building more complex agents, you probably want an output schema, so you can force structured outputs or you can chain tool calls.

37. 7:21

    So the output schema, my feeling, is getting more and more important later on.

38. 7:26

    And then the tool call will look something like this. Uh, I believe this is LangChain. Um, could be CrewAI as well. So you would have your tool definition. Whenever the tool is being called, it needs the name and then it needs the arguments.

39. 7:38

    Say you want to get the count of customers, you probably need to have filter like country, but then the customer count will return a string which includes customer. It might also be, um, a string which is just JSON that you decode.

40. 7:54

    And this also works for other languages, uh, like SQL. So I'm going to show you a bit about dynamic tools later on. So we've did some tests with GraphQL and SQL and using those to create dynamic tools.

41. 8:05

    So instead of getting the count of a customer, you could also give it access to a database, give it access to an API, then have the model create the query like query or GraphQL.

42. 8:17

    But let's also look at where the tool is being called from because this is getting more important as well. As I mentioned, people are building their tools at the same place where they're building their agents usually.

43. 8:26

    So if you look at where the tool is being called from, you would have your agent loop, which has the model, which has the tools, which has perhaps some memory.

44. 8:34

    Where do you implement the logic to define which tool is being called when?

45. 8:40

    I think it was in a blog from Cloudflare where I saw them introducing their Workers API and they mentioned, "We have traditional tool calling and then we have another form of tool."

46. 8:50

    I kind of like that they put traditional in there because, I don't know, we've been doing this for like two years, maybe three. So it alwa-already feels like weird to call something traditional while time is moving so fast.

47. 9:02

    The traditional tool calling is... And don't mind the chaos on this screen. You would have your client application, you would have your application which has the, the agent, where you have, you define your tools, where you're doing your server side logic.

48. 9:16

    So assume you have my question like how far does the Moon orbit expresses in trips between Amsterdam and San Francisco. If you would type this from your client app, which might be a chat interface, it goes to your agent or AI application.

49. 9:29

    It's going to put this in a prompt. It's going to send it over to the model together with some...

50. 9:36

    Model is going to tell call. So somewhere between your client app and your server app, you're going to define

51. 9:43

    tool calls. So you're going to look at the incoming recommendation from the large language model. You're going to call those tools based on the tools you have defined in your server app or your agent.

52. 9:54

    In here, you would have some callback function, call them, and then you give back the tool response to the model. And at some point the model is going to do this a couple of times because

53. 10:04

    calls. And finally, you will have your answer which you can display in your client app. So there's a lot of back and forth between the application where you implement logic and the large language model.

54. 10:14

    And then between your client app and the agents, there's a lot of logic in there as well because it's not a closed system. It's the agent and then within the agent you need to define tool calling logic.

55. 10:25

    This is what is called be-being traditional tool calling. Um, if you implement this in something like LangChain like this, you would define your tools, then you set the callback function for a tool.

56. 10:37

    You can see we have our models here from IBM's watsonx system. And then you set... You look at the answer. So you look at the return from the large language model, and there you're going to filter the tool call messages.

57. 10:49

    So you need to explicitly look for a message that has the right role, and you need to make sure that whatever's in there is being parsed and it's being executed based on the callback function you have.

58. 11:00

    Then you need to handle all sorts of things like getting the tool calls, handle retries, handling errors, all these kind of things that you need when you're building any...

59. 11:10

    But this is how it all started. This is what agentic looked like two years ago, maybe three.

60. 11:18

    If you look at what it looks like today for most, most frameworks, it's something people like to call embedded tool calling. So this is where the system is a closed, closed system.

61. 11:27

    You pass your tools almost like a black box and what comes out is the answer.

62. 11:33

    If you look at this example again, you would have your agent. This time your client app will only ask a question like how far does the Moon orbit? It's going to go to your agent.

63. 11:43

    Your agent is a closed system. It has the tools. It's connecting with the large language model. It's connecting with the tools. It's doing the tool calls. Finally, you get the answer back like the Moon or Moon's orbit is about forty-four round trips between Amsterdam and San Francisco.

64. 12:00

    So this is what's being called embedded tool calling because it all is executing on the right side of this screen. The client app or the application logic you write doesn't include any of the tool calling logic.

65. 12:11

    It's all being handled by the agent framework or wherever you have your component running. The tools in there, they're all defined in the same agent as well, and this is all doing the logic.

66. 12:21

    So it's really is a black box. You send in your question with a prompt and a tool, and what comes out is the answer. No control over whatever is going on inside of it.

67. 12:32

    If you look at LangChain again, uh, so stick to the LangChain examples for now, you import all the different LangChain interfaces, you import your model, you connect to the model on your tools at a prompt, and then you pass it into this create React agent function.

68. 12:48

    This will take your tools, model on the prompt, and then the agent executor is going to make sure it's executed. If you compare this to the previous screen, it's the same functionality, but this time it's all had, it, it's all hidden within that create agent function.

69. 13:02

    It is what people like to call embedded tool calling because it really is a black box where tools come in and the answer comes out.

70. 13:11

    Let's say this is easy to implement. If you're getting started building agents, you don't need to worry about errors, retries, any of that stuff, uh, but you don't have any control over the tool calling process.

71. 13:22

    You don't really know how the tools are being executed. You don't know how any of the decisions are being made. You don't really control any of the formatting other than the callback function you provide together with your tool definition, which of course has the description with sort of system prompt for the agent to understand how to use

72. 13:39

    the tool. As a longtime developer, I'm always looking for separation of concerns. Uh, I'm not saying you should build microservices or micro-frontends, uh, but I do want to keep some systems a little bit more separate.

73. 13:54

    I, I started developing at one point in PHP and then tools like, um, well, tools, it's really a framework like Laravel, and everything was sort of connected there. You had your, uh, your sort of backend code, you had your frontend code.

74. 14:07

    The frontend code had ways to, to render view. Uh, I believe it was View or maybe it was even jQuery in the beginning. It was all a closed system, which I didn't really like.

75. 14:16

    So at some point, I started to detach my backend from my frontend. I used Laravel to build the APIs, and then I connected it to frontend, like Angular or React, or maybe it was jQuery, I don't really remember.

76. 14:30

    Um, so there's a lot of thing you can do to separate concerns without making things too big. You don't need microservices, micro-frontends per se. Some separation between different parts of your system is something I always like to prefer.

77. 14:42

    Uh, and you can keep it all in the same repository. Put it in different, different big repository.

78. 14:49

    The MCP is one good step in this direction. If you didn't hear about MCP yet, it's a protocol introduced by Anthropic and now being adopted by, uh, many more people in the industry as a way to separate the client side and the server side of building agentic applications.

79. 15:08

    The MCP stands for Model Context Protocol, and this is where you have your host, uh, which also has a client in there. So the host could be Cloud Desktop.

80. 15:17

    Uh, it has a client in there that's able to connect to servers. And the servers, think of a server as a backend that has access to tools or, uh, assets like data files.

81. 15:27

    So the server is the only thing the, the host and the client see. It doesn't really see the tools per se, because the tools are made available through the server.

82. 15:35

    But think of the server as a small backend that is able to give you any results from tools. So I really like this separation of concern because now there's a clear distinction between the front side of the house and the back side, and the MCP server is where you handle the logic to define your tools, import your

83. 15:52

    tools, and then the MCP host and client, they understand how to call these servers. I wrote a small blog post on how you can get started building these MCP servers using TypeScript, so feel free to, uh, to check that out.

84. 16:05

    The link is probably a bit too long, so if you go to my website, you can find the, uh, blog section there. MCP is really exciting. I guess it's a good step in the direction of making, uh, tool calling really separate from the agentic framework.

85. 16:19

    There's much more you can do. There are also other considerations, and I hinted at this in the beginning. Uh, you can, for example, look at a standalone tool platform.

86. 16:30

    And this is something that's getting more attention in the market. I think there are a couple of Y Combinator startups as well that are jumping into this space. So a standalone tool platform means that instead of having a closed agent loop where you define your tools inside of the agentic framework, you define them separately.

87. 16:47

    And then inside of your agentic framework, you can import those tools by writing an SDK or by doing an API call. Your tools are on remote servers. The tool creation and the tool, tool hosting or execution is really being done separately.

88. 17:01

    The only thing the agent does, it, it takes in the tool definition and uses the LLM to decide what tool to call, and then it just passes this on to the tool platform where being executed.

89. 17:12

    Tools might be chained there. Tools might be abstracted away. There's a lot of things you can do in such a platform. As I mentioned, as a developer, I really like to separate concerns.

90. 17:23

    The tool platforms allow you to easily create tools out of APIs or databases. So typically, these platforms of, of two things, you can call them a framework as well, I don't really mind.

91. 17:35

    They consist of two things. They have a place to create the tool, and then they have a place to, to execute the tool. And the tool creation is usually done via code or via CLIs, and then the tool execution or the tool surfacing is being done via SDKs that you connect to LangChain or CrewAI or whatever agentic

92. 17:56

    framework you might be using. In there, you can also do things like chaining tools. So sometimes you can see repetitive patterns in agents. If you want to get the number of customers in a certain country, maybe you need to do a tool call first to retrieve the country of the person asking the question based on their IP

93. 18:13

    or whatever. Uh, then you want to chain that to another tool call to get the actual customers out of your database or CRM system. So this chaining of tool calls is something you can do in these platforms or frameworks.

94. 18:25

    And this is also where you can handle authorization and errors and these kind of things. If you think about tool calling and, and authorization and authentication, um, you can easily imagine that you have different systems you're connecting.

95. 18:37

    Let's say it's a CRM or let's say it's a database. They require different sort of credentials. Uh, where you set those credentials and how you pass them to these underlying systems, it's, um...

96. 18:48

    well, it could get messy quite fast. And the question is, do you wanna...

97. 18:53

    do you wanna put all of this in your agentic framework or do you want to separate it into different systems? This separation also allow you s- allows you to be flexible.

98. 19:01

    So imagine you have a tool platform where you build your tools. So building the tools usually is the thing that takes me the longest because for all the reasons I mentioned.

99. 19:10

    If you want to take these tools and bring them into LangChain or LangGraph, if there's-- if it's as easy to bring those into CrewAI or bring them into AutoGen, then it really becomes easier for you to switch those different frameworks.

100. 19:22

     Build your tools once and bring them into different agentic frameworks so you get all of this flexibility.

101. 19:29

     As I mentioned, there is a couple of tool platforms in the market. I'm a bit biased here because at IBM I'm actually building one. Uh, but there are other great ones as well, like Composio, Toolhouse, Arcade AI, which is, uh, run by a couple of people I met, and then Wildcard, which recently got into Y Combinator.

102. 19:47

     If you look on the right, you can see what the, the servicing part of a tool platform might look like. So I'm bringing in my model. In this case, uh, Watson X models from IBM.

103. 19:56

     I'm then creating a LangGraph ReAct agent. I'm bringing in my tool platform, which is called WX Flows. You can see there is a specific SDK integration for LangChain. I'm connecting to the tool platform, so I build my tools in one place, and then I'm connecting to these tools from my agent framework.

104. 20:12

     So I need an endpoint, an API key, but you can also run this locally and then just pull in the, uh, the functions or interfaces that way. You retrieve the tools, so you need to get the tools from one place.

105. 20:23

     And whatever you pass to that agent, that black box, is still the tools, it's the model. But this time, the tool execution is being done on that remote end.

106. 20:31

     So instead of passing in callback function, you pass in a way for the model, or in this case the agent, to actually call these tools. So now you have separation of concerns.

107. 20:41

     You have your tools in one place and the agent in the other. You're still doing a meta tool calling where the agent is making sure the tools are being called and things are working as expected.

108. 20:50

     It's calling subsequent tools. But the entire execution is now being put on the tool platform or framework. So I really like this because now I've pulled the two pieces apart and I can easily substitute LangGraph with something else.

109. 21:05

     Or I can use different interfaces. Let's say LangGraph is gonna

110. 21:09

     create different ways to create those black box agent, just ReAct, maybe also other patterns. Or I can use their graph based approach. It all works in the same way.

111. 21:18

     Uh, I still have tools in one place and agents in another place.

112. 21:23

     And then I hinted at this in the beginning, what about dynamic tools? So I did some tests with running GraphQL or SQL with some of our clients. And one thing we really saw is you could spend a lot of time creating all your tools, but you also don't want to create a million tools.

113. 21:37

     Let's say you have a CRM. You don't want a different tool for each of the different components of this. You can get your customers. You don't want to have customers based per country.

114. 21:48

     You want to have customers based on, I don't know, whatever search filters you might have. Next to customers, you also have orders maybe in the CRM. You have contracts.

115. 21:58

     You have employee details. Whatever you can have in a CRM or in a database. So let's say you have a database with product data or order data or payment data.

116. 22:08

     You don't want to create all these different tools. You... if you run some tests, if you put in like ten tools, it's probably fine. If you're going to paste in a hundred tools to the agent, it's gonna get really confused.

117. 22:19

     So you're going to need routers and all these things on top of your, uh, your tool definitions. But these tool platforms might be able to handle it. Maybe some agentic frameworks are able to handle it.

118. 22:28

     But another option is looking at dynamic tools. And I'm a big GraphQL guy, or I used to be mostly. Let's say you can also create a dynamic tool. This is where you create a tool that's able to connect to the GraphQL schema.

119. 22:42

     And then instead of passing in arguments that are sort of fixed, like filter, like get me the customers from, from Europe and your filter would be Europe. Instead of doing this, you pass in a GraphQL query.

120. 22:55

     Or you don't really pass it in yourself. You have the large language model create this for you. So what you give to the model is you give the model tool and you say, "This is GraphQL.

121. 23:03

     You need to generate valid GraphQL to do so."

122. 23:06

     Then you pass in the schema. So the GraphQL schema is usually it's kind of small and it's easy to read for humans and thereby easy to understand for large language models, which has your, your type definition, so it knows what's coming out.

123. 23:19

     And then it also has a list of all the available operations, get customers or get orders or get customers by whatever. We found out by testing this with models like Llama or OpenAI, OpenAI's models, but also Claude is really good at this.

124. 23:36

     It's really good at generating GraphQL as long as you pass in the GraphQL schema and you don't do too complex things. So if you want to have your custom scholars in GraphQL, you want to get rid of those because it gets confused.

125. 23:48

     You don't want to nest data too deeply. So maybe nesting at one level deep is fine. Nesting like five levels deep is gonna be really confusing. And this is kind of cool because most models actually understand how to use GraphQL.

126. 24:00

     They understand things like fragments. And I can probably spend like an hour talking about just building these kind of dynamic tools on GraphQL or SQL or the query languages.

127. 24:11

     But it's a really easy way to bring in existing APIs and databases to your agent frameworks. You have less implementation, uh, in the downstream implementation. So where you connect your tools, where you bring your tools to the framework, you don't need to define twenty different tools and duplicate your business logic.

128. 24:29

     That you can take the existing business logic and put it directly in here. So this allows for more flexibility, but it also comes at a cost because there are trade-offs.

129. 24:39

     LLMs might hallucinate. Sometimes it could do GraphQL really g- well, and then the other time it does really poorly and messes up all the different syntaxes. So there are some trade-offs here, but I, I see a real feat- future for building dynamic tools rather than the static tools that we see today.

130. 24:56

     So in short, as we're all building agents, let's make sure we don't forget the tools are as important as the agents we're building. Um, so thank you for listening to my talk.

131. 25:06

     If you're at the conference, I hope to see you. If you aren't and you're watching this live, make sure to follow me on social media and connect there. So thanks again and I hope to see you later at some point.