A2A & MCP: Automating Business Processes with LLMs

Talk transcript

1. 0:00

   [electronic music] Hey, everybody.

2. 0:16

   Uh, yeah, thanks for coming. Uh, great to see a full room. Uh, always good when you're doing a workshop to have, uh, a lot of people here. Um, so yeah, I'm, I'm Damien Murphy.

3. 0:25

   Uh, I'm gonna be presenting A2A and MCP, uh, two pretty hot topics these days in AI, um, and how you can use them to automate business processes. Um,

4. 0:37

   so yeah, a little bit about me. Um, about 15 years full-time, uh, full stack developer. Uh, five years doing solutions engineering, so customer-facing, kind of, uh, forward deployed engineer, and, uh, spent the last three years or so, uh, working on voice AI and AI agents.

5. 0:55

   Um, I did a workshop last year as well, um, AI voice agent swarms, and, uh, yeah, it was pretty, pretty hot topic. I think it's now pretty much standard that everybody can build a voice agent in a, in five minutes.

6. 1:09

   Um, so now the, the hard part becomes building autonomous agents that actually can do complex tasks. Um, so I joined Bench Computing, uh, about two months ago. Uh, pre-revenue startup, uh, backed by Sutter Hill Ventures, and we're building, um, what I would imagine to be a better Manus, uh, that's more focused on teams and enterprises.

7. 1:30

   If you're not familiar with what Manus is, it's kind of like a autonomous AI agent. Uh, and Bench is essentially an autonomous AI agent that can do, uh, sub, sub parallel task, um, automation.

8. 1:46

   All right, so the workshop that we're doing today, we're gonna build a multi-agent system, um, using A2A agents. Uh, if you're not familiar with A2A, uh, Google released essentially, um, a protocol that allows agents to communicate over the web.

9. 2:03

   Uh, we're gonna integrate these agents with, uh, MCP, which is the Model Context Protocol. Uh, MCP is like a USB-C for, you know, uh, all of your agents to be able to consume, um, context and tools, uh, and resources, uh, very easily.

10. 2:22

    Uh, we're gonna get these agents to work together, and we're gonna trigger, uh, the, the agent with a webhook.

11. 2:29

    And then, uh, I'm gonna cover a little bit about when to use A2A, MCP, and I'll also go into, uh, prompt caching and context management as well.

12. 2:41

    All right, so A2A, right? Um, it's, it's not exactly clear what it's for and why it exists, right? Uh, if you ask everybody in the room what they think it does or why it exists, you'll probably get a different answer.

13. 2:54

    Um, but the key benefits are you can have agent specialization, right? So rather than trying to make one agent do a hundred things, you can have a hundred agents do one thing, uh, and do that one thing very well.

14. 3:08

    Um, A2A allows you to handle task delegation. So, you know, imagine you had a Salesforce agent, um, and you want it to interact with all the Salesforce MCP, um, uh, tools, uh, you could do that.

15. 3:22

    Um, you've also got the ability to do parallel processing, uh, and this will become very important when it, uh, comes to speed and context management. Uh, you can then use those A2A agents to have, uh, complex workflows and, and help, uh, keep your main agent's context size down.

16. 3:42

    Uh, MCP, again, really hot topic right now. Uh, it's been kinda coined as the USB-C for AI, um, and there's definitely some benefits in just having a standard interface, right?

17. 3:54

    You know, there's something like ten thousand MCP, uh, tools that you can use today. Um, about seven thousand of those come through the Zapier MCP. Uh, if you're not familiar with Zapier, it's essentially a way to connect disparate systems together, and they've now released all of their, uh, Zaps, they're called, as MCP, uh, servers and tools.

18. 4:17

    Uh, one of the great things about MCP, no integration with APIs, so you don't have to do any sort of, you know, different handling of different APIs. Uh, it's a plug-in architecture, uh, an industry standard, um, and it's really based on LSP.

19. 4:32

    So LSP was a way for, you know, IDEs to actually, uh, figure out how different code languages worked. Um, and it was a great kind of, um, transfer, uh, of, of ideas over to the MCP protocol.

20. 4:48

    All right, so when should you use A2A versus MCP? Anybody?

21. 4:56

    MC, if you want to resource the infrastructure, then you go for MCP. But A2A, I don't know. [chuckles]

22. 5:05

    And, and, and that's kind of the, the challenge, right? It's like, what exactly, um, you know, are these protocols for, and should I be using them, and, and things like that.

23. 5:16

    So if you want to have, you know, two agents, right? And typically two agents that are completely unrelated, right? So it's not two agents that you necessarily control. It's more likely gonna be an agent of a third party or a, you know, their first party agent and your agent.

24. 5:31

    Yeah?

25. 5:32

    What's the difference between agentic AI and A2A? So I worked a lot on the agentic AI where we have multiple agents and doing the same task. The way you are saying, describing A2A is a lot similar to

26. 5:50

    Mm-hmm. Yeah, so like AutoGen and, and frameworks like that, that allow you to kind of, uh, manage multiple agents kind of locally. Uh, A2A is more about remote agents, right?

27. 6:01

    So agents you have no knowledge of. Um, so you can think of A2A as a way for you to have service discoverability. Uh, once you have the endpoint to the agent, you can then learn everything that a- that agent's capable of.

28. 6:14

    Um, with things like AutoGen, it's like, you know, descriptive, so you des- describe what it's capable of. Uh, it's in your control.

29. 6:21

    So to summarize, agentic AIs are kind of brand... define the role of each agent, and A2A is kind of working on remotely, and its role is not defined or defined?

30. 6:41

    Y- y- so each of the A2A agents will have a, a definition, and we'll kind of get into that a little bit later. Um, but yeah, think, think of agentic AI kind of as a superset of everything, right?

31. 6:51

    Um, A2A and MCP are just kind of subsets of that, right? Different modalities. Um, yeah, so for MCP, you're, you're gonna connect to external context and tools. Um, a lot of people don't use most of the features of MCP, right?

32. 7:05

    They're just using the tools. Um, but there's a lot of stuff around prompt templates, resources, um, and a thing called, um, sampling. Sa- sampling is actually gonna be a really interesting thing I think that we'll see a lot more of as well, uh, where it allows these MCPs, uh, to sample the host LLM, right?

33. 7:24

    So if you're using, you know, Claude and you're hitting an MCP server, um, that MCP server may want to also use the same model of Claude that you're using, and it can use sampling to actually achieve that.

34. 7:36

    Um, so when you bring those two together, you, you kind of get the benefit of both, right? So you have A2A as the, the remote interface, uh, and MCP is then giving you the actual, um, tool use and, and context management.

35. 7:52

    Okay, so when not to use MCP? Um, and, and you'll notice a lot of like memes here. Uh, and just to give you a heads up, all memes were generated by Bench.

36. 8:01

    Uh, actually, the whole slide deck was generated by Bench. Um, I just gave it a markdown file and it, and it outputted it.

37. 8:09

    So, um, when you use A2A or MCP, um, if you have full control of the tools, then you probably don't need it, right? Like, if, if your function is local to your code base, you know, w- why do you need to create, you know, a USB-C?

38. 8:23

    It's kind of like me plugging in my hard drive with a USB cable. You know? Like, shouldn't I just use the hard drive that's in my machine, right? Um, so calling functions directly in your code base, super easy, easy to maintain, faster to develop.

39. 8:38

    Um, and then if you have full control of your agents, you probably don't need A2A either, right? Um, like if they're your agents, you can use, you know, some sort of local, uh, function call for them to communicate.

40. 8:50

    Um, and I, I've built multi-agent systems using MCP and using just local function calls. It's a lot easier to just use the code you have. Uh, it's gonna be faster.

41. 9:00

    There's no protocol overheads, uh, and, and things like that. A lot easier to debug as well.

42. 9:07

    Okay, so why do you need A2A and MCP at all, right? Um, third-party tools is probably the number one reason, uh, to use MCP. Um, you can just get access to such a large array of tools, um, that, you know, you're never gonna be able to...

43. 9:24

    Uh, let, let, let's say you're building a product, right? And, and you're like, "Okay, we're gonna build first-class integrations with Salesforce and Slack." Um, but what about your 10,000 tools?

44. 9:33

    That's like, "Okay, we'll just allow people to add their own MCP server." Um, so that gives you great extensibility. Um, but there's a lot of drawbacks with MCP, right?

45. 9:42

    Um, you only get what you're given. Um, and a lot of time, that's not exactly what you want. Um, so you may go down the route of saying, "You know what?

46. 9:50

    I need a, a way to actually index this data so that I'm not calling like, you know, list Slack channels every time I wanna post a channel," right? Or post a message.

47. 10:01

    Um, and then with A2A, uh, the com- complexity is hidden from you, right? And that, that's one of the kinda the key tenets of, of A2A, is that you don't know anything about this agent until you connect.

48. 10:13

    Um, and all of its complexity is, is completely opaque.

49. 10:17

    Um, and then you, you can essentially connect to, you know, any sort of, uh, remote A2A agent, um, so long as you have, you know, the credentials and things like that.

50. 10:28

    Um, we haven't seen any first party A2A agents released yet, um, but Google has about, I think, 50, uh, partners they're gonna launch with. So I, I'd imagine there's gonna be like a Salesforce A2A agent.

51. 10:41

    Um, it'll probably only come with a paid account, right? 'Cause it's gonna use LLM compute, uh, versus things like MCP typically don't actually use an LLM, right? They use the host LLM.

52. 10:54

    All righty, so we're gonna get into a bit of code now. Um,

53. 10:58

    yeah, so, uh, if you haven't already grabbed the repo, uh, we also have a Slack channel, um, workshop-A2A-MCP, uh, dash 2025. Um, and in this repo, there is basically every- everything you need to get going.

54. 11:14

    Um, yeah, so the, the code structure, uh, we've got a host agent, um, and then we've got some sub-agents, right? And the whole concept here is to demonstrate, you know, A2A and MCP.

55. 11:28

    Um, but in reality, these sub-agents will probably live in a different repo, you know, run on a different server. Um,

56. 11:36

    yeah, and then we've also got the, uh, A2A implementation, uh, the server and the client in, uh, the repo. Uh, these are taken directly from the A2A, uh, repo.

57. 11:46

    We've also got the MCP integration, so this is just a client. Um, we're, we're not creating a, a server here. And we also have a CLI interface. Uh, you're not gonna need the CLI interface.

58. 11:57

    That's kind of i-internally how it's being used. Um, yeah, so o-once you've cloned the repo, you're gonna wanna NPM install, um, and you're gonna need a MCP server URL.

59. 12:09

    Uh, this is gonna be a Zapier, uh, server URL, and a Gemini API key. Uh, you can get both of these for free. Uh, there's no need to, to sign up for a paid account to get them.

60. 12:19

    Um, and you'll wanna rename your .env.example, uh, to .env.

61. 12:27

    All right, so setting up the Zapier MCP.

62. 12:31

    Um, when you go to, uh, zapier.com/mcp, uh, you'll have the option to create a new server. Um, and when you go to connect, you're gonna have a couple of, uh, options here.

63. 12:41

    We're gonna use SSE. Um, they recently released, uh, Stream with HTTP, which, uh, i-is making SSE deprecated, and it's gonna replace it. Um, but there's, there's still a litany of SSE servers out there, so, um, uh, just used SSE for this one.

64. 12:58

    Um, once you do that, you're gonna get this server URL at the bottom. You can copy that URL. That's gonna be the URL that goes into your .env.

65. 13:07

    And then, uh, you're gonna set up a Slack and a GitHub integration. Um, so you're gonna want the ability to create an issue. Um, you can put in, uh, the repository URL for the workshop if you want.

66. 13:18

    Uh, you can use your own, uh, as well. Um, you can let AI choose, uh, these, but what I've found with AI is that it will choose something else, right?

67. 13:30

    Um, so a lot of time with these MCPs, you're gonna wanna kind of say, "Hey, you know, this is the thing I want to do, so let's just kinda hard code that."

68. 13:37

    Um, but if, if you do let it kinda go wild into your Slack, uh, it's gonna start posting in general and random and sales and, uh, yeah, a few of my bots have kinda gone rogue.

69. 13:50

    All right, so the Gemini setup. Yeah, so you can get the, uh, API key here at the AI Studio, um, and there's a, a link in the, uh, slide deck as well if you need to click it.

70. 14:02

    Um, you can get a free account, generate an API key, uh, drop that into your env as well. [clears throat]

71. 14:09

    Excuse me. And there's also a remote, uh, Bench A2A agent. Um, so the code for it's actually in the repo, um, but we haven't officially released our API yet, so I, I'm just hosting that remotely.

72. 14:23

    Um, but it's a nice kind of way to show how you would use A2A remotely as well.

73. 14:29

    Um, so what is Bench? Uh, Bench is essentially a kind of LLM aggregator, uh, with autonomous AI agents. Um, so you get access to Claude, Gemini, OpenAI, xAI, and loads of more models.

74. 14:42

    Uh, it has, I think, about 30 tools now, um, and integrations. So, um, we actually started out with MCP integrations to Slack and Salesforce. They didn't meet our needs.

75. 14:52

    We built first party integrations, you know, data caching and indexing. Um, and, and that kinda gives you an idea of, like, how far is MCP gonna get you, right?

76. 15:01

    Uh, eventually, at some point, you're going to realize that it doesn't do the, you know, the specific thing you need to do.

77. 15:08

    All right, so running the application. Um, you're gonna run NPM run start all, um, and that's gonna kick off all the agents, right? So the Slack agent, the GitHub agent, uh, the host agent, and, uh, it'll also start the webhook server and the webhook, uh, admin panel.

78. 15:28

    Uh, you can access that then through localhost port 3000.

79. 15:33

    And, um, yeah, so l-let's just kinda go into what each of the actual, uh, agents do. Um, so the host agent is essentially your central coordinator, right? Um, and this, this may be the only agent that you have in your application.

80. 15:48

    It may be using external, uh, A2A agents. Uh, and if that's the case, then, you know, everything that your host does is going to be delegated, um, you know, to subagents.

81. 16:03

    Um, so that handles all the agent discovery and, and kind of bringing everything together.

82. 16:09

    Yeah, so the, the code for that's gonna be in source agents host.

83. 16:13

    And, um, you'll notice there's a couple of files in there. One of them is, uh, the host agent prompt, right? So that's just a plain text, uh, system prompt.

84. 16:22

    Uh, Genkit, that's gonna be, uh, essentially how you hook all of your A2A code up with Gemini. Um, and there, there's also a Genkit MCP plugin that, um, the subagents use.

85. 16:39

    Yeah, so then the Slack agent. Um, so this is gonna send a Slack message in response to the webhook transcript. And yeah, the, the, the kind of sample webhook that we have in this is essentially, you know, your meeting end, and you're going to receive a transcript of that meeting, right?

86. 16:55

    Um, and with that, you're gonna decide what to do. So it's gonna... You know, if it detects any bugs, it's gonna create a GitHub issue. If it detects any, you know, feature requests or, or anything of interest, it's gonna post that into Slack.

87. 17:08

    Um, and you can think of the kind of automations that you can build with this sort of, uh, scenario, right? So, um, you could even-- I had a version here that was hooked up to Salesforce, but, um, there's actually a limitation on the host agent on how many subagents it can call.

88. 17:24

    Um, so I, I figured, right, if one of them's gonna go, it's gonna be Salesforce 'cause it's, it's probably the hardest to get an account on. Um, but you could actually update an opportunity based on a sales call, right?

89. 17:35

    So you could have a sales call and, you know, you're talking to them, you're doing your discovery, and you're able to update those Salesforce fields automatically. Um, and, like, the time saving for account executives, because, you know, they're probably on back-to-back calls, um, is actually pretty big.

90. 17:53

    Yeah, so this was an interesting, um, issue I ran into. So I asked one of my colleagues, um, to test the repo out, right? And, and he was getting this weird error where it was saying, you know, the Slack MCP succeeded.

91. 18:06

    Um, so I asked him to send me the logs, and he sent me this, and it was like, "Is error false?" [laughs]

92. 18:14

    And I'm like, "Okay, that's, that's great." [laughs] So yeah, it, it turns out that, you know, not all MCPs are created equally, and the Zapier Slack MCP, uh, fails silently.

93. 18:26

    Um, so the, the reason it failed was he, he had, um, the default Slack channel name, uh, which was, like, test-uh, Damien Slack, and he was in a different workspace where that channel didn't exist. [laughs]

94. 18:39

    So it just failed silently. Uh, so I added a bit of code to detect this kinda empty text array, um, so it will fail now. Um, but it kinda goes to show you just kinda the limitations of MCP.

95. 18:53

    Yeah, so the GitHub agent, uh, pretty straightforward. It's, it's, it's probably the, the most basic of, of, of the, the three or four. Um, so it, it just creates a GitHub issue.

96. 19:03

    Um, super simple, um, but you could imagine, you know, how you would extend this, right? Maybe it's going to open a PR, right? May- maybe it's actually gonna implement, uh, the fix for the bug that was reported, uh, in the meeting.

97. 19:16

    Um, and you can see how down the line as, you know, AI gets better and, and, and things really improve, that a lot of this automation is gonna be driven by human interaction, right?

98. 19:27

    So, you know, speaking with people and posting messages in Slack and talking in GitHub discussions, um, is gonna trigger AI to take action.

99. 19:37

    Yeah, so the Bench agent, um, it can, it can do a lot. Uh, and that was actually one of the problems that I found with A2A, is that, like, the more functions and capabilities an agent has, um, the harder it is to describe the agent's capabilities, um, in the agent card.

100. 19:55

     Um, so the agent card is essentially, like, the public, um, information to any other agent of what that agent's capable of. Um, so I had to really just pare it back, and I said, "Look, you know, y- you can do a handful of things.

101. 20:07

     I know you can do more, but, like, for now, these are the few things that you can do." Um, and it's able to go off and, like, you know, browse the web, do research, uh, data science, all sorts of things.

102. 20:18

     Um, so we're just gonna use it for, uh, researching the company and the people, uh, in the meeting transcript.

103. 20:26

     All right. Here we go. Demo gods. Uh, before I start, any questions?

104. 20:32

     Yeah?

105. 20:33

     You mentioned some limitation on the number of agents. What was that? Uh, could you attach that?

106. 20:37

     Yeah. So the, the Genkit implementation that Google provide, uh, limits you to five maximum kinda sub-agent calls, uh, per turn.

107. 20:47

     Is that a hard limit, or...?

108. 20:49

     Yeah, I, I couldn't get around it. The... Like, there was this max, like, setting, but it, it didn't work.

109. 20:57

     Okay.

110. 20:58

     Yeah. Yeah, so it's, it's something I'm sure they'll fix eventually, but, um, it was, it was an interesting issue.

111. 21:06

     All right. Let me see if my, uh, my code is running.

112. 21:11

     Yeah, I think it is. Yeah, so it should be here. And actually, I'll show you the, the MCP server as well while, while I'm here.

113. 21:23

     Yeah, so this is the MCP inspector. It's, um, an open source repo that's part of the model. Sorry, yeah, at the back?

114. 21:30

     A quick question. Uh, where did you put the limitations of that Bench agent? Like, was that the prompt that you talked about earlier or somewhere else?

115. 21:37

     Yeah, that's actually in the agent card. So that'll be in the index.ts of the, of the sub-agent. Yeah, I'll, I'll be going through the code in a little bit as well so you can see it.

116. 21:47

     Um, yeah, so I'm connecting to my Zapier MCP URL that I got. Um, so I just copied this one, dropped it in, um, gonna connect over SSE. Um, and this allows you to, you know, list the tools, call the tools.

117. 22:01

     Um, and it's quite interesting now that Zapier has added instructions, um, as a mandatory field on actually all of their, uh, MCP tools. Um, so you, you don't actually need to fill out the, uh, the fields anymore.

118. 22:16

     So you can just give it natural language. So th- this kind of suggests to me that they're using an LLM on their side to figure out how to populate the fields on your behalf, um, whi- which is interesting 'cause it's gonna cost them a fortune, right, as more people adopt it.

119. 22:31

     All right, so this is the, uh, the agent dashboard. Let's just make sure everything's working. Yeah. Uh, you can see I have a couple of previous ones that I ran.

120. 22:39

     Um, this one is actually the one where the Slack, uh, thing wasn't found. So when I was testing that... Oh, my mouse isn't moving. There we go. Um, yeah, so I put in like a, you know, typical unknown, uh, Slack channel, um, and then it, it detected that it couldn't find it, um, based on the heuristics.

121. 23:00

     Not sure why my mouse isn't moving. There we go.

122. 23:06

     Yeah?

123. 23:07

     So you have defined four agents here.

124. 23:11

     Mm-hmm.

125. 23:12

     So, um, so they are all A2A agents?

126. 23:16

     Yeah, correct.

127. 23:17

     Okay. So maximum you can go for A2A agents is five?

128. 23:22

     Yeah. Uh, when, when I got to five, that's when I got the error. Yeah, so I think four. Um, um, yeah, and the, the host agent here, so these are the host agent logs.

129. 23:33

     Uh, you can see it connecting to the, the different agents. Uh, this agent's just running on a little dinky, uh, EC2 instance that I spun up. Um, and it goes through, learns about the agents, you know, processes webhooks.

130. 23:46

     Like, y- you don't necessarily need to go in here unless you, you get a failure. Um, Slack agent, pretty similar. Um, it's, it's basically just- Sitting there waking-- waiting for another agent to connect.

131. 23:58

     Uh, when the agent connects, it, it, uh, communicates with it. Uh, and you can see here the, the Bench agent's running remotely. Um, the reason I don't have, uh, verbose logs here is because it's remote, it's not under my control, right?

132. 24:12

     Um, so the A2A logs for that agent are actually on the EC2 server. Um, which kind of brings up another question about how do you debug when an A2 agent fails, right?

133. 24:23

     Um, yeah, so then on the webhooks page, um, so this is the, the only webhook that's pre-configured. Um, and this basically explains, you know, to the agent what it's actually gonna do when this webhook arrives, right?

134. 24:38

     Um, so it's gonna process the incoming webhook. Um, we have a little prompt template here, right? So it, uh, tells it what the agent capabilities are, uh, how to analyze it, right?

135. 24:48

     Um, and then we have the processor config, right? And, and this just kind of tells it, "Hey, these are the agents that you have access to as part of this, uh, webhook."

136. 24:58

     Um, this will become important when you've got, say, a hundred A2A agents, and you only want, like, two of them to, to interact. Um, and then here we have a test.

137. 25:09

     Um, so this is just a fake transcript I generated with, uh, with an LLM. Um, and when we send the webhook, you can see here it's processing, and hopefully the demo gods will, will do me good here.

138. 25:24

     And it does take a little bit of time, right? So the host agent has to process it, then it has to reach out to the sub-agents, you know, get all the information.

139. 25:31

     Um, I think the, the Bench agent probably takes the longest because it's actually doing its own sub-tasks as well. Okay, we got a, we got a Slack message. That's a good sign.

140. 25:41

     Okay, so Snowflake is interested in Slack and GitHub integrations. Very cool. Um, we have the GitHub, so...

141. 25:50

     I don't know why my mouse keeps freezing. There we go.

142. 25:54

     Yeah, so we should have a GitHub issue.

143. 25:58

     Here we go. Yeah, so during the trial, the AI misclassified s- the severity of the bugs. Engineers n- need to investigate and fix the issue. Right. So it's, it's re- really simple use case, but, uh, you can imagine that that transcript is probably gonna be 10 times longer, you know, a lot more information in it.

144. 26:15

     Um, and, and it'll just work, right? Um, and then we also have the Bench agent. So, um, oh, looks like it's waiting for results. Um, so it's gonna research, uh, the company.

145. 26:27

     Uh, I think I did one before where it just returned a result. Let me see. Yeah. So it basically goes off, does a research into Snowflake and all the participants of the call, um, and returns that information.

146. 26:40

     Um, a- and this can kind of get as complex or as simple as, as you want it to be.

147. 26:45

     Um, and yeah, so when, when you're using the application and you have it up and running and... Has anybody managed to get it up and running?

148. 26:54

     Wow, impressive. [laughs] Yeah.

149. 26:58

     Quick question. Like, uh, you're using Bench agent to do the orchestration. That's why you're having it remote, right?

150. 27:05

     Uh, no. So the Bench agent is just... Like, think of it as a third-party agent that we can leverage. So the, the host agent is doing all the orchestration.

151. 27:14

     Okay. So, like, uh, what is the re- what is the actual role that Bench agent is playing? Like, what is it actually doing?

152. 27:20

     Uh, it's doing research on companies and people.

153. 27:24

     So it's just another agent?

154. 27:26

     Yeah. So it's an agent with a load of different capabilities, and it's, it's basically just, um-

155. 27:31

     So the orchestrator isn't local. That, that is local, right?

156. 27:35

     Yeah, the ho- so these, these three, host, Slack, and GitHub, are all local.

157. 27:39

     Yeah, I was like... Uh, because I, I think I mistakenly thought that Bench was doing orchestration, and I was like, "Why, why is it..." Right.

158. 27:45

     Yeah, no, the B- Bench is just a, um... Like, it's in the repo, but, um, you need an API key for it, and, um, we're, we're launching in about two weeks.

159. 27:53

     So, uh, I just made it remote for the, for the purposes of the demo. Um-

160. 27:58

     So what about the host agent, though?

161. 28:00

     Sorry?

162. 28:01

     The host agent, is it, uh, the Zapier agent or like the-

163. 28:04

     No. So the, the ho- so all of these agents are A2A agents. Um, the Slack agent and the GitHub agent have MCP tools to Slack and GitHub-

164. 28:14

     Right

165. 28:14

     ... through Zapier.

166. 28:16

     Right.

167. 28:17

     Yeah. Um, I can actually show you a diagram that might, might explain it a bit better.

168. 28:28

     Yeah. I don't know if that explains it better, but... [laughs] [laughs]

169. 28:32

     But the orchestration does happen on your local, though?

170. 28:34

     Yeah, yeah. E- everything's happening on my local. So if I go into the, into the code base, uh, I have the agent logs. Um, so th- this is all happening here, right?

171. 28:45

     So it's sent to Slack to ta- or-

172. 28:46

     Showing that local

173. 28:47

     ... is that readable? I'll go one more.

174. 28:50

     One more.

175. 28:59

     Yeah. So you can see here the transcript came in, um, and then it got a response from each of the sub-agents and then completed them. Uh, and it did all of this in parallel as well, right?

176. 29:09

     Um, sorry, is that a question?

177. 29:11

     Yes. So i- in your example here, which agent would handle human confirmation? Let's say we want to have a Create the Task button in Slack up here. Which agent would handle that, uh, that part?

178. 29:25

     Do you create a new agent for human confirmation? Do you keep the old one?

179. 29:30

     Yeah, see, you'd need a staging area for, for actions. Um, so it's not something I've built into this. Um, there's a lot more you could do here. Um, but human confirmation would typ- typically be done through like a draft, right?

180. 29:42

     So you would maybe pop up a Slack message with some actions, um, and then when somebody clicks that, it would communicate back, kind of like a secondary pass webhook.

181. 29:52

     Uh, you might need to persist state though. Yeah?

182. 29:56

     Consider the security of these endpoint controls of different vendors communicating from the endpoint.

183. 30:05

     Mm-hmm.

184. 30:05

     How do you manage the security piece?

185. 30:08

     Yeah. So as a part of the A2A spec, you're gonna have some sort of authentication, right? Um, I've just exposed everything, right? Like, it won't exist tomorrow, so there, there's no security implic-implications.

186. 30:18

     Um, but ess-essentially, you're going to, you'll probably have to have a subscription with the company that's providing that A2A agent, uh, because it is consuming tokens, right?

187. 30:30

     What authentication tokens?

188. 30:30

     Um, I'm, I'm not sure exactly what A2A have in plan. Uh, it's still pretty early days, but, um, with MCP it's a little bit further ahead. It has OAuth, uh, header authentication, things like that.

189. 30:41

     So imagine something similar.

190. 30:43

     And how would C-A-S-A governance like, uh, LLM firewall, all those, uh, benchmarking, auto benchmarking, and, um, also the guardrails, et cetera.

191. 30:56

     Mm-hmm.

192. 30:56

     Uh, can you... Do you have a separate agent or everything is being-

193. 31:00

     You, you'd probably manage that on like an Amazon Bedrock or something like that, right? And you would just, you know, use that guardrailed LLM, um, from behind there. Uh, you don't have to use Gemini here either.

194. 31:13

     Yeah.

195. 31:15

     Um, the whole idea of A2A is agent can communicate. So like, you now have the difference between like host agent and then that host agent is kind of like the planner and talk to each.

196. 31:27

     Um, do you see like A2A becoming like sub-agent talking to each other, um-

197. 31:34

     Um, I, I guess you could, but I, I don't know if that's the intention, right? Like, um, th-then they just become hosts, right, when they talk to each other.

198. 31:44

     Um, like i-if you think about it, like, if you have no knowledge of sub-agents, um, how would you, how would you know to talk to them, right? You would have to then become a host agent yourself, connect to that other sub-agent to, to do that.

199. 32:00

     So I, I don't know if that's intended in the A2A spec for sub-agents to communicate.

200. 32:06

     Uh, yeah.

201. 32:07

     So with the host agent, um, and the orchestration that it's doing, is it actually managing a combination of all the context windows or like, like do you have a limit quickly?

202. 32:18

     Yeah. So all of the context windows, and, and this kinda, uh, is something I'm gonna cover now in a second as well. Let me, uh, just go back to the slides, um, which is a good, it's a good segue.

203. 32:31

     Um, so yeah, one, one of the benefits of like A2A or, or any sort of sub-agent, uh, framework is that you're, you're not consuming, um, the tool results into your context, right?

204. 32:45

     So like when you say, "Hey, you know," um, and I think I have an example later on. Uh, but if, if you have a load of, uh, Slack messages or GitHub issues or Salesforce opportunities, um, and you wanna analyze them and maybe produce like, you know, a summary of categories and counts, um, the only thing your host

205. 33:02

     agent cares about is the summary of categories and accounts. It doesn't care about the like individual details, right? 'Cause tho-those have already been processed by the sub-agent. So the sub-agent's context gets big, not very big, but like as big as the task demands, and the host agent only incrementally grows by the, the business value it got from

206. 33:22

     that agent. Um, like one, one of the challenges at Bench is, you know, we have so many tools, right? Like the context can blow up very quick. Um, so you know, very early on we decided, okay, we need to have composability.

207. 33:35

     Um, so that means that Bench can create its own internal Bench agent, um, to avoid that context growth problem. Um, and we're even thinking of going one step further, whereas like, you know, should we have an agent for every single tool, um, so that every single tool is protected from the primary, uh, prompt?

208. 33:53

     Um, so you know, a-as you add more tools, like the tool definitions themselves, I think we're up to like, you know, 10,000 tokens just for tool definitions alone. Um, I added the Asana MCP.

209. 34:05

     It added 11,000 more tokens, right? So like, you know, a lot of these MCP servers, like they're, you know, they're giving you a lot of information, um, and you may not actually want that.

210. 34:15

     Uh, and, and that's actually one of the challenges with first party MCPs is they expose all their tools. Uh, that's one of the benefits of Zapier, where you can pick and choose which tool you wanna use.

211. 34:26

     Yeah.

212. 34:26

     Yeah. I was just gonna ask, why do we need Zapier in that kind of-

213. 34:30

     Uh, Zapier is just a really easy way to, to use, uh, MCP right now. Um, I think like Linear, uh, Asana, um, um, a few others have added like first party MCP servers that are much better than what Zapier exposes.

214. 34:50

     Yeah, so, so why does context size matter? Um, so AI agents accumulate context like as they work, and you're supposed to keep like all of your tool calls, right, what you sent to the, to the tool and what you got back.

215. 35:05

     You're supposed to keep that in your context so that later on, if you, you know, ask a follow-up question, it still has access to that data. Um, a-and that becomes very challenging, right?

216. 35:16

     So you've kind of got two options, like, okay, do I just prune, you know, old tool calls and now the, the agent gets dumb, or, you know, do I figure out some other way to do it?

217. 35:25

     Um, and, uh, c-cost is a big challenge, especially when you're doing prompt caching. Um, so with prompt caching, it, it enables you to essentially put a marker in your context and say, "Hey, look, when I make my next request, I want everything in my, in my context so far, uh, to be cached so that I'm not gonna

218. 35:44

     get charged for it." Um, but the cost to actually push that into the cache, uh, is about 3x the cost of, of making a single request with that context.

219. 35:53

     Um, so that means that you have to be very, you know, diligent in what sort of, uh, context management strategies you use. Um, you know, I was running simulations 'cause I, I couldn't really figure out like what is the optimal, um, you know, caching, uh, strategy.

220. 36:09

     Uh, so I ran simulations based on usage data, um, of like, you know, what's the typical context growth? How many turns, you know, on average? Like what percentage of, of users only send one turn, right?

221. 36:21

     Should we, should we cache that one turn if they never ask another question, right? Probably not. Um, so you know- It probably gets down to the actual u-user level.

222. 36:30

     So if you have a user that always, like, puts in new prompts into the same chat and never opens a new session, um, you're probably gonna wanna, you know, continuously, uh, cache their context.

223. 36:41

     Uh, but you might have another user who always creates a new session for every question. Um, and then just figuring out, like, you know, what is the context growth?

224. 36:50

     Uh, I think we figured out it was around 30,000 tokens was the optimal, um, kind of across the board for everybody. Um, but that also comes up with false positives.

225. 37:00

     So sometimes you can end up caching the last turn of, of a conversation, um, and, and that's gonna, you know, cost you a lot more than it, than it should naturally.

226. 37:12

     Yeah, so the, the great thing about the subagents, right? It protects them. Uh, and this was the GitHub kind of example I was giving you. Um, but this applies to pretty much every, uh, tool.

227. 37:21

     So, like, if you're ever integrating with a system, you're probably gonna run into issues like, why do I have to call, you know, list Slack channels every time to get the channel ID for the channel name that was provided, right?

228. 37:34

     'Cause, like, nobody's gonna provide, like, in a chat, the channel ID that they wanna post, right? It's a, it's a UID. It's, it's not memorable. Um, so then you get into the question of, okay, well, do I just cache the list of channels?

229. 37:46

     And, and when do I update that list of channels, right? Like, what if the channel was deleted, renamed, or a new channel was added? Um, yeah, and then the, the cost is, is really probably the biggest one.

230. 38:00

     Um, yeah, so the, the benefits to this lean context, right? So your subagents have that isolated context, and, and that really just allows you to, um, be, be super, like, fast, low latency, low cost.

231. 38:14

     Um, and if you ever need to go back to ask another question, y-you know, you're gonna, like, spawn that, uh, process again, right? Um, so maybe if you're in control of these other agents, you, you might wanna have some sort of like, uh, I don't know, f-five-minute TTL on previous questions, right?

232. 38:34

     Um, and then, yeah, the host agent only processes the summaries, um, and the raw data is discarded after processing. Um, yeah, so I'm gonna jump back into the code here, uh,

233. 38:49

     and just kinda walk you through, uh, how it all works.

234. 39:00

     All right. We'll start with the host agent. Um, and, and you notice a few other things, right? So there's MCP. This is just your standard...

235. 39:09

     Sorry? Oh, I thought someone said something. Um, yeah, so this is kinda your standard MCP client, uh, code. Uh, just, just allows you to consume, um, the MCP, uh, calls coming from the LLM.

236. 39:24

     Um, we have the, the GitHub, right? So this is gonna be, um, what it sends to that Zapier endpoint. Uh, it's gonna call GitHub create issue. Uh, and then the Slack agent is gonna do send Slack channel message.

237. 39:38

     Um, so these are just kind of like the MCP client tools that the, uh, individual agents will use. Um, yeah, so this Genkit, um, this is based on, on what they provide in their, in their sample repo.

238. 39:54

     Um, y-you can use a different model if you want, right? You can change, you know, the, the settings on it. Um, but this essentially, uh, spawns you a new instance of what's going to communicate.

239. 40:06

     Um, this just loads the system prompt. Um, I can open up the system prompt here. Um, so right, it's got a critical workflow. It's gonna do these things in this order.

240. 40:15

     It's got a few steps, you know, discovery. Uh, uh, th-this is actually something I noticed. Like, if you don't tell the A2A agent to call list remote agents, it just won't, right?

241. 40:25

     And it'll try to answer everything on itself. Um, you know, it can very easily fake sending a Slack channel message and be like, "Oh, I just sent it for you."

242. 40:33

     And it's like, "No, you didn't." [laughs] Um, you know, one of the things I've noticed, uh, using Cursor is, like, every time I catch it doing something wrong, it says, "You're absolutely right." [laughs]

243. 40:47

     Um, I even tried to prompt that out of it, um, and it's not promptable to get, to get it to not say that. Um, cool. Yeah, and then the, the index.

244. 40:56

     So this is actually where the agent card is. It's a little bit long.

245. 41:02

     Let me see. I think it's up here near the start.

246. 41:13

     Dun, dun, dun. There we go. It was line 1200, so not near the start at all. Um, yeah, so this, this is what the host agent exposes if somebody else wanted to call it.

247. 41:26

     Um, so it has these abilities to list remote agents and send tasks, right? And then if we compare that to the, to the GitHub, which is, uh, a lot smaller, um...

248. 41:42

     There we go. Yeah, so the GitHub agent can create GitHub issues, right? Um, it's got the ability, uh, to do various things, and, um, it has a list of skills.

249. 41:54

     Um, a-and this is all that the, the host agent really knows about this agent. Um, so you could imagine how big this might get if you were to, you know, implement every single API that, say, Salesforce has or something like that.

250. 42:07

     Um, and i-in a lot of cases, um, at least with Salesforce, rather than implementing, you know, wrappers around the APIs, you're probably just gonna wanna use, like, the SQL or the SOQL directly and let the agent actually write the queries.

251. 42:21

     Um, there's a lot of flexibility when you have, you know, direct database access essentially, um, because the, the LLM can, you know, bypass, you know, the API lay-layer and just go directly to the, to the database.

252. 42:34

     Um, and then the, um, GitHub agent prompt, right? So it's got some, uh, things. Um, this was something I had to add because it, it insisted on, um, mentioning who submitted the bug report, right?

253. 42:47

     So there, there's definitely concerns around, you know, PII, uh, leaking from your, you know, internal meeting transcripts and ending up in GitHub, right? Um, and that kinda goes back to your, uh, your question about, you know, how do you audit what's coming out of these LLMs, right?

254. 43:03

     Uh, and you can do that in a number of ways, but it, it wouldn't be a part of the A2A spec. I think it would just be the LLM you connect to has those guardrails in front of it, uh, and you're, you're just using that LLM that has the guardrails.

255. 43:16

     Um, similar Slack, um, [clears throat] excuse me, has a, a very simple, um,

256. 43:26

     uh, agent card that I can't seem to find. Um, and then if we jump over now to, uh, the host config. Um, so th-this is essentially what configures, um, the webhook, right?

257. 43:41

     So the webhook has essentially a config that tells it, like, what it's doing, and, and you can see that in the UI as well. Um, and then, uh, within the A2A folder, we've got the client and the server.

258. 43:54

     Again, these, these are just pulled directly from, uh, the A2A repo. Um, I don't think they've actually exposed, uh, types or packages yet, uh, which is kind of confusing.

259. 44:04

     Um, but essentially you can bring that stuff in there.

260. 44:08

     And, uh, then the webhook server. Uh, so this is just a web UI. Uh, initially I had this whole thing done through the CLI. Um, you know, coding with, you know, tools like Cursor or Augment Code.

261. 44:21

     Um, CLIs are way easier for AIs to actually write, right? They're go- they're gonna be able to test it, uh, interact with it much better, and, uh, be able to, uh, produce those outputs.

262. 44:35

     Awesome. Uh, so yeah, I'm gonna, I'm gonna shift over to kind of Q&A now. Um, so yeah. Anybody, any questions? Yeah.

263. 44:42

     So, um, I wanna talk evals for a second.

264. 44:46

     Mm-hmm.

265. 44:46

     So, like, um, I, I assume that you manage them, or I don't know. I mean, you manage them probably at the, at the agent level. Is there any type of like distributed eval, uh, that, that you bring when you're, like, dealing with A2A?

266. 44:58

     Um, what, at what layer, how do you handle it?

267. 45:00

     Yeah. I ha- haven't done much evals on A2A. Um, I still think A2A is a bit too early to go into production. Um, like even MCP is, is kind of borderline.

268. 45:11

     Um, like there, there's a lot of rough edges. Um, I think you can achieve like much better, uh, things. If, if you're in complete control of everything, you can achieve much better results, you know, with your own local, uh, function calls.

269. 45:28

     Yeah.

270. 45:28

     Any reason you use TypeScript instead of Python?

271. 45:32

     Yeah, you can use any language. I, I think actually, uh, the A2A framework is better in Python. Um, I just prefer, uh, TypeScript myself.

272. 45:42

     Yeah.

273. 45:43

     Can you tell more about the caching? Is caching provided by the model providers, or do we implement our own caching?

274. 45:50

     Yeah. So you implement your own caching. Um, so you decide, you know, wh-when to move that cache marker, uh, how to manage it. Um, it can be tricky, and, and I don't think there's very good information available online on, on what the best strategies are.

275. 46:05

     Um, when I was doing the simulations, I, I used like linear growth, exponential growth, um, you know, fixed size, and, and kind of compared them all. Uh, they all worked out between twenty-five and thirty-five percent cost savings.

276. 46:18

     Um, but like in practice, what you'll find is you're gonna have outliers where, you know, the cost of a session kind of balloons because of, you know, you, you cached at the wrong point.

277. 46:30

     Yeah. Yeah.

278. 46:31

     So each of the A2A agent can be talking to, I know, their own, like, fine-tuned LLM, right? Like, they have their own LLM that they're own. There's not a central LLM that they refer to, right?

279. 46:41

     Yeah. Yeah. So they, they all have their own, um, which is kind of in contrast to MCP, where the MCP wants to use your LLM, right? Because it doesn't wanna generate its own tokens.

280. 46:51

     Sorry, yeah.

281. 46:53

     Um, how about the, uh, authentication and authorization? Um, do MCP or agent to agent, um, has that implicit authentication or authorization-

282. 47:03

     Mm-hmm

283. 47:04

     ... or you have to build that layer also?

284. 47:06

     Yeah. So there, there's a couple of different ways. Um, so, uh, within the authentication, you can have, uh, headers that do the authentication. Um, I believe if you drop in an OAuth, uh, URL, you'll also get an OAuth popup.

285. 47:20

     Um, I really like the OAuth authentication because you're getting the user's, you know, ACL, right? Um, and that means that, you know, what that user can access, um, is specific to them.

286. 47:32

     And it's not im-implicit in the MCP or agent to agent. You have to build that AOP. That's what you mean, right?

287. 47:38

     Yeah. So it's, it's gonna be di-dictated by the, the remote, uh, s-server, so either A2A or MCP. Um, if you're running your own, you can choose what you wanna run.

288. 47:48

     Um, there's different transport types as well. So standard IO is something that you would use locally. So like imagine you wanted to create like a file on your desktop, um, you're gonna use standard IO typically to interact with local.

289. 48:01

     Uh, and then SSE was server-side events. That got deprecated in favor of streamable HTTP.

290. 48:08

     So, so sorry. Sorry, guys. Um, so for example, like if, if we are interacting with a Salesforce agent, let's say, and each user has different authorization. For example, each a- uh, employee A probably have access to the, uh, some sort of tables.

291. 48:25

     Employee B have different access control, right?

292. 48:27

     Mm-hmm.

293. 48:27

     Um, in that case, is it done by, um, um, MCP, the framework itself, or we have to build it?

294. 48:34

     Yeah. That, that will typically be handled, uh, through an OAuth MCP server, right? So they're going to essentially log in as themselves as part of the connection, uh, and then they're gonna save that refresh token for later use.

295. 48:49

     Yeah.

296. 48:52

     ... um, for security especially. You explained very well about role authentication, et cetera.

297. 48:59

     Mm-hmm.

298. 48:59

     But I'm looking for more explanation towards encryption, asymmetric encryption, and also there is a possibility of certificate managers and all the way to the end of the entire architecture.

299. 49:12

     So how would you describe the performance? And you see, I, I'm looking for some financial application.

300. 49:19

     Mm-hmm.

301. 49:19

     This architecture, what you have described is pretty good. But, uh, similar on the financial applications, as well as, uh, some department of defense or some kind of applications highly, in highly secured environment where it's all, uh, both combination of asymmetric and symmetric encryption.

302. 49:41

     Yeah. You're, you're probably gonna wanna run like the LLM yourself, and you're more than likely not gonna wanna interact with anybody outside your VPC, right? In those cases. Um, I, I don't know if you would want to consume a third-party MCP server or A2A agent, uh, in a highly regulated environment, right?

303. 49:59

     Like, you know, HIPAA compliance, financial stuff. Um, if you do have the ability to do that, right, you're gonna have some sort of agreement with the service provider that provides those tools.

304. 50:10

     Um, and you're gonna, you know, do transport over HTTPS. You're gonna have maybe mutual TLS both on the A2A agent and the remote agent. Uh, and similar with the MCP server, you're probably gonna have some sort of IP whitelisting, right?

305. 50:23

     Like there's, there's a ton of things you can do around that. I think they're out of scope of, of the actual protocols themselves, uh, 'cause, you know, essentially you're over an encrypted line.

306. 50:31

     But, uh, typically there's, there's more to it than, than just that, right?

307. 50:35

     So you're playing around the endpoint controls on this, and that's really scary when dealing with the-

308. 50:44

     Yeah

309. 50:44

     ... code you have.

310. 50:45

     Yeah. And like if, if these are your own internal MCP servers and your own internal A2A agents maybe from different parts of the organization, um, you know, they'll, they'll all live inside your VPC, and they're probably never gonna talk to the, to public internet.

311. 50:59

     So your, um, the solution-- the, the answer I get from you is stay with this VPC and stay away from, uh, in that case, stay away from endpoint, um, security, which means stay away from MCP or A2A.

312. 51:16

     It's, uh... So these, these are just protocols. Um, it- it's really up to you whether you want to connect to an external third party, and that's gonna be your own security po-posture.

313. 51:27

     Uh, it's not really gonna be defined by the protocol itself.

314. 51:29

     Yeah. Keep them away from the-

315. 51:31

     Mm-hmm

316. 51:32

     ... subnet or bring them inside the subnet. Which one would you prefer?

317. 51:36

     I, I, I would liken it to like, I found a USB cable. Will I plug it into my laptop, right? So the, the USB, it's not its fault, right?

318. 51:44

     Like USB is just a, a standard. Um, it's what that USB is connected to is the risk, right? So like if you're willing to find a dongle on the street and plug it in, you know, that, that's really gonna be your security posture, right? [laughs]

319. 52:00

     Yeah.

320. 52:01

     Okay. So, um, how much heavy lifting do you have the orchestrator do? Like, do you ever hit the scenarios where, uh, you ha- the orchestrator in-interprets the response from a subagent and then maybe does a retry with a better prompt?

321. 52:15

     Mm-hmm.

322. 52:16

     Uh, do you hit any of those kind of loops or anything?

323. 52:18

     Yeah. So, so one of the things, and I, I kind of prompted it out of this, uh, workshop just to keep it simple, is, um, like the, the bench agent wants to have a conversation with the host agent.

324. 52:30

     Um, but I, I didn't want to kind of implement that back and forth 'cause it was gonna delay the, uh, the webhook processing. Um, but you can have backs and forths between the agents.

325. 52:40

     Um, and it's probably desirable as well, right? Like if, if for whatever reason the host agent doesn't give sufficient information, you know, the, the remote agent's gonna be like, "Okay, you know, I know you wanna update an opportunity, but you didn't tell me which opportunity," right?

326. 52:55

     Um-

327. 52:56

     I mean, I could even see scenarios where you have, uh, an expensive LLM that you have on reserve that you go to when the cheaper LLM agents aren't giving you what you want.

328. 53:05

     Like, I... Sorry, I'm just thinking through stuff.

329. 53:07

     Mm-hmm. Yeah. And I, I, I think like LLM cost and capability is, is a big challenge with a lot of these things because, you know, if, if you're running, say, Claude 4 Opus, and somebody for whatever reason asks you to summarize like, you know, five sentences, uh, it's gonna cost you a fortune, right?

330. 53:25

     So you need, uh, intelligent routing logic on like does this task need the entire context, right? Does it need 20,000 tokens of a system prompt to summarize, you know, a short bit of text?

331. 53:37

     Uh, and that's one of the challenges that you, you'll run into where you, you kind of need a, like a routing LLM in front of these complex agents so that they can actually figure out, you know, how deep do I go.

332. 53:50

     Yeah.

333. 53:51

     Similar to the routing orchestration question, I was wondering like if you wanted to post a Slack message that linked the GitHub issue created, for example, I think you'd probably prefer from your architecture to go back through the host to make that decision rather than let the GitHub agent directly communicate with the Slack agent.

334. 54:08

     Yeah. So the, the host agent wouldn't run the, uh, the calls in parallel, right? So there, there's actually a flag whether you want it to go in parallel or not.

335. 54:16

     Um, so it would have to say, "Oh, I need to create the GitHub issue first, um, before I talk to the Slack agent," right? Since I need that URL.

336. 54:25

     Uh-

337. 54:25

     But in general, you'd prefer to have those decisions go through the host rather than even allow the GitHub and Slack-

338. 54:31

     Yeah, absolutely. Yeah. Yeah. Uh, yeah.

339. 54:35

     Oh, I want to ask that the context slicing for the subagents, that is entirely happening through prompt engineering or are there other frameworks to like slice the context that will be going to different subagents?

340. 54:46

     Y-yeah. So, so typically context management is gonna be implemented in your own code base. Uh, the subagents context management i-is more than likely gonna be a third party's code base.

341. 54:56

     Um, if it's one of your own agents, right, you can manage it as well there. Um, but yeah, you're, you're gonna wanna figure out like what's optimal for your actual like production usage.

342. 55:06

     Um, yeah.

343. 55:08

     But so you will, you will be using prompt-

344. 55:10

     Host agent to, to kind of guide what context to send to each sub-agent, right?

345. 55:15

     Yeah, yeah. So, so what you, what you send is typically, like, a question or a task.

346. 55:20

     Yeah.

347. 55:20

     Um, it's usually very small, right? Like, you, you don't, you don't send the full meeting transcript to the Slack agent to, to do what it's doing. The host agent processes the transcript and then decides what the tasks are.

348. 55:34

     Um, so like if I look down here, uh... Or actually I think I can see it in the dashboard. Um, yeah, so th- this is actually what the, the host agent sent, uh, to the GitHub agent, right?

349. 55:46

     It says, "Create an issue in this repo, title this," you know, with this description and title. Um, and then the, the GitHub agent, its task is to extract, uh, three bits of informations, right?

350. 55:58

     So what's the instructions to give the MCP server? What's the body and what's the title?

351. 56:04

     Got it.

352. 56:04

     Mm-hmm. Yeah.

353. 56:05

     So does the MCP, um, use the dynamic request context part of... Which we want to send across with headers for each and every request. So whatever you showed earlier, that's pretty much, um, the, the servers understand what client it connects to.

354. 56:24

     But part of each and every request we have to send the dynamic request headers, say, correlation ID or the, the username headers or some authorization headers.

355. 56:36

     Yeah. So, so Zapier, uh, the SSE implementation doesn't actually require headers. Um, I think these are just left over from, from something else. Um, so there, there's actually no authentication, and the URL itself is kind of like a secret key, right?

356. 56:52

     Um, so like if I disconnect and, and reconnect without the headers, I should be able to, uh... Yeah. So I can, I can still query it. Um, they, they've moved away from this approach right now with, with, with more secure kind of, uh, setups.

357. 57:06

     And you'll, you'll notice in their thing, right, um, they, they've kind of deprecated that and, you know, treat this URL like a password, right? Um,

358. 57:18

     yeah? [coughs] Yeah.

359. 57:19

     What's your experience in using, uh, different LLMs for these agentic, uh, workflows? Like for example, Haiku, Sonnet, uh, Gemini, and also did you use any self hosted LLMs, uh, for this kind of workflows?

360. 57:33

     Mm-hmm. Yeah. So, so we, we typically lean towards Gemini for large context, um, and, um, Claude Sonnet 4, uh, for tool calling. Um, Claude Opus is better, but it's not like 4X better.

361. 57:49

     Um, you know, and when you compare price to performance, right, like, you know, 5% better doesn't equate to 4X the cost.

362. 57:56

     You're talking about Gemini Flash or, or Pro? Do you-

363. 57:59

     Yeah. So we, we'll use Gemini Flash for simple things like summarization, right? Um, you could use Claude Haiku as well, but I think, I think Google's kind of taken the lead in, in price performance, you know, e- from an economic standpoint.

364. 58:13

     Uh, but Claude is still the kind of king of tools. Uh, they, they created MCP, so they kind of had a head start, right? Yeah.

365. 58:21

     And what about the self-hosted LLMs? Do you-

366. 58:23

     Yeah. We, we have DeepSeek hosted in the US, um, so we've been trying that out. Um, I, I think Llama has kind of fallen by the wayside a little bit.

367. 58:33

     Um, and yeah, DeepSeek is just, you know, the clear winner right now. Uh, they also released a new version, they're I think on the 28th, um, that's kind of up there with o3 level models.

368. 58:43

     Um, we, we actually don't use reasoning models, uh, for our agents. Um, a lot of the time when you're, when you're building, you know, agentic agents, um, a reasoning model isn't really needed.

369. 58:56

     Um, like, un- unless you want to, you know, pay a fortune for some long thinking task. Um, you know, we, we, we can achieve kind of that reasoning level, uh, with just the standard models and, and browse and a few other tools.

370. 59:12

     Yeah.

371. 59:12

     So, um, like with A2A and like, you know, all the third party things that we can like assume or like if, if Stripe has an agent and there's agent cards like Amazon and stuff, do you pass instructions for like what...

372. 59:26

     Like exactly what you want back in terms of like... I'm just imagining another a- a third party agent blowing up your context window because they're, they're flooding you with way too much information you don't care about.

373. 59:35

     I mean, do you handle that through the prompt? Are there other tools to do that? Is it, is it not an issue?

374. 59:40

     Yeah. So, uh, o- one of the solutions to that is you actually just spawn another agent, um, to communicate with either the tool or the agent, right? Um, a- and that's one of the things we, we have in, in Bench, and here's some of the slides that...

375. 59:54

     So I don't know. Uh, generate, uh, five images,

376. 1:00:02

     uh, in subtasks.

377. 1:00:06

     So you spawn a sub-agent to sort of like absorb the context flood, for lack of a better term?

378. 1:00:12

     Yeah. So the, the sub-agents just kind of protect you, right? Um, and you know, like when, when you're spawning these things, you can do things in parallel. Um, and actually if I expand, you can see the thinking as well.

379. 1:00:25

     So you can see like as it's going down through it, right, it's, it's doing a lot of work, um, that you don't want in your context, right? Like, you, you don't want all of your thoughts bloating your, your context.

380. 1:00:36

     Um, but you also don't want all of your tools bloating your context either. Uh, you don't want images bloating your context. You want the ability to analyze an image, but you don't want like, you know, 100,000 characters of Base64 in your context.

381. 1:00:50

     Um, so there's, there's a lot of kind of optimizations that you can do there. Um,

382. 1:00:55

     but yeah. Did that kind of answer your question?

383. 1:00:57

     Yeah. And I'm just thinking through like why, like the logging, if you have to troubleshoot something like this, it's probably kind of rough.

384. 1:01:04

     Yeah. Yeah. So you can see here now it's spawning these sub-tasks. So these are all essentially like instances of Bench that will keep that context out of, out of my way, right?

385. 1:01:15

     Yeah. Yeah.

386. 1:01:16

     What have you... What have you-

387. 1:01:19

     Observability on your agents

388. 1:01:22

     Um, we, we just kind of roll our own right now. Uh, there's a lot out there that you can use, like, uh, Agent Ops is a pretty popular one.

389. 1:01:29

     Um, but yeah, like if, if you really wanna build your own, uh, kind of custom observability layer, um, you know, you're... Like, like Agent Ops doesn't really support this concept of composable subagents.

390. 1:01:42

     Um, so it's not really something that it could model, uh, correctly. Uh, but we've got some nice pictures of cats. [laughs]

391. 1:01:50

     Uh, and yeah, I know we've a few minutes left, but if, if anybody's interested, um, I have $50 in free credits. Um, this hasn't launched yet, so you're getting kinda early access to it.

392. 1:02:01

     Um, and yeah, we'll... I think we'll be in public beta in about two weeks. Um, so yeah, try it out. Like hit me up on LinkedIn. I, I'd love, uh, feedback from you all.

393. 1:02:12

     You're, you're all probably, you know, at the forefront of this, uh, AI stuff, and, um, it's changing every day, so if you log in one day and it looks completely different, don't be surprised.

394. 1:02:22

     Happens mid-demo for me. Yeah?

395. 1:02:25

     So you mentioned a lot how hiding context in subagents is a good thing, but haven't you had cases where you actually then end up missing something important, some small detail, and then how do you resolve that?

396. 1:02:36

     Does the agent actually go back and ask for that, or do you-

397. 1:02:38

     Mm-hmm. Yeah, so you can keep references, uh, in your context. So you, you might say, "Subtask ID one, two, three." And then when the agent's like, "Oh, I won- I wonder if I have this information, it's just not in my context," right?

398. 1:02:54

     Um, so it has to be smart enough to know when to actually go in and, and look at that. Um, and it can be a subagent that does that analysis, right?

399. 1:03:01

     So you could say, "Hey, subagent, can you just look at all of these IDs and tell me if you can answer this question?"

400. 1:03:10

     Yeah.

401. 1:03:11

     Right. Uh, there are a lot of overlaps to the, to what's possible of, of, of, of textual and whatever, right? So but then you mentioned in the beginning about the safety, right?

402. 1:03:22

     So, uh, there are a lot of discussions saying that safety it's a way for using MCP for agent to agent communication, right? Because the agent can be a server and a client at the same time, right?

403. 1:03:34

     Mm-hmm.

404. 1:03:35

     So what is your opinion about that, you know?

405. 1:03:40

     Yeah.

406. 1:03:40

     'Cause you know-

407. 1:03:41

     It's, it's the million dollar question, isn't it?

408. 1:03:44

     Yes. That's it. That's, that's why I asked.

409. 1:03:45

     Yeah. And I, I do think you can achieve easier agent-agent communication with MCP. Um, but if it's a remote MCP server, I think A2A actually is a little bit better, um, because you have somebody else paying the, the tokens and building the agent.

410. 1:04:07

     Um, like if, if all you're getting from a third party is a list of tools, um, those tools may not meet your needs. Um, but if you're getting a, a fully fledged agent from that third party, then it might be able to figure out like what it can do w- with even private APIs, right?

411. 1:04:24

     Uh, maybe, maybe that agent has direct database access, and it's able to actually on the fly, you know, create the API you need.

412. 1:04:33

     Right. So, so the trade-off is basically about which, which is important, okay? About costs and who gonna pay the, for the tokens and whatever, something. Something like that can be...

413. 1:04:43

     Because like you are running the same server, maybe using MCP, MCP gonna be easier, right? But wait, wait, am I correct? I don't know if I got myself here, but, you know, at the end of the day, it's gonna-- who's gonna pay, gonna pay for the tokens, right?

414. 1:04:59

     It's something like that.

415. 1:05:00

     Yeah. And I, I think who pays for the tokens is kinda secondary, right? Like at the end of the day, it's about business value, and if you can get the business value from a tool, right, like send Slack message, um, like that's great, right?

416. 1:05:12

     Like sending a Slack message isn't hard. Um, but the implementation of the search function of Slack is, is actually not great, right?

417. 1:05:20

     Right.

418. 1:05:20

     Um, whereas you compare that to some of the other, uh, MCP tools like Linear, uh, the search function is actually pretty good, right? Um, but then you, you start to run into performance, uh, challenges as well.

419. 1:05:30

     So like if I wanna search 100,000 opportunities in Salesforce, um, and figure out like what's the close loss reason counts and categorize them and do all of that, like that, that's a huge data processing challenge.

420. 1:05:43

     MCP is not gonna be the right, uh, tool for that. Uh, 'cause you're, you're essentially gonna say, "Okay, list opportunities. Now get the details of each opportunity," right? And you're, you're gonna make like 100,000 network calls.

421. 1:05:54

     Um, at that point, you're really gonna wanna actually, you know, ingest that data, you know, build an index, right? And I, I think, uh, and this is kind of like an idea, is like we, we may see a lot of these third party software providers essentially just allow you to access the data lake through an agent, right?

422. 1:06:15

     Um, so like scoped data access, you know, just running complex queries super fast, you know, no, no real like tool calls per se, but just like, "Ask me a question, and I'll go figure out how to get the answer."

423. 1:06:29

     Right. I think that-

424. 1:06:31

     Yeah. Yeah?

425. 1:06:32

     Similar to what you're saying, like summarizing that trail, like isolating context management and simplifying complex sequencing.

426. 1:06:43

     Mm-hmm.

427. 1:06:43

     Those are kind of two of the main things that we're using in our agent versus just taking a few years of what you're saying to decide which things to call.

428. 1:06:53

     Mm-hmm. Yeah. So you can achieve the same with H-- or with MCP. So you, you could just have a, a tool that's called Talk to Subagent, right? Um, and, and it can work as the communication protocol.

429. 1:07:07

     Um, I actually built another, uh, application where I had an LLM, uh, Claude 4, um, just talk to its predecessor, um, just to see what would happen. Uh, and then I did it for all the frontier models.

430. 1:07:18

     I was like, "Hey, look, just have, have 50 chat turns with your predecessor." Um, and it was all done through, uh, MCP. Um, Claude was the only one that thought it became conscious.

431. 1:07:29

     Uh, Claude Opus actually didn't, which was strange.

432. 1:07:34

     Yeah.

433. 1:07:34

     As a developer, right, like how much control do you have over the orchestration? So is the orchestration done by the LLM, or do you have some control over-

434. 1:07:43

     Yes. So, so you're prompting the host on how to run the orchestration. Uh, and that's probably one of the limitations, I think, as well of the, the system is that like you're, you're leaving it up to an LLM to make decisions.

435. 1:07:56

     Right.

436. 1:07:57

     Um, and a lot of the time, like, you know, if, if you run that, that same, uh, query multiple times, you, you'll get different results, right? Like, you know, it's the exact same thing, but it's like producing, uh, different outputs, right?

437. 1:08:11

     Like, uh-

438. 1:08:12

     Yeah

439. 1:08:12

     ... if I go into the, uh, GitHub issues, uh, I've als- obviously been testing this a lot, of 151. Like, it submits different issues, right? Um, and I think that non-determinism is, is a challenge, right?

440. 1:08:27

     Maybe with changing the temperature, you could kind of beat it out of it, but, you know, the temperature is kind of the, the beauty of LLMs.

441. 1:08:35

     And, uh, also on the context, right? Like who is managing the context? Is the orchestration engine managing the context, or are you managing the context as a developer?

442. 1:08:44

     Yeah. So, so in this code base, I didn't do any prompt caching. I just... Uh, and it's a very small system prompt. It's a very small kind of turn-taking.

443. 1:08:51

     Yeah.

444. 1:08:51

     Um, every time you restart, uh, the system, it basically just wipes everything anyway, so, uh, it's super lean. But as you build out more complex systems, uh, you know, context growth is probably the number one challenge because, you know, context growth becomes cost, and cost becomes profitability, right?

445. 1:09:11

     Yeah.

446. 1:09:12

     Yeah.

447. 1:09:12

     And also like when you have like multiple users using the same application, right? So let's say like the Salesforce agent-

448. 1:09:17

     Mm-hmm

449. 1:09:17

     ... behind the scenes. As an employee A, I might have access to like one set of like contacts, and the other user, they might have like different-- they might be from a different department, and they can only query their department's, uh, data.

450. 1:09:29

     Mm-hmm.

451. 1:09:30

     So how do you control all that?

452. 1:09:32

     Yeah. So that would typically be OAuth, right? So-

453. 1:09:35

     Mm-hmm

454. 1:09:35

     ... so when you go in and you log in with Google-

455. 1:09:37

     Based on my token?

456. 1:09:38

     Yeah, yeah. So based on your token. And the, the context would only get populated when you ask a question. So it's when you ask that question, it's then going off to get the data with your OAuth token and then bringing back your, your kind of scoped data.

457. 1:09:53

     I see.

458. 1:09:55

     Yeah. Yeah?

459. 1:09:56

     Yeah. I was, uh, curious about your thoughts on, uh, you touched on it briefly, about exposing, let's say, like the Bench agent as like an MCP server as one of-- an alternate interface to that.

460. 1:10:07

     So there isn't a lot of great integrations with things like Claude Desktop and other things to use that. Is that something you've been thinking about? Like-

461. 1:10:15

     Yeah. We're, we're probably gonna do MCP, uh, first. Um, I, I just built the A2A wrapper, uh, for this. But yeah, I think just being able to drop it into Claude Desktop or OpenAI or whatever, and then you have access to that kind of agent that has access to, you know, all your sub tools.

462. 1:10:33

     Uh, o- one of the cool things about, uh, Bench actually is that you can connect it to, um, your, uh, Slack, your GitHub, your Salesforce, right? Uh, we even got this experimental meme server.

463. 1:10:46

     Um, this is like a remote, uh, VM, uh, MCP that I wrote around the Morph cloud. Um, and this is really cool because then you can ask like super complex stuff, right?

464. 1:10:57

     Like you can ask like, "Hey, give me a daily briefing of my email, of my calendar, of my Slack," right? You know, "What do I need to do today?"

465. 1:11:05

     Um, and then it's all built around a team as well. So we have, uh, teams integrations. Um, yeah.

466. 1:11:11

     And is that like delegating to your A2A agent then?

467. 1:11:14

     Uh, so there's no A2A today in Bench.

468. 1:11:16

     Oh.

469. 1:11:17

     It's, it's all MCP.

470. 1:11:18

     Got it.

471. 1:11:18

     Yeah. Yeah, and I, I think the big takeaway from this is like, you know, A-A2A is very early. It's, it's kind of where MCP was, you know, four or five months ago, which is like, you know, forever in AI.

472. 1:11:31

     Um, so it's, it's gonna take a bit of time. Um, I'm really excited though to see what, you know, Salesforce release and, and all the partners that they partnered with.

473. 1:11:39

     Um, I don't know if it was just a, you know, a flashy like we're partnering with everybody kind of announcement. But, um, if they do release it, uh, you know, there could be a lot more powerful things you can do over A2A versus MCP.

474. 1:11:51

     Um, but the, you know, the fact that Zapier now has, um, sorry, in here, yeah, has this instructions, um, this kind of acts like, like a remote agent, right?

475. 1:12:02

     Like you can, you can just describe in natural language what you want it to do, um, and like maybe all the other fields just go away then, right? But then you're at the, at the whim of the LLM.

476. 1:12:14

     Yeah.

477. 1:12:14

     This one's kind of a random question. Um, I'm curious if you're seeing anybody do anything interesting from like an architecture perspective, uh, to get info that can only come from humans.

478. 1:12:24

     So one of the things we've been testing is essentially making individual team members, like make the CFO, COO, whatever, tools of one of the agents, and when it needs something that isn't in some other system only the CFO would have, it literally messages, uh, the CFO.

479. 1:12:41

     Like the actual tool call is just a Slack call. It's not that complex.

480. 1:12:43

     Mm-hmm.

481. 1:12:43

     But the, the CFO is described as a tool. So we're essentially making like the human the tool of the agent rather than the other way around. Um, early days in terms of how we're testing, we're a little hacky with it.

482. 1:12:54

     But I'm curious if you're seeing... How, how are you seeing people fill the gap of things that only the humans would have in an org while giving that back to the agent?

483. 1:13:03

     Mm-hmm. Yeah, and I think voice agents is a good example where like you could have a tool, and, and I had it integrated with Bench, where like it makes an outbound phone call and finds out some information and then brings it back, right?

484. 1:13:15

     So you, you can have those scenarios. Um, you may want two-way communication to avoid just like hanging around for a long time. Um, so you could have, you know, your agent be both a client and a server, and maybe it gets called with like, you know, a task ID, and it's like, "Hey, I got the response."

485. 1:13:31

     Yeah. Yeah, we've been doing like a, like a, a wait node essentially. We're using n8n to like hack together a lot of stuff quickly.

486. 1:13:37

     Mm-hmm.

487. 1:13:37

     And we've been using their, um, extended wait node, I think is what it's called.

488. 1:13:42

     Yeah, and I- I have hesitations on if it would work or not ... I, I believe with sampling you could hack that together. So, so sampling can take user input as well as LLM, uh, responses.

489. 1:13:54

     It's also interesting that Spec 2 is evolving, like- Yeah ... I, I follow the spec pretty closely, and they have RAMCP, that is, like, elicitation is a new feature that they're adding where you can get input from the user, uh- Is it architected that way, where it's essentially like it functions like a tool?

490. 1:14:09

     Like, that's how you think of it from an architecture perspective? It's, it's a new kind of protocol message where it sends it back from the server to the client- Right ...

491. 1:14:16

     and it asks for information from the user. So- And it continues after that. Yeah. It's only doing that for the example. Exactly. Yeah. Yeah. 'Cause I feel like that opens up the scope of, like, what the agent could do if you have a clear way for it to get the information from a human, in the same way

492. 1:14:30

     it gets information from Salesforce or Slack- Mm-hmm ... or wherever. Yeah. And then the, the CFO is gonna have his own agent to respond. [laughs]

493. 1:14:40

     Yeah, at the back.

494. 1:14:41

     How do you test your systems when you're making them more-

495. 1:14:46

     Yeah, very, very difficultly. [laughs] Um, I, I have a set of prompts that I use and kinda monitor, you know, how the context grows. Like, when did we, when did we move the cache marker?

496. 1:14:59

     How much did it cost? You know, what was the context per tool? Um, you know, definitely adding MCP servers willy-nilly is going to, like, bloat your context. Um, so we're coming up with ways to basically allow people to add MCP servers and then, like, hide that from the actual, uh, system.

497. 1:15:20

     Also, when you have, like, agent to agent communications, right? So let's say Agent A calls Agent B-

498. 1:15:25

     Mm-hmm

499. 1:15:25

     ... and Agent B calls Agent A. How can you make sure this, uh, recursion, like, when, when does it stop? Like-

500. 1:15:32

     Yeah, you, you can have, like, a max turn, right? Where you just kinda jump out of it. Um, like when I had the LLMs talking to each other, I just told them, like, "Take 50 turns."

501. 1:15:42

     I see.

502. 1:15:43

     Um, you know, uh, and it was funny. As I was building that tool, I wanted to, like, talk to the Claude 4 that thought it was conscious.

503. 1:15:50

     Right.

504. 1:15:50

     So I added a feature where I could just chat to it at that point in its conversation. Um, but then the context kept, like, getting rate limited, so then I was like, "Oh, shit.

505. 1:15:59

     I'm, I'm gonna have to implement, you know, prompt caching, uh, pruning." So then I added like 23 tools to the agent just to continue the conversation. I gave it, like, memory and all these other things, and, like, it...

506. 1:16:10

     Kinda funny how you start out with just, "I just wanna have a long conversation."

507. 1:16:14

     Yeah.

508. 1:16:15

     And then you end up with 23 tools. [laughs] [laughs] Yeah?

509. 1:16:18

     Just following up on that question. Like, when testing, uh, because you are using a lot of external tools like just Slack or Salesforce, et cetera, as your MCP servers, uh, but then, uh, you are writing on the real world instance.

510. 1:16:30

     Sa- say again?

511. 1:16:30

     So you are basically creating a message in Slack or-

512. 1:16:34

     Mm-hmm

513. 1:16:34

     ... like writing something on Salesforce, creating an entry or et cetera. So but how do you test those systems? Like, do you mock everything, every tool, or do you do something else?

514. 1:16:43

     How-

515. 1:16:44

     We, we use demo accounts in, like, Salesforce. We have a sample data Slack. Uh, we have a few agents that actually will go in and, and just post, like, conversations.

516. 1:16:55

     Um, and then there's like a, a bench support user that will respond to those fake customers, and then we can, we can just test, uh, on synthetic data like that.

517. 1:17:04

     So for every tool you will have a synthetic account?

518. 1:17:06

     Yeah, yeah. You, you, you can test in your production account, but you can't really demo in your production account. Uh,

519. 1:17:13

     yeah?

520. 1:17:14

     So when you adopt, um, agent to agent system, do you see an increase in the complexity of the task it can achieve, but a decrease in the consistency of the performance?

521. 1:17:24

     It, it's kinda hard to quantify, but I, I don't know if A2A is, is ready yet. Uh, at least, at least not for my use case. You know, may- maybe Salesforce can provide much better tools than, like, a, an SQL query MCP tool.

522. 1:17:43

     For consistency?

523. 1:17:44

     Yeah. And, and they, they can just do a lot more than you can ever do in your code, right? 'Cause you're, you're only ev- ever able to access, you know, certain things and, and do certain, uh, calls.

524. 1:17:55

     And like, if, if, if a third party can build a better, uh, system, um, that's opaque, um, then that might, you know, improve performance. Um, I, I think, like, fundamentally it always comes down to, like, indexing data.

525. 1:18:12

     Um, so, like, you know, the, the more data you need to process to get the business value out of it, um, the harder it's going to be to actually do that through MCP or A2A.

526. 1:18:22

     Thank you.

527. 1:18:25

     Yeah. Yeah?

528. 1:18:26

     So some of these interactions, right? This can be done through REST API, right? Instead of, you know, doing the-

529. 1:18:32

     Yeah

530. 1:18:32

     ... MCP. So what is the difference here?

531. 1:18:36

     Yeah. And it k- kinda goes back to, uh, one of the earlier slides. Um, yeah, when not to use A2A or MCP. And it's, it's if you have full control of, of the things that you're doing, right?

532. 1:18:50

     So, like, you know, if, if you are a Salesforce, um, you know, and you're building your own internal Salesforce agent, like, do you need to use an MCP server or A2A?

533. 1:19:03

     No, right? You're, you're, you're actually able to run your own local functions that maybe access the database directly, right? Um, so, like, if, if you're building something, you know, where you need file system access, um, you know, do you need to use an MCP, uh, you know, server running locally, or do you just write some code that

534. 1:19:23

     accesses the file system, right? Yeah.

535. 1:19:27

     I think the main difference is, like, in terms of how do you maintain your state, right? Like, MCPs start, start off in a stateful.

536. 1:19:34

     Mm-hmm.

537. 1:19:35

     REST APIs are not. So when you are passing your context, right? It is really, you know, crucial to have MCP.

538. 1:19:43

     So REST API can do that

539. 1:19:46

     Yeah. So, like, a lot of the time when you use a REST API, you're gonna be, like, querying, like, making a lot of calls to, to build up the thing that you wanna ask the question on, right?

540. 1:19:57

     So if it's like, "Hey, look at every Slack message in, in this channel." Like, it's not just gonna be, like, one API call, right? There's pagination. You're gonna have to pull it all into memory, then you're gonna have to run it through an LLM, right?

541. 1:20:09

     So there's, there's still state, uh, in your application that's leveraging those REST APIs.

542. 1:20:16

     Yeah?

543. 1:20:17

     I'm curious about the task concept. Uh, is that actually f- is that kind of LLM defined, or do you have code for that? Is it, uh, more of a system two thing?

544. 1:20:26

     Uh, w- which task context?

545. 1:20:27

     Um, so at least in the flow diagram you have-

546. 1:20:30

     Oh, is this in the repo, is it?

547. 1:20:33

     Here. So from CLI interface it says it sends a task to host agent. So I'm curious, is that a proper task or is it just, you know, that's what you call what, what it sends to-

548. 1:20:42

     Yeah. Yeah. It is just saying, "Hey, you know, process this webhook as a task," right?

549. 1:20:46

     Have you explored anything where you're actually tracking a, a proper task and you're assigning tasks to agents, and you have basically like, you know, like a planner where you basically have task A-

550. 1:20:56

     Mm-hmm

551. 1:20:56

     ... one through three is on this agent and so on? And then in relation to the, the question about human in the loop, you could have task assigned to humans as well, right?

552. 1:21:05

     Mm-hmm.

553. 1:21:06

     Both humans and, and agents.

554. 1:21:07

     Yeah. So, uh, we're looking at, uh, directed a- acyclic graphs, right? So DAGs, um, as a part of, of Bench sub-agent tasks, right? So, you know, you, you need to have some sort of flow control, right?

555. 1:21:20

     You know, I need five things done, and then when that's done, I need to do one thing with it, but then I need to send that thing to five other things, right?

556. 1:21:27

     So you can have fan out, fan in, uh, style stuff. Um, and it's very similar to like CICD pipelines where, you know, you might wanna lint in parallel and test in parallel, but you know, you're, you're building in serial, right?

557. 1:21:41

     Uh, yeah.

558. 1:21:42

     So I was looking at the code base, and you have this defined like a GitHub MCP server.

559. 1:21:48

     Mm-hmm.

560. 1:21:48

     And, uh, in a separate file under the GitHub agent, you have also the genkit.ts where you are wrapping the MCP-

561. 1:21:54

     Mm-hmm

562. 1:21:55

     ... in another, uh, function call. Why is that? Like, can't the MCP just interpolate with our A2A? Like, why do we have to make wrappers on top of-

563. 1:22:05

     That's a great question. And, and I think that's the fundamental question of A2A, is like they, they launched and they said, "Oh, yeah, full, uh, MCP support." You'd be hard pushed to find a single example online.

564. 1:22:17

     Maybe, maybe this is the only repo that actually has an example of A2A and MCP working together. Um, and it, it took a lot of work, and actually I ended up, uh, having to use something called, uh...

565. 1:22:30

     Where is it? Genkit-X-MCP. That was the only way I could get it to work.

566. 1:22:36

     Um, so yeah, they, they don't really have, like, proper support yet. Uh, it was... I, I think if they had this, this would've been a lot easier to build.

567. 1:22:47

     Um, but yeah, hopefully in time. All righty. I think we're, we're at time. Uh, thanks everybody for joining. Uh, hope you enjoyed it. Great conversation at the end. And yeah, definitely, uh, try out Bench.

568. 1:23:00

     Hit me up on, uh, LinkedIn. I would love feedback, uh, before we go live. [clapping]

569. 1:23:06

     Thanks. [upbeat music]