How Google DeepMind Runs Agents at Scale — KP Sawhney & Ian Ballantyne, Google DeepMind

Talk transcript

1. 0:00

   [upbeat music] Okay.

2. 0:16

   Hello, everybody. Uh, so my name is Ian Ballantyne. I'm a developer relations engineer at Google DeepMind.

3. 0:23

   Hi, folks. I'm, uh, KP Sawhney, a software engineer in, uh, DeepMind's AI platform team.

4. 0:29

   Uh, and we're gonna do an agentic panel today to talk a little bit about, uh, how DeepMind thinks about agentic software, how we build kind of our own stacks.

5. 0:37

   Um, we're gonna start very briefly by just showing a quick demo, and then we'll go into like a, like a discussion about some of the stuff that KP works on.

6. 0:45

   Um, and hopefully we can just... You can ask lots of questions, find out all the things you wanna know about how DeepMind and Google think about agents, and, uh, yeah, let us know your thoughts as well.

7. 0:56

   So quick show of hands, who's actually used Antigravity before as a tool?

8. 1:01

   Okay, so like four or five people. Um, I just wanna show you one quick thing 'cause one of the things that we have, uh, in Antigravity is a lot of people know that it's like a Visual Studio style interface.

9. 1:13

   But they don't actually know that it actually has a whole, uh, agent manager and agent manager framework behind it. So you can run and spawn multiple agents working on different projects, um, and it's integrated into the IDE, but it's also again kind of like an agentic platform in itself.

10. 1:28

    So you can do things, uh... I've got this project here, and you can have this chat panel on the side, and I can just say like, uh, build an example of this spec, and then I can give it a particular file.

11. 1:43

    Uh, let's do the spec file. Let's just implement that one. And I'll send that off to Flash. Oh, experiencing errors right now. Let's restart. Should we try that? [chuckles]

12. 1:56

    I am connected to the Wi-Fi. I'm pretty sure I am.

13. 2:02

    Yes. And if you're not connected to the Wi-Fi, you should use the Gemini models instead. That's my plug. [laughs] Uh, so let's start that up again.

14. 2:10

    And hopefully I can do the same thing. Build,

15. 2:14

    uh, build the spec marked here. Okay. Let's see whether we get it this time. Okay, there we go. So again, the model will just go away thinking, uh, on the side there, and hopefully what it should do is it should have a look at the spec.

16. 2:34

    It should analyze it, and it should use a bunch of its in-own internal tools to decide how to do that. Uh, it's actually found that there are already existing files that I have built.

17. 2:41

    It might even tell me that it's already done it, [laughs] uh, which is interesting. But the tool itself has like built-in to-dos. It has a, like a planning system, and what it's just done here is it's thrown up a browser, so it can actually check, uh, how the applications run.

18. 2:57

    And this is also- is controllable by, um, uh, by Antigravity itself. So if I just do that, if I give it two seconds, it should take control over the browser, and it should actually try and run.

19. 3:07

    So what it's probably doing now is it's probably looking at the implementation already to see what's actually already done, and it's gonna then analyze it. Should we just have a quick look, see what it's actually done there?

20. 3:17

    Uh, let's try that again. It'll pick up. So it's hit a snag. So it wants to try and analyze it. Oh, uh, one thing that it can do is it can actually inspect the DOM.

21. 3:29

    So it can look at like the actual web page itself as part of that and give-- and feed that back. And when it finally finishes doing the task, it will give you like a report at the end as to what it was able to achieve and what it im-implemented, so you can then review.

22. 3:42

    It can also capture, for instance, like a, a screenshot or like a video. So if it's like an interaction, if you say like, "Add this feature to my webpage," it will go through and then actually try and run it.

23. 3:52

    I don't know how good it is at playing games, but we're about to find out. It's either gonna try and play the game or, uh... Should we have a look, see what it's doing at the moment?

24. 4:00

    It hasn't loaded it yet. Let's go back. Okay, we've got an implementation plan here. So this is what it thinks it should change about the file. So what you can do is you can just go in, and you can like edit a line and say like, "Actually, I want this different behavior," or, "That's not what I meant

25. 4:15

    at all." So it kind of gives you that human-in-the-loop feedback. And then when you're done, you just say proceed, and then it will go away and do that. Um, while we're waiting for that to happen, do you want to say a little bit about what you work on and like how this kind of relates to the-- your

26. 4:31

    kind of day-to-day job?

27. 4:32

    Yeah, sure. So, um, one of the things I, I worked on a few months ago was the, the deep research agent, which is now available via the, the Interactions API.

28. 4:42

    Um, and so that's, that's been great. But as we continue to iterate on that, my, my focus has turned now to making best use of, of this h- Antigravity harness internally.

29. 4:53

    And so of course that, that applies to scaling it for, for all of the coding we're doing, and, you know, we have a gigantic mono repo, so it's pretty complex.

30. 5:03

    Um, but now we're starting to think about how we generalize this to a variety of other use cases. So potentially deep research itself, rather than passing around huge, huge blobs of, of text from, from the searches that have been done.

31. 5:17

    Why not have the different parts of that pipeline collaborate in, in a shared file system? Um, and so yeah, that's, that's really been the focus for me, really, um, tightening up this harness and, and making it excellent at, at not just coding, but a variety of other tasks too.

32. 5:34

    And do you have any interesting use cases for like how people within Google or DeepMind are kind of using it at the moment? Like what kind of, what kind of things are they doing with, with the agents in DeepMind?

33. 5:46

    Yeah. So, so there's a huge amount that's, that's been going on. Um, we have quite a few exciting things that you'll probably learn more about in a few months at I/O, which I, uh, can't go into too much detail about right now.

34. 5:58

    Um, but at least internally, there's been a huge, um, amount of focus on, on building up a huge library of, of skills that, that enable folks to, to do their job better.

35. 6:09

    Um- And, you know, skills are great, but in an organization as large as, as Google, um, there's a risk of skills really sprawling out, out of control. Um, and so that's a, that's a big area of focus for us right now is, um, improving those skills, making sure that only the best ones, uh, survive really, almost Darwinian

36. 6:30

    nature. Um, but it really is helping folks to, to deliver good code at a way faster pace, which, which is obviously awesome.

37. 6:40

    Awesome. Thank you. Uh, so you can see this little blue bar around the edge at the moment. That's the, uh, Antigravity taking control of the, of the game. It seems to figure out how to start the game.

38. 6:50

    I don't think it knows what the controls are, so maybe might end up looking those up. [laughs] Um, but yeah, this is... So this is what it edited. Uh, the file that was there before was actually generated by a different model, so I can tell you it's not even the same game.

39. 7:02

    It's completely rewritten it from scratch, um, based on the spec we gave it. Um, and then what you would get at the end... Oh yeah, you can see there, it's like actually li- looking at the DOM, looking for any errors, trying to figure out how to actually use it.

40. 7:14

    You should get a video at the end here.

41. 7:19

    Uh, oh yeah, this is a... It's scratchpad. This is like its notes that it's writing as it kind of goes through and does the task. So you can, like, you can actually get a bit of a trace as to what behaviors it's trying to figure out.

42. 7:28

    Again, you can go in and you can review these things. If you don't like what they're doing, you can kind of interrupt it. So in terms of, like, the workflow, this is kind of, I guess, pretty common to, like, a lot of different agent harnesses at the moment.

43. 7:40

    Um, but this is how we kind of think about using, using the things like Gemini models, um, as well, so that we can use it for our own development.

44. 7:49

    Um, I will close that off now. So I guess one big question on my mind related to, like, uh, agents, how do we do things at kind of Google scale?

45. 8:02

    So if we think, you know, research, Deep Research is a feature within Gemini app, but then also for everyone within Google to use it, like, what kind of challenges come along with that kind of scaling of agents?

46. 8:14

    Yeah. So I mean, the, the, the thing that's kind of top of mind for us at the moment is, um, how token hungry this, this stuff can be. Um, and so as you can imagine, just, just managing the quota on a, on a per user or, or per team basis is, is, is really quite, um, important.

47. 8:31

    Um, and so there's a lot of work we're doing around making that, uh, more efficient and, and, and lower cost. Um, and I think we're- Ian and I were chatting before, and, um, I think what's going to be really interesting for, for folks like yourselves is, is mixing and matching between models like Gemma 4, which, which are

48. 8:52

    effectively free from a quota perspective, just using your whatever GPUs or TPUs you have. Um, and then, you know, using the more advanced, uh, models for, for specific, uh, components of the agentic system.

49. 9:06

    Um, and so also e- e- evaluation as well is, is a big, um, thing we're focusing on at the moment. Um, particularly with these really complicated, uh, workflows, how, how do you actually evaluate that it was successful?

50. 9:20

    How do you minimize the cost of that? So looking into things like mock TPUs so that, you know, you can test the harness itself and, and the agentic flow, but not necessarily using up a ton of, uh, TPU hours.

51. 9:34

    Yes, 'cause I mean, I'm sure people are aware of this, but like [laughs] definitely limited in that capacity at the moment within the world trying to get enough compute to do a lot of this stuff.

52. 9:44

    Uh, just quick show of hands from the room. Who's, who at the moment is either building their own agent architecture or harness at the moment?

53. 9:52

    Okay, fantastic. For... So just quick question, like what kind of scale are you looking at? Who are you trying to... Who are your customers? Anybody?

54. 10:05

    Personally, I'm building for myself first, and also the next thing is going to be for the hospitals to use.

55. 10:11

    Mm.

56. 10:12

    Uh, it's going to be for the radiology department because they're going to like com- they're going to try to combine EHR data with the radiologist data. So that is what we are doing.

57. 10:24

    Okay.

58. 10:24

    And probably we're going to use demo models because like we want open source models.

59. 10:30

    Mm.

60. 10:30

    We want open source models.

61. 10:35

    Gotcha. Is anybody using any harnesses off the shelf or you kind of built your own ones? Like what's the general vibe?

62. 10:42

    So we're building memory as a harness, uh, infra layer, um, essentially an ontology-driven complex graph, uh, that developers can use, fully open source, 15,000 PK stars.

63. 10:55

    Oh, nice.

64. 10:56

    Nice, nice.

65. 10:58

    And how many users are you... Do you expect to be able to scale that to?

66. 11:01

    As we know, uh-

67. 11:02

    Okay

68. 11:03

    ... we know like pipelines, like generation of complex graphs, and it's around 3 million per month.

69. 11:09

    Wow.

70. 11:10

    Wow.

71. 11:10

    That's some scale.

72. 11:11

    You know, like one user can create a thousand in a day.

73. 11:15

    Yeah. [laughs] [laughs] That's the, yeah, the, the challenge too. I'm sure that's the challenge for us too. How do you stop like one user from... Well, actually, okay, I'm going to turn that into a question.

74. 11:23

    How do you stop one user from taking down a whole system? We could by-

75. 11:26

    By making-

76. 11:27

    ... spawning multiple instances and multiple... 'Cause I'm sure that's like, you know, the more we get better at doing this, these tasks and like we've got power users kind of spinning stuff up, and they've got their team of 100 people like working for them, 100 agents.

77. 11:42

    Like how do we kind of manage like the per user like behaviors? If you see what I mean.

78. 11:47

    Yeah, no, it's, it's, it's a great point. And, and like I said earlier, honestly, right now it's kind of brute force with the, the quota. So we have, we have some real power users at DeepMind, and ultimately it gets to a point where it's like- Okay, you've got to just stop right now. [laughs]

79. 12:03

    Um, but, you know, i-i-in general, I think that raises an interesting point about how this stuff is going to be priced in the future as well. Like, uh, you know, you saw, um, Anthropic, uh, blocking the, uh, the OpenClaw, um, stuff, um, because these, these agentic systems are so token hungry, and the subscription model doesn't really work

80. 12:23

    for that. Um, and so yeah, I think that-that's really top of mind for me as well right now, how to, how to mitigate that.

81. 12:30

    Yeah. I, I always used to joke when I joined Google that, you know, you've got all these resources, you know, available in data centers to use for different projects.

82. 12:38

    Like, how do you know when too much is too much? And one of my colleagues once told me, he just said, "Oh, they'll tell you." And I'm like, "Who's they?" [laughs] [laughs]

83. 12:46

    And yeah, sure enough, there's people monitoring these things twenty-four/seven, like, you know, looking at spikes and graphs and all our SRE, SRE team and, and they do just, like, reach out to you and say, "Can you just, like, stop this job running on this one cluster, please?"

84. 13:00

    Uh, so yeah. I think, yeah, that's also an interesting one. Ev- any questions from the audience?

85. 13:06

    Well, I think if you're an agent at such a huge scale, how do you do observability and the tracing? Because that-

86. 13:12

    That's a fantastic question. Um, so we built a, um, sort of custom, uh, web app for that essentially. Um, and essentially there's, there's one, uh, agent backend system that is used for, for a lot of stuff at, at Google.

87. 13:28

    And essentially anytime, um, a user issues a, a query to, to an agent hosted on that system, it then sort of automatically appears in this UI, um, where you can drill down at various levels of, of hierarchy each of the pieces of the, um, of the system.

88. 13:46

    Um, if needed, you can drill all the way down to the sort of raw predict request made to the model and, and so forth. Um, and so that's been useful.

89. 13:55

    Um, and we also have a concept of, like, an agent trajectory store as well. Um, that's more focused around the, the coding piece, where obviously you can have a huge number of, of steps, uh, going on there.

90. 14:09

    Um, and it can be really important to kind of diagnose at, at what exact point looping started happening or, you know, the model went off the rails. So yeah, it's all custom internally, uh, for now.

91. 14:21

    But I'd be interested to hear what, what you folks use for, for observability as well.

92. 14:26

    Well, I'm just transporting it to a log file and then basically going from there. [laughs] [laughs]

93. 14:33

    Yes. Any other questions? Yep.

94. 14:36

    Yeah. Is, uh, so Deep Reasons-- Gemini Deep Reasons is something I've used quite a bit. I really, really love it.

95. 14:42

    Oh, awesome.

96. 14:43

    Is there, uh... Is that going to be available in Antigravity?

97. 14:47

    Yeah, I mean, that's a, that's a fantastic question. Um, obviously I can't go into sort of too much detail on, um, on sort of like release plans and so forth.

98. 14:55

    But what I can tell you is, um, yeah, that's something we're actively exploring. Um, you know, we hope that it, it will make it faster, um, cheaper and, and, and hopefully better results if we can effectively orchestrate that, that deep research using the same harness.

99. 15:12

    Mm.

100. 15:12

     Um, and 'cause, you know, right now, uh, without going into too much detail, there's a, there's a huge amount of, of context that's, that's passed all the way through that deep research system, um, which gets quite expensive and, you know, consumes the context.

101. 15:28

     And so we're really thinking about, okay, how, how do we make each element of this system more like a collaborator as, as part of a, a workspace, um, which is kind of how it would work if humans were, were researching something deeply, right?

102. 15:41

     Um, and I think that opens up, uh, a lot of nice potential for things like, um, infographics and additional supporting artifacts and documents. So yeah, it's definitely an area of focus for me.

103. 15:55

     Yeah. How-- Let me ask, sorry, just to follow up. Uh, how many, uh... I struggled a little bit with this with Copilot. How many levels of sub-agents can you run, uh, within Antigravity?

104. 16:09

     That's a great question.

105. 16:12

     Sub-agents versus another one, another one like me-

106. 16:13

     I-- Good question. I don't know is the short answer to that. Um, the...

107. 16:20

     I think we-- They're not-- They're a bit more, um... What's the word? They're a bit more opaque in how they're kind of presented. The way, the way you kind of think about it is like you have multiple simultaneous ones working on different tracks, but it's not-- We don't kind of...

108. 16:36

     Yeah. The, the short answer is it's not as obvious as to which agents are actually working w- on a particular task for that. It's not kind of like a massively parallel system in that sense.

109. 16:44

     It's more like you can give them different, um, like, trains of operation a-among a particular project. But you can tell them to kind of work on specific things, or you can have jobs that kind of overlap a little bit, but it's not kind of...

110. 16:56

     Yeah, I don't have a huge amount of detail on the, like, the specifics of how the sub-agents work.

111. 17:01

     I do think that's gonna be, um, the, the future though, is how do we make agent-to-agent communication, uh, efficient, and then also how do we give us as the human the ability to really shape that and, and almost act like a supervisor on a digital assembly line, you know?

112. 17:18

     Um, so yeah, watch this space, I guess.

113. 17:21

     Question?

114. 17:21

     You were next.

115. 17:22

     Um, there has been a recent debate, of course, you know, skills, MCPs, CLIs. Uh, community is going so fast about it. Uh, we have a trend every single week.

116. 17:33

     Where do you see with Antigravity this thing going? Like, uh, we-- Is it going to be a combination of skills with CLIs, self-improvement? What's your take on that?

117. 17:45

     Yeah. For me, I, I really like, uh, skills and, and they've been working very, very well for me. Um, uh, perhaps this is controversial, but I did always think that MCP may be a little bit of a flash in, in the pan.

118. 17:59

     Um, I like it for the-- f-from the auth perspective. I think that's, that's very powerful. Um, but for me, a combination of, of, of skills and, uh, guardrailed CLI, um- ...

119. 18:12

     interactions has, has, has worked really well. Um, and it, it speeds up my job so much, you know. I've got a skill for, to your point, to de- debugging raw logs and, you know, it can do most of that from the, from the CLI.

120. 18:25

     Um, and in a business of our size, the, the great thing is we have these skills contributed by folks who are absolute experts in that particular area, and then I kind of...

121. 18:35

     I and the agent get that knowledge for free, you know. So I'm definitely Team Skills, if that helps.

122. 18:41

     I mean, we, we support both of them, and I think that's the intention going forward. Again, it's like what the community uses, like, you know, we want to make sure that they work with the harness, work with the models.

123. 18:50

     So I think, yeah, whatever you guys keep using will probably still be supported is probably the way to think about it. Yep. Uh, at the back. Shall we go with that?

124. 18:59

     Yeah, on skills, uh, KP, you were saying earlier that, uh, there was an explosion of skills, um, and only the best survive. But how are you evaluating skills and, I guess, the downstream issues that come with that?

125. 19:12

     Yeah, no, that's, that's a fantastic, uh, sort of point. Um, eval-ing this stuff is really hard. Um, even just the, the mechanical nature of spinning up all of these sandboxed environments set up in, in, in, in the way needed to, to evalua- evaluate a particular, um, problem set.

126. 19:32

     Um, I think the, the trickiest part is coming up with new data sets, you know. Um, there's a lot of good, um, you know, open source ones that, that are good for benchmarking externally, but you're right.

127. 19:45

     For specific skills, the onus is almost on like the, the, the author of the skill itself to r- to come up with, um, some form of test in that.

128. 19:54

     But people are also experimenting with the agents designing that as well. Um, so it's a little bit meta. But, um, yeah, a lot of work to do in that space.

129. 20:04

     Anyway, a question over here.

130. 20:06

     Yeah. When I'm using Antigravity, like the browse testing, I love when it works.

131. 20:12

     Mm. Mm.

132. 20:12

     Because I am really struggling, uh, with the limits. Uh, so you also experience like the retry or like the high network? Are there any things from my side I can tweak?

133. 20:23

     Like, how do I know, uh, when I hit the limits?

134. 20:26

     Uh, you mean in terms of for like the model and the model usage?

135. 20:30

     Yeah. I, I, I have like AI Ultra plan-

136. 20:33

     Yeah, yeah

137. 20:33

     ... highest I can get, but I still,

138. 20:37

     yeah, I struggle with like maintaining, uh, the limits, and I don't feel Antigravity is telling me like, "Hey, it's now not working, not because of the network, but because you've hit your limit."

139. 20:49

     Mm. Yeah, that's a good question. Uh, it's, it's funny actually 'cause we have, we have worse limits than you do. [laughs] Because obviously we prioritize customers and not ourselves, so-

140. 20:58

     Yeah

141. 20:58

     ... the fact I was like clicking like 100 times to keep going is because it recognizes I'm a Googler, and that's my fault. Um, you know, it's a good question.

142. 21:05

     Um, I think there's kind of two parts to this. I think the, uh, there will always be limits, especially like within different tiers. I don't have Ultra, for instance, so...

143. 21:15

     Sorry, my corporate account does, but my personal one doesn't, so I have, I have to change my behavior. But I think we're going to end up in a world...

144. 21:22

     I mean, we have hopefully somewhere around here is Kevin from the Antigravity team who could probably talk specifically on that. Um, but I think we're gonna be in a pattern whereby it's commonplace for you to run out of credit or run out of capacity on like one model or something and then move to something else, but do

145. 21:38

     it seamlessly under the harness. So you can give us preference rather than having like, you know, you've hit your limit on tokens for the pro model, so it will automatically put you onto Flash.

146. 21:48

     Or, uh, you've reached your limit on everything you have in your, in your subscription, so you use a local model or... But it, uh, but to your point, like that it will not interrupt the workflow that you're doing or, you know, the notification you get is not, is not the completion of the task.

147. 22:01

     It's like, oh, we've run out of quota, sorry. Like, you know, what if you were off doing something else while you'd sent it off doing a job and you come back to find that it spent the last hour not doing anything because you hit a limit?

148. 22:12

     So I think the way... When I spoke to him last about this, it was very much more about like trying to make it clearer when you get to that point and making sure that you can.

149. 22:21

     So yeah. Uh, we have time for, I think, one more question. You want to have a go there?

150. 22:24

     Uh, I have a question in, I guess, in terms of how guys review PRs at Google, as in we have like thousands of PRs today-

151. 22:32

     Mm. Mm-hmm

152. 22:32

     ... and, uh, maybe closing the development loop somehow. Uh, I wondered if you maybe thought about fine-tuning some like PR models that would go through like your past Google comments and commits and perhaps then, uh, better review that code and produce the work?

153. 22:48

     Yeah, so, um, you're, you're right. Um, one... A- and I really like the fact that we have some really good stuff in place for that already. So, um, uh, the way it works is, you know, on a per language basis, we have a specific like auto review model that has been, um, fine-tuned on all of our style

154. 23:07

     guides and, and, and all the rest of it, um, and previous like good examples of, of code. But then also on a PA or, or, or product, uh, basis, folks will come up with their own, um, like specific SIs and, and prompts and so forth to, um, to make sure that, um, you know, uh, the, the, the,

155. 23:28

     the other reviewers get a good signal for, for how good the code is. Um, and just yesterday actually, I sent a PR for review and, um, and, and I didn't even have to trigger the auto review thing.

156. 23:40

     I just got an, an agent that someone had spun up commenting on, uh, the PR with, uh, quite a good suggestion. So yeah, um, I think it's important and I think to the point earlier about us being supervisors of a digital assembly line, it's like how do we get help with that piece of it as well?

157. 23:58

     Mm.

158. 23:58

     Then we can all go sit on the beach. [laughs]

159. 24:00

     Yeah. Yeah, we... Uh, to, to your point, like we... You can imagine like the scale of all the Google engineers submitting 100,000 lines of code now being done by agents submitting even more code with more reviews.

160. 24:11

     They've built a lot of kind of infrastructure for us. But then also we have like tools like Jules, for instance, like which are... If you've ever played with that, it's like you've got like a web interface where you can go and do that on your own PRs and GitHubs, so like...

161. 24:22

     And you get review components of that. So yeah, I think this is, I think this is an area that's gonna be with the ballooning of... I think there was a comment yesterday about like, uh, the amount of m- the trillions of lines that GitHub is getting at the moment generated by agents and like that process.

162. 24:37

     So as much as we hate our own boring work, I'm sure the agents hate their boring work too. So we've got to figure out a way to do that.

163. 24:44

     But, um, yeah. Um, well, I think we're out of time for questions, but we will be around to chat afterwards if you want to come and join us, or if you want to head down to the DeepMind booth later, we'll be around there too.

164. 24:53

     So thank you very much for listening.

165. 24:55

     Thank you all. [applause] [outro music]