Proactive Agents

Talk transcript

1. 0:00

   [upbeat electronic music] I'm so excited to be here.

2. 0:22

   Uh, I love New York, and I love meeting everybody here. And I am Kath Korevec. I'm from Google Labs, and I work on this little team called Ada, and I'm gonna be talking about some of the stuff that we've been doing on this project called Jules.

3. 0:36

   So a few months ago, in my household, our dishwasher broke, and while it was being repaired, my husband decided that he was going to do all the dishes. And so he told me he was gonna do this, but every single night, I found myself reminding him to do the dishes.

4. 0:51

   And you can imagine, that got old pretty fast. And I realized that even though I wasn't physically washing the dishes, I was still carrying this mental load. And I know a lot of you can probably relate to this.

5. 1:02

   I was keeping track of whether or not that task was done, following up, making sure that things kept moving. And I realized in that moment that that's exactly where we are with asynchronous agents today.

6. 1:14

   They can handle some of the work, but we're still the ones as developers carrying that mental load and monitoring them. So here's the truth. Humans, we are serial processors, not parallel ones.

7. 1:28

   We can juggle multiple goals, but we execute them in sequence, not all at once. When you manually kick off a task in Jules, you're usually waiting to be able to move on, and it's that pause, it's that gap in attention where we really lose momentum.

8. 1:44

   And this is actually backed up by science, where, uh, humans actually think-- we think we're multitaskers, but we're actually executing many tasks very rapidly. But switching between these tasks comes with a huge cost.

9. 1:58

   It can cost up to forty percent of your productive time. So that's, like, half a day lost to switching contexts and reloading. So if humans are unitaskers, what's the solution here with agents?

10. 2:13

    So for async agents, in order, in order for them to succeed, developers can't be expected to babysit them.

11. 2:21

    We've all seen that post on Twitter of sixteen different Claude code tasks running in parallel on sixteen different terminals on three different huge browser... or huge monitors. And when I first saw this, I thought, "God forbid that that is the DevX of the future."

12. 2:37

    I wanna... I don't wanna manage work. I don't wanna manage my agents. I wanna be a coder. I wanna build. And so we need to think, we need, uh, uh, collaborators in our system that we can trust, agents that really understand context, can anticipate our needs, and they know really when to step in.

13. 2:56

    And then, uh, I think finally we're reaching that point with models where they're getting better and better at executing end to end as long as they understand what our goals are clearly.

14. 3:08

    And that's where trust really becomes this unlock, where you can trust s- the system to know what's missing, to fill in the gaps, and to really keep progress moving forward while you manage on something else, wh- where...

15. 3:20

    while you focus on what matters most. And essentially, we want Jules to do the dishes without being asked. So most AI developer tools today are fundamentally reactive. You open up your CLI or your IDE, and you ask the agent to do something, and it responds.

16. 3:35

    Or it waits for you to start typing, and then it autocompletes a suggestion. And there's a benefit to this model. It's very efficient. It only uses compute when you explicitly ask for it.

17. 3:45

    But the real question I'm asking myself is, "Is this how I wanna manage AI?" And if you think about in the future, imagine a world where compute is not a limiting factor anymore.

18. 3:56

    Instead of a single reactive assistant for instructions, you could have dozens of small proactive agents working with you in parallel, quietly looking for patterns, noticing friction, and taking on the boring tasks that you don't wanna do before you even ask.

19. 4:14

    It can do things like fixing authentication bugs that you've been avoiding, uh, updating configs, flagging potential order, uh, errors, preparing mi- uh, migrations, and all of this can happen in the background, triggered off of things in my natural workflow.

20. 4:30

    So I really think there are four essential ingredients that make up proactive systems today. There's observation. The agent has to really continually understand what is happening and of what your code changes are, what your patterns are, what your workflow is, et cetera, to get context about your entire project.

21. 4:46

    And then there's personalization, and this one's difficult. It has to learn how you work, what you care about, what you tend to ignore, what your preferences are, the code that you absolutely don't wanna ever touch.

22. 4:56

    And then it has to be timely as well. If it comes in too soon, it's gonna interrupt you, and if it's too late, then the moment is lost. And it also has to work seamlessly across your workflow.

23. 5:06

    It has to insert itself into spaces where you naturally work already, in your terminal, in your repository, in your IDE, not forcing you to go somewhere else to some application that's secret or that you forgot about.

24. 5:19

    So bringing all these tools together, you can imagine, is not trivial. [upbeat electronic music] [laughs] [audience laughing]

25. 5:26

    Someone's running this presentation. Um, and, uh, you, you wanna be able to ask your agent to understand your workflow and anticipate your needs and then intervene at exactly the right moment without breaking your workflow.

26. 5:41

    And that's when it really starts to feel like magic. The interesting thing is pro- these proactive systems are all around us today. One of my favorite examples is Google Nest, where you put it in your house, you install it, and then you configure it, and then it starts to learn your habits as you leave the house, as

27. 5:58

    you come back, uh, as you go to sleep, as you wake up in the morning, and then pretty soon you don't have to think about climate control in your house anymore because it's learned what your habits are.

28. 6:08

    Another one is your own body. Your heart rate elevates as you go for a run or start to work out, or it anticipates that you're about to fall, and so it reacts before you consciously think, "I'm gonna put my hand out."

29. 6:20

    So when you look at it like that, proactivity is actually not that-- proactivity for AI is actually not that futuristic. It's very familiar, and it is very human, and that's exactly the point.

30. 6:32

    What we're building is tools that behave more like a good collaborator and less like command line utilities. So we're already doing this in this tool called Jules, which is this, uh, proactive, asynchronous, autonomous coding agent from Google Labs.

31. 6:47

    And we're doing this in kind of three levels of, of, uh, proactivity. Level one is where a collaboration really starts to emerge, and this is how Jules works today, where it can detect things like missing tests, unused dependencies, unsafe patterns, and then it starts to automatically fix those things as it's doing other, other tasks that you've asked

32. 7:07

    it to do. This is sort of like this attentive sous chef in your workflow where it's keeping the kitchen clean, the knives sharp, the kitchen, uh, stocked, so that you can focus on what comes next, and that's the beginning of proactive software.

33. 7:21

    At level two, the agent becomes more contextually aware of the entire project. It observes how you work, the code you write. If you're a back-end engineer, maybe you need help with React.

34. 7:32

    If you're a designer, maybe it wants you to-- may-maybe it'll help, uh, uh, write the database schema. And then it learns what your frameworks are and what your deployment style is, et cetera, and this is the kitchen manager.

35. 7:43

    This is the person in your workflow keeping the rhythm and anticipating what you need next. And then comes level three, and this is what we're working on pretty hard right now going into December, and I'll show you a little bit of what we're, what we're gonna be shipping in December in a minute.

36. 7:58

    But level three is where things start to converge around that context. It's where the agent starts to understand not just context, but also consequence, how these choices are actually affecting the users of your products, the performance, and the outcomes.

37. 8:13

    And at that level, we have this thing, Jules. We also have an agent called Stitch, which is a design agent, and another one we're building called Insights, which is a data agent, and they're all coming together to build this collective intelligence across your application.

38. 8:26

    Jules can see what's breaking in the software, Stitch understands how users are interacting with it, and Insights connects behaviors from real-world signals like analytics, telemetry, and conversion rates. And then together, they can propose improvements across boundaries of how the system all works together, doing things like performance fixes to improve UX and then design changes to prevent regressions,

39. 8:51

    and then all of that is organized based on live data. So the trick here is that the human stays firmly in the loop. You're observing what the agents are doing, you're refining when you, when they, when you need to intervene, and then you're redirecting it when it has, when it has been misdirected.

40. 9:09

    So level three isn't really about autonomy anymore. It's actually about alignment to your project, aim-- ag-agents and humans collaborating together across the full life cycle of your project.

41. 9:23

    So right now, Jules is focused on this code awareness piece. It understands the environment, the frameworks, and the project structures, and we're moving towards more of that system awareness.

42. 9:33

    So things that we're introducing in Jules now, we've added something called Memory, which I'm sure a lot of you are familiar with. It's the ability for Jules to write its own memories, and you can edit them and interact with them.

43. 9:44

    It can edit them, and it understands that and builds this memory and context and knowledge of, of your project as you work with it. We've added a critic agent, which works adversarially with Jules to make sure that the code is, is high quality, but then also does a full code review.

44. 10:00

    And then we've added verification where Jules will write a Playwright script, take a screenshot, and then put that back into the trajectory for you to validate. And then we're also doing things like adding, uh, a to-do bot that will look through your code and look through your repository and pick up on anything that-- where you've said, "This

45. 10:18

    is a to-do I wanna get to in the future," and it will start to proactively work on those things with that context. We're also adding in things like best practices, where Jules will understand best practices and start to suggest those, and also environment setup.

46. 10:32

    We have an environment agent that we use internally for running evals, and we're extending that externally to better understand how environment, how your environments work and, and set those up for you.

47. 10:43

    And then we also are adding something called the just-in-time context. It's like a Jules cheat sheet, where if it's doing something very specific, it can, and, and gets stuck, it can just immediately look at that cheat sheet instead of reaching out to you.

48. 10:56

    So this is all moving Jules very close to being that proactive teammate, not just this reactive assistant. Okay, so this morning, I was talking to my team back in San Francisco, and I was thinking, "Okay, I'm gonna do a live demo," but the live demo gods did not align with me this morning.

49. 11:13

    We still have CLs that are being pushed to staging right now. So I'm gonna walk you through a little bit of this, and if you know Jed, he's gonna, I think, be talking tomorrow.

50. 11:22

    We're gonna, um, affectionately try to fix Jed's code here. [laughs] Um, so this is a view of, of proactivity, and this is, this is Jules where you prompt it, and the first thing you, that you do when you configure and enable proactivity is Jules will index your entire, uh, code base.

51. 11:39

    It'll index your directory and start looking for things that it can do, and then it'll sh- that'll show up on the screen. So right here, we're looking at a little bit more in this, um, in this repository, adk-python.

52. 11:52

    And, uh, and it's indexed the repository, and it's found a bunch of to-dos. It's found a bunch of best practices that it can update, and it's giving me some signal about what it's finding.

53. 12:02

    And so you can see the signal is high confidence, medium confidence, and low. And so it's actually telling me what it thinks it can achieve based on what's in my code and what it wants to do.

54. 12:14

    And that's, so it has high confidence in green, medium in purple, low in yellow way down at the bottom. Um, and so I can go through this, and I can manually click these up and say, "I wanna start these," and so I don't have to think about the prompt.

55. 12:28

    I don't have to look at the code. I don't... I, I can do kind of less cognitive load here. We're working on something to just start these automatically, and so that's coming in the future.

56. 12:37

    But I can also delete these. I can say, "Hey, this one isn't, isn't for me, isn't good." And so once it gets started on a task, I can kind of drill into it and see a little bit more.

57. 12:46

    I can peek into the code that it is suggesting, uh, that, uh, it's suggesting it work on. I can find the location of that code, and it also gives me some rationale about why it wants to work on that code, why, what it's doing, et cetera.

58. 13:01

    And so it's giving me a lot more context and helping me trust that it knows what to do here.

59. 13:09

    Okay, so that's proactivity. That's coming in December, and hopefully we'll be able to give that to everybody here. Uh, we're very excited about it, and I wanna tell you a little story about, uh, something my husband and I were working on just to kind of set, set, wrap things up.

60. 13:27

    We, uh, tinker a bunch with hardware, and we live on this slow street in the middle of San Francisco in [REDACTED:location]. And so on Halloween, we get a lot of people walking by our house, and so we were trying to take advantage of that with our Halloween decorations.

61. 13:40

    And so we built this six-foot animatronic head that sits in the front of our house. It's this old Victorian house. And he sculpted it out of foam, epoxy, and fiberglass, and then I...

62. 13:54

    Our, our kids also called this lovingly the bald head [laughs] and it's based off of, if you ever j- see- saw Pee-wee Herman from the '80s, it's based off of the Pee-wee Herman Pee-wee's Big Adventure's head.

63. 14:05

    Um, so while my husband was doing this, I was spending my time working with Jules on updating the firmware, controlling the stepper motors, working on the, um, on the LEDs and the sensors.

64. 14:16

    And for me, that's the fun part for me is, like, really getting creative with what the LEDs are doing. So I wanted to focus on that, the LED animations, but I ended up spending most of my time actually fixing bugs and swapping libraries and doing things like that.

65. 14:31

    So what I would do is I would prompt Jules, I'd wait 10 minutes, and then I would repeat. And I found that process very, very tedious, and what I wanted was actually Jules to do the research.

66. 14:43

    I wanted it to handle the, the ugly parts, where it was researching how to fix a bug, uh, doing the debugging itself, and I wanted it to do this so that I could focus on the creative parts.

67. 14:55

    I wanted the eyes to move and, like, follow people as they walked down the street and, like, have lasers coming out of its eyes and stuff like... I mentioned it was Halloween.

68. 15:03

    It was very scary. Uh, and, and this... But, but I couldn't really do as much of that, and I ended up actually not shipping as much as I wanted to with this animatronic bald head.

69. 15:14

    And so it's that gap that we actually wanna close. It's the space betwe- with Jules. It's the space between that tool friction and creative freedom that we're trying to unlock with these kinds of proactive agents.

70. 15:28

    So what I really want you guys to take away from it, I give this advice to the, the folks on, on the Jules team a lot, is that the product we build today actually won't be the proj- the products that we have in the future.

71. 15:42

    And I think a lot of us know that, but in reality, I want everybody in this room and everyone building, working with AI to be able to take those big steps.

72. 15:51

    I think the patterns that we rely on today, Git, uh, your, your IDEs, even the code, how we think about the code itself might not exist a year from now, might not exist six months from now, and that's the exciting part for me.

73. 16:05

    It's sort of we get to invent the future right now. We get to describe and decide how software is made and built, uh, kind of all the people in this room.

74. 16:14

    So my, my challenge to you is to not be afraid to question the old ways of how you're building software, 'cause really the future is coming faster than any of us know.

75. 16:26

    It's probably already here, and the cool thing is, we get to build it together. Thank you. [upbeat music]