The Missing Primitive for Agent Swarms

Talk transcript

1. 0:00

   [upbeat music] Cool. All right.

2. 0:15

   Hello, everyone. We're getting to the back end of the conference. I don't know if that's a good thing or not for you, uh, to start the weekend or maybe sad that the conference is over.

3. 0:21

   So gonna be talking about the, the missing primitive for agent swarms. So talking a bit about subagents and swarms within the context of, of coding agents and the infrastructure underneath them.

4. 0:32

   So just a quick introduction. So my name is Lou. I'm the field CTO at a company called Ona. Uh, previous life, I was principal engineer and sort of platform engineer, joined Ona doing product management, and now, um, I work a little bit more with our sort of customers on the field side.

5. 0:48

   So in my world at least, everyone is trying to build a form of a software factory. Uh, I've seen a few different talks at this conference of similar sort of ideas, trying to take coding agents and then apply them across the software development life cycle.

6. 1:00

   Uh, I'm, I'm wondering actually if this statement is entirely correct, 'cause I've definitely chatted to a lot of people, uh, over the course of this week that are not yet at this point of thinking about this.

7. 1:08

   Uh, but it's been very much, uh, occupying my head space for the last few months as well. So I added a definition in this slide as well, just to quickly define what I believe to be as a software factory, which is sort of the commitment to incrementally moving the human out of the loop, uh, within the SDLC,

8. 1:23

   uh, such that the human is not proactively, uh, interacting with a computer. I say that because I've seen a few talks and some people talking about software factory with these, like, parallel agents, like one individual IC running lots of coding agents at the same time.

9. 1:35

   It's not my personal definition. My personal definition is that you're slowly bringing the human out of it, and then work is flowing from sort of development into production, theoretically in an automated fashion.

10. 1:45

    But we're extremely early in terms of, uh, where we're at with software factories.

11. 1:50

    So obviously, there's a bunch of funky little visualizations that I've been showing, and a lot of these are actually taken from this website, backgroundagents.com, that I created, um, which seem to resonate quite a lot.

12. 2:01

    Um, so please do have a look at it. It encapsulates some of these ideas as well. But we have, like, different patterns really for running these agents at scale, uh, coding agents at scale.

13. 2:10

    So one of them really is sort of this swarm pattern, which, for my definition, is starting off with an, an individual intent, firing that out to a number of different agen-agents, and then funneling that back in, let's say, to an individual PR or task.

14. 2:22

    And that's almost like your typical kind of subagent process that we've, we've seen quite a few times. Uh, fleets is something... You, you can't see this very well in the room, unfortunately.

15. 2:31

    But on the top right, the sort of fleets is where you're fanning out agents across, let's say, a number of different repositories inside of an organization, uh, which is a capability we've had in Ona for quite some time now, and, uh, you see it a little bit popping up here or there, but not so much, but I

16. 2:45

    think it will be something that organizations will, will start to take advantage of, uh, a lot more in the future. And then at the bottom, you've kind of got events.

17. 2:53

    So if you think about how do we take the human out of the loop and build this sort of software factory, you need to know how and when are you going to trigger those agents?

18. 3:00

    When do they come online? And a lot of this already exists with existing sort of webhook infrastructure and things like that. PR is raised. Linear ticku-ticket is created, et cetera.

19. 3:11

    One thing that's been useful for us over the last couple of months is a lot of these large companies have also come forward and shared some of their implementations of this infrastructure.

20. 3:19

    So, uh, I'll, I'll mention a few notable ones. Uh, but Stripe has one. The, the- they're what they call Minions, which is built on top of their existing infrastructure where they've then plugged in these coding agents, and they're able to then drive thousands of pull requests inside of, inside of Stripe.

21. 3:34

    Another very noteworthy one, and Ramp has been so loud on social media these last couple of weeks, it's been quite insane. Uh, but they also built one that they internally call Inspect, which again is their sort of infrastructure for, for running these background agents.

22. 3:49

    And it's something that we've been doing now for quite some time. Um, so Ona as a platform has been infrastructure for development environments, uh, for about six years. But over the last year or two, obviously integrating further with the agent side of that as well.

23. 4:02

    What Ona effectively does is allow you to spin up a n- any number of different development environments, but one of the additional features that we have that I mentioned is this fleet feature.

24. 4:10

    So you can automate on schedules or triggers agents that spin up, uh, to resolve, uh, issues across a number of different repositories. Like, use cases for this is things like CVE remediation or bumping test coverage or enforcing something at scale.

25. 4:25

    As it stands today with current LLMs, that seems to be often tasks that are somehow simple, but the hard part is that you're doing this across a number of different teams, thousands of teams or thousands of repositories.

26. 4:38

    Which is the, what, what basically I'm showing you here. So it, it works kind of like a workflow kind of creation. You're adding prompts, scripts, and things like that, and then using that to, to drive this change across your organization.

27. 4:49

    I did want to give a notable shout-out also to the Harness Engineering blog that OpenAI and Ryan created, because this encapsulates much of this mindset of trying to then take and encode as much of your process into your repository, into your context files, your AGENTS.md, in order to build effectively that software factory.

28. 5:07

    Over the last couple of days, I had a few questions with people talking about harness engineering and what is it. Uh, for me, it's really another extension on context engineering whereby everything in your repository, from skills to AGENTS.md to unit tests, everything that you could possibly use to give feedback to your agent, uh, is for me harness

29. 5:23

    engineering. So I include this little visualization in the top corner 'cause as well for me, harness engineering is very much about doing things, letting the agent run through, figuring out where the agents gets lost, and then encoding that knowledge back into, uh, your repository or context, again, to try and get the agent flowing through the software factory

30. 5:40

    as much as it can. So if we take a step back and then think about from an infrastructure level what you need effectively to build this form of sort of software factory, the first piece of that puzzle is a runtime, or you need somewhere for the agent to run.

31. 5:53

    And I believe this mostly is, is pretty much a solved problem now. You then need a way to orchestrate these, so you need to run them at scale. So you need to, a way to run these agents, you know, scale up, scale down horizontally.

32. 6:04

    You need some way to trigger them, but for me, one of the biggest difficulties if you try and build this today is effectively agent coordination. So how do you get the agents to sort of interact with each other, pick up tasks from each other?

33. 6:15

    How do they collaborate? For runtimes, lots of, you know, different approaches here, but you can run agents as separate threads. You can isolate them more in worktrees. You can then go one step of abstraction further and put them in containers and VMs or microVMs, or what Ona does, basically, we really call these dev environments.

34. 6:34

    So, like, the sandbox conversation has made this very blurry, uh, but we at least believe that for running sort of proper development tasks, it has to be inside of a virtual machine.

35. 6:43

    The reason for that is, uh, for the isolation from a security standpoint, a container is not a bulletproof, um, isolation boundary. So if you have an agent running in there and you want to secure it, you're-- there's challenges for a container.

36. 6:56

    They're also bursty. If you run them on Kubernetes or in pods, you have noisy neighbor problems, you're gonna have compute contention across different containers. Uh, and only with having sort of the full isolation of a VM

37. 7:09

    will you be able to effectively do this properly. Let me run back through my presentation, sorry.

38. 7:17

    Cool. So let me actually just quickly... It's a good point, actually, at this point. I will show you a quick demo of how this actually looks inside of the Ona interface, because a lot of this is a little bit theoretical.

39. 7:25

    Um, I did a quick recording of this yesterday in my hotel room, just in case the Wi-Fi was terrible in here. But, um, let me just run you through this.

40. 7:32

    So in here, you see the Ona interface. On the left-hand side, you have all the different tasks that I have running. So I, I kicked off two different tasks here.

41. 7:40

    The first one I, I asked Ona to implement me Symfony. So Symfony has, uh, a spec in the repository that talks about how to implement Symfony. It's a very detailed spec, so I wanted to use it as an example.

42. 7:52

    I gave one of the agent and asked it to spin this up using process-based agents, uh, so subagents running within the environment itself. So take the VM, run the agent inside, and that agent will spin up subagents within that VM.

43. 8:03

    The other one I asked it to is effectively to run me a fleet with a number of different VMs. So the agent is actually then empowered to create other VMs inside of the platform, and it can spin up technically infinite of these.

44. 8:14

    So wherever you're running this, you're only really inhibited by as much as you're willing to pay and as much as your cloud provider can scale to. So if I run this through, what we see, and I might have to skip through here a little bit, is we see...

45. 8:28

    I jumped too fast. [laughs] Um, this bottom agent, the VM one, then it will spawn these three different subagents. So it spins up the different VMs, which we see coming in on the left-hand side.

46. 8:40

    The, the parent is the, the controlling agent, and the subagents obviously then are given small bits of context, individual tasks to complete, and then will do message passing back to that parent agent to control and govern the overall task itself.

47. 8:54

    One challenge for sure we have is the, how do we build the UX for this? Like, as you build more and more complicated tasks, how do you think about sort of managing and controlling these subagents, and how do you think about the UX on top of them?

48. 9:05

    So as this progresses, you'll see also on the left-hand side, eventually you start to see, um, as the agents come online, they start their environments, and they start to work through tasks.

49. 9:14

    Both of these have, for some reason, seven sort of sub items that they're working through, so you can see those. When that gets to the end, it's then obviously going to terminate those VMs.

50. 9:21

    That's all going to collapse down, and your task is, is complete.

51. 9:26

    The second form of UX for this that we have with the, the subagents is this one here with the process level, uh, which I'll pause so that it's not jumping around.

52. 9:34

    Um, when you launch a process-level subagent, it happens all within the single agent window. So you actually see at the bottom here a, a stack of a number of different subagents that are starting.

53. 9:44

    And then when you click on those, you can then open up a new chat window, which is almost your new context, and use that. So you've got this, like, two different forms of this, one entirely isolated VMs to scale out these swarms, and another one at the sort of process level where you can run it within the

54. 9:59

    individual VM itself. So lots of stuff going on there, but I wanted to show you conceptually, people say, "What does this swarm look like in, in, in reality? How does this actually look?"

55. 10:09

    And this is how it looks within, within Ona.

56. 10:16

    So give me one second to fly back through.

57. 10:20

    So coming back to the software factory idea. So how do I know, uh, about some of the challenges about software factories? And that is because I've also tried to build one.

58. 10:27

    Um, obviously, I implement many of these ideas in, in our own projects, but I also, similarly to what OpenAI did building out their sort of Symfony project and their harness engineering blog, is, okay, what-- can we build a project without touching any lines of code, and can we automate as much of this process as possible such that

59. 10:43

    the agent can actually then develop everything as autonomously and self-driven as possible? And it turns out, yes, the technology is there today, but there are some different challenges that you have.

60. 10:53

    What's missing? One thing that we-- you find out is that the SDLC is not this. It, you know, this is the conceptual SDLC that we present and talk about with each other, the very coarse-grained five steps of the SDLC.

61. 11:07

    Agents don't respect this, and they don't understand... I mean, these boxes contain a ton of complexity. So if we take something like plan or a plan stage, actually within that, our SDLC has a ton of different sort of micro-steps almost to it.

62. 11:19

    And if we're wanting them to train agents to then step through the SDLC, we need to find ways to actually break down the SDLC into these, some of these micro-steps.

63. 11:27

    And that happens all the way through the SDLC. So if you want to build some form of software factory, we then now need to start figuring out how to solve these sort of micro-steps.

64. 11:35

    How do we get agents to sufficiently follow those steps and do them in sort of deterministic ways as well?

65. 11:43

    And the hard part of this is context. Probably not a, you know, huge innovation to everyone in this room. Context windows are some of the hardest parts about working with LLMs because, as the last speaker said here as well, context rot.

66. 11:55

    Context, you know, once the context window becomes consumed, the, the agent starts to lose track of where it's going and things like that. It gets less effective. They also skip steps typically.

67. 12:05

    Um, you know, they want to please us. They're quite sycophantic, so they will might, you know, ask them to write some tests, and then they're going to skip some tests, um, in order to complete the task.

68. 12:16

    So a lot of the tools that we have today as well for coordinating these agents as you start to spin them up, once you've got this infrastructure, you have the runtimes, you have the orchestration.

69. 12:24

    The missing piece then you also need is coordination. So if you use something like GitHub, GitHub is not a coordination layer for agents. It gets incredibly overwhelming. You know, you can have your agents raise a pull request, you can review it, you can then solve the merge conflict, you can fix your CI build, et cetera.

70. 12:40

    But this gets incredibly noisy for you as a human to make sense of where you should step in to intervene with that agent itself. And GitHub is a, a poor solution for this.

71. 12:50

    Uh, Symphony is built on top of Linear, but suffers the same problem, that we're re- reusing existing sort of human tools in very weird ways for agents.

72. 12:59

    But I think we are now at the cusp of now effectively solving this, and I think it might be solved in a few different ways. One is, um, through sort of state machines, you know, by building out workflows and, uh, effectively state machines that we've had for a very long time, and then building these as our versions

73. 13:13

    of our SDLC. I do believe, to a degree, some of the ideas from Durable Executions comes into this as well. You know, lots of companies have pioneered this over time to be able to run a process in a, in a durable fashion.

74. 13:24

    Um, but we do need to solve also the gates and compliance, uh, section of this. And I do believe actually there is definitely a, a gap right now also for packaging this in some form of like CLI construct whereby you can run this locally in a sort of development environment, but also then remotely in, you know, in

75. 13:39

    a CI or some other fashion like this.

76. 13:44

    But that's effectively it. If anyone wants to, to chat and go a little bit deeper, I think we've got a couple of minutes. We can also take some, some questions if we need to.

77. 13:51

    Um, yes. Basically, my definition of software factory is moving this human on the loop so that they're not necessarily driving each of these d- individual changes. Context and context management is by far the hardest part of building this software factory.

78. 14:06

    Out of these primitives, I do believe we've effectively solved the runtime. There are many options for this now, sandboxes and, and containers and other solutions. The orchestration is effectively solved.

79. 14:15

    The triggers are solved. Um, but the thing that's missing for me is coordination. Uh, also, one of our folks here is here from, from our security team. Uh, and security is another piece of the, the puzzle that we really need to solve as well to, to drive more automation.

80. 14:28

    But it's this coordination layer that I think is, is largely what's missing.

81. 14:33

    Obviously, a lot of this is a little bit high level. On, on the 6th of May, we'll run a, a virtual summit to go into these topics, uh, about software, um, software factories and background agents.

82. 14:41

    So if anyone here would like to attend, uh, you can do. It's on backgroundagents.com, and there's... We have a CFP up for it as well. If anyone would like to speak, um, that would also be great.

83. 14:50

    There's not many people in the world doing this, so if you're at least experimenting with it, uh, it would be great to also to talk to you as well.

84. 14:57

    The other thing is a few people that I've been speaking to at the conference, especially about software factory, complain that obviously a lot of these talks are a little bit theoretical, which is true.

85. 15:05

    Uh, so some of you might want to be really getting into the weeds of this. So actually, next week, um, with Zach from Ona, we're actually just gonna build one of these in public, starting from scratch, build a software factory.

86. 15:15

    We'll, we'll do this over the course of two weeks in public just to show what that looks like with today's technology, um, just to really, like, show all the ins and outs of the workflows if anyone wants to see what that actually looks like as well.

87. 15:27

    But there you go. Thank you very much. Um, we do have two and a half minutes if anyone has questions, but if not, I'll also, um, be outside and downstairs.

88. 15:34

    Yes.

89. 15:34

    Thank you for the, the talk. Super helpful. Um, could you maybe, uh, go back to the slide where you were talking about the problems with the coordination layer?

90. 15:42

    Yep. Um-

91. 15:44

    With, um, memory, and then also if you could just elaborate a bit more on what you propose as a solution in more concrete, um, that would be pretty interesting to hear your thoughts.

92. 15:53

    Yeah, sure. So I have been... Uh, so I need to repeat the question actually for the, for the stream. So the question was, um, problems with the coordination layer and sort of more concreteness about the solution, what could that potentially look like?

93. 16:04

    So I have, um, we have a number of different sort of prototypes for this internally, and I've also been trying to think about how we... what we do with this.

94. 16:12

    So I think one form factor that, that works is the CLI. Um, so I've seen some other solutions, graph-based ones. There are some open source things right now where defining the workflow effectively as a, as a graph, um, you know, like kind of like what you drive sort of mermaid diagrams out of, like n8n type of, of

95. 16:30

    workflows, um, where you can define sort of these prompts in, in that form. Um, I think that ultimately needs to be packaged in some form of CLI, though. Um, what I mean by that is I think the...

96. 16:39

    If you have a local running agent, Cloud Code, whatever, any local running CLI, I think that now needs to have something that integrates with its tools, so it can invoke this to say, "Hey, have I achieved this part of my SDLC, and can I now proceed to the next part of it as a, a CLI?"

97. 16:53

    If you want as well, I can-- I have a full spec for this I can show you, um, after the talk as well. We can go a bit deeper into it.

98. 16:59

    Um, but I'm seeing this solved in a number of different ways. People are solving this in, in a few different ways. There's the n8n sort of

99. 17:07

    workflow type of diagram way. The CLI sort of gateway of- is the prototype that I have as well. Um, and then there's an, a range of different sort of hacky interim solutions.

100. 17:17

     Uh, there's one even the OpenClaw folks have as well, which the name is eluding me, but I will... I think ACPX, which they've built on top of ACP, um, which builds out some of this workflow.

101. 17:27

     There's a GitHub one, Fabro, some folks are playing around with, but it's very nascent right now. Um,

102. 17:35

     yeah. Any other questions?

103. 17:38

     Are you using, uh, are you using some protocol for the CLI? Because, you know, ACP, there's ACP, there's A2A, there's a bunch of, uh... Yeah, I, I don't know if people have agreed on the, on the standards. [laughs]

104. 17:48

     Cu- currently not. So the inter- the implementation that I have, I'm trying to understand whether or not to release it as an implementation or as a standard, to be honest, because I largely kind of don't really care about the implementation.

105. 17:57

     I almost care about the, the standard for it, so we can collaborate on the standard. Um, but it's not built on top of ACP, or at least not yet, but it might be.

106. 18:05

     Um-

107. 18:06

     And it's not A2A either?

108. 18:07

     Not A2A either, no. It's solving, I feel, like a slightly different problem space, so...

109. 18:16

     Cool. All right. Thank you very much. Enjoy the conference, everyone. Thank you. [audience applauding] [upbeat music]