Automating Large-Scale Refactors with Parallel Agents

Talk transcript

1. 0:00

   [upbeat music] All right, thank you all for, for joining for Automating Massive Refactors with,

2. 0:25

   uh, with Parallel Agents. Um, super excited to talk to you all today about, uh, you know, what we're doing with OpenHands to really automate large scale chunks of software engineering work.

3. 0:36

   Lots of, uh, lots of toil related to tech debt, code maintenance, code modernization. Uh, these are tasks that are super automatable. Uh, you can throw a-- at them, but they tend to be way too big for like, you know, a single just one shot.

4. 0:48

   Uh, so it involves a lot of what, what we call agent orchestration. Uh, we're gonna talk a little bit about how we do that, uh, with OpenHands and also just more generically.

5. 0:58

   Uh, a little bit about me. Um, my name is Robert Brennan. I'm the co-founder and CEO at OpenHands. Uh, my background is in dev tooling. I've been working in open source dev tools for over a decade now.

6. 1:10

   I've also been working in natural language processing for about the same amount of time. Um, uh, I've been really excited over the last few years to see those two fields suddenly converge as they'll obviously are really good at writing code.

7. 1:22

   Uh, but I'm super excited to be, to be working in this space. Uh, then OpenHands is an MIT licensed coding agent. Uh, OpenHands started at OpenDevin about a year and a half ago when Devin first launched their, uh, demo video of a fully autonomous software engineering agent.

8. 1:37

   Uh, my co-founders and I saw that, got super excited about, you know, what was possible, what the future of software engineering might look like, uh, but realized that that shouldn't happen in a black box, right?

9. 1:45

   If our docs are gonna change, we want that change to be driven by the software development community. We wanna have a say in that change. Um, and so we started OpenHands, uh, then OpenDevin, uh, as a way to give the community a way to help drive what the future of software engineering might look like in an AI-powered

10. 2:01

    world. Uh, so hopefully not, uh, controversial for me to say that software development is changing. Um, I know my workflow has changed a great deal, uh, in the last year and a half.

11. 2:13

    Um, uh, I would say now, like, you know, pretty much every line of code that I write goes through an agent. Uh, rather than me opening up my IDE and typing out lines of code, I'm now, uh, asking an agent to do the work for me.

12. 2:25

    I'm still, you know, doing a lot of critical thinking. You know, a lot of the, the mentality of the job hasn't changed, but what the actual work looks like has changed quite a bit.

13. 2:34

    Uh, but what I wanna convince you all of is that it's still changing. We're still just in the first innings of this change. We still haven't realized all the, um, all the impact that large language models are-- have already brought to the job and, uh, are gonna continue to bring to the job as they improve.

14. 2:49

    Uh, I would say even if you froze large language models today, and they didn't get any better, you would still see the job of software engineering changing very drastically over the next two to three years as we figure out ways to operationalize the technology.

15. 3:02

    Uh, I think there's still a lot of, uh, sort of psychological and organizational hurdles to adopting, uh, large language models within software engineering. Um, and we're seeing, uh, a lot of those hurdles disappear as time goes on. [clears throat]

16. 3:20

    Uh, brief history of kinda how we got here. Um, everything started, I would say, with what I call context-unaware code snippets. Um, some of the first large language models, it turned out, were very good at writing chunks of code, especially things that they'd seen over and over again.

17. 3:33

    So you could ask it to write bubble sort. Uh, you could ask it for, you know, small algorithms. You know, how to, how to access a SQL database, things like that.

18. 3:41

    Uh, it was able to generate little bits of code. It was able to, to, you know, seem to understand the logic a bit. But this was totally context-unaware, right?

19. 3:48

    It was just dropping code into a chat window that you had asked for. It had no idea what project you were working on, what the context was.

20. 3:56

    Shortly thereafter, we got these context-aware code generation. Uh, so like GitHub Copilot as autocomplete, um, was probably, like, the, the best example here, right? Um, suddenly you had something within your IDE.

21. 4:08

    It could see, you know, where you're typing, you know, what the, what the code you're working on is. Uh, and it could generate code that was specific to your code base that referenced, you know, local variable names, that referenced, you know, local table names in your database.

22. 4:20

    Uh, huge, huge improvement for, um, uh, you know, our productivity. So instead of copy-pasting back and forth between the ChatGPT window and your IDE, now all of a sudden you can see the little robot get eyes.

23. 4:31

    It can see inside your code base, and it can actually generate relevant code for your, for your, uh, code base.

24. 4:37

    And then I think the, the giant leap happened in early twenty twenty-four, um, with the launch of Devin, and then, uh, the next day the launch of OpenDevin, now OpenHands.

25. 4:47

    Uh, this is where we first started to see, uh, autonomous coding agents. So this is when AI started not just writing code, but it could run the code that it wrote, and it could Google an error message that came out, find a Stack Overflow article, apply that to the code, add some debug statements into the code, and

26. 5:01

    run it, and see what happens. Uh, basically automating the entire inner loop of development. Um, this was, this was a huge, uh, step function forward. Um, you can see the little, the little robot gets arms in this picture.

27. 5:14

    Uh, this was a, this was a huge jump, at least, at least in my own productivity, um, being able to, like, just write a couple sentences of English, give it to an agent, and let it churn through the task until it's got something that's actually working, running tests are passing.

28. 5:29

    And then now what we're seeing is, uh, parallel agents, what we're calling agent orchestration. Uh, folks are figuring out how to get multiple agents working, uh, in parallel, sometimes talking to each other, sometimes spinning up new agents under the hood.

29. 5:42

    Um, you have agents creating agents. Um, this is, uh, I would say kind of bleeding edge of what's possible. Um, people are just starting to experiment with this, are just starting to see success with this at scale.

30. 5:54

    But there are some, uh, some really good tasks that are, um, uh, very amenable to this sort of workflow. Uh, and it has the potential to really, uh, automate away a huge mountain of tech debt that sits under, you know, every contemporary software company.

31. 6:13

    A little bit about kind of like the, the market landscape here. Uh, again, you can kind of see that same evolution from left to right, where we really started with, uh, you know, plug-ins, like GitHub Copilot inside of our existing IDEs, then we got these like AI, AI-empowered IDEs, IDEs with like AI tacked onto them.

32. 6:29

    Um, I would say your, your median developer is kind of adopting local agents now. They may be running Cloud Code locally for, uh, one or two things, uh, maybe some ad hoc tasks.

33. 6:39

    Uh, your early adopters though, are starting to look at cloud-based agents, agents that get their own sandbox running in the cloud. This allows, uh, those early adopters to run as many agents as they want in parallel.

34. 6:52

    Uh, it allows them to run those agents much more autonomously than if they were running on their local laptop. Right? If it's running on your local laptop, there's nothing stopping the agent from doing RMRF/, trying to delete everything in your home directory, whatever it might do, installing some weird software.

35. 7:05

    Whereas if it's got its own like containerized environment somewhere in the cloud, you can run a little bit more safely knowing that, you know, the worst it can do is ruin its own environment.

36. 7:13

    Uh, and, um, uh, you don't have to like sit there babysitting it, hitting the Y key every time it wants to run a command. Uh, so those cloud-based environments, much more scalable, uh, a bit more secure.

37. 7:24

    Um, and then, uh, I would say at the far right here, what we're really just seeing the top like one percent of early adopters, uh, start to experiment with is orchestration.

38. 7:33

    This idea that you not only have these agents running in the cloud, but you have them talking to each other. Um, you're coordinating those agents, you know, on a larger task.

39. 7:42

    Uh, maybe those agents are spinning up sub-agents within the cloud that have their own sandbox environments. Um, some really cool stuff happening there. Uh, I would say, you know, with OpenHands, we, we generally started with cloud agents.

40. 7:54

    Uh, we've leaned back a little bit and built a local CLI similar to Cloud Code in order to meet developers where they are today. You know, these, these types of experiences are much more comfortable for developers.

41. 8:04

    Uh, you know, we've been using autocomplete for decades, just got a million times better with GitHub Copilot. Um, I would say these experiences on the right side are very foreign to developers.

42. 8:12

    They feel very strange to like give off a task to an agent or a fleet of agents, uh, and let that do the work for you. It feels kind of like, uh, for me at least, uh, the jump that I made when I went from being an IC to being a manager, uh, is, is what it feels

43. 8:26

    like going from writing code myself to giving that code to agents. Uh, so a very, very different way of working. I think one that developers have been very slow to adopt.

44. 8:33

    Um, but again, the, the top one percent or so of engineers that we've seen adopt the stuff on the right side of this, uh, landscape, uh, they've been able to get, you know, massive, massive lifts in productivity and tackle huge backlogs of tech debt that their teams just weren't getting to.

45. 8:50

    Uh, some examples of where you would wanna use orchestration rather than a single agent. Uh, typically these are tasks that are going to be very repeatable and very automatable.

46. 8:59

    Uh, so some examples are things like your basic code maintenance tasks, right? Every code base has to, uh, you know, there's, there's a certain amount of work to do to just keep the lights on, right?

47. 9:09

    To keep dependencies up to date, to, uh, make sure that any vulnerabilities get solved. Uh, we have one client, for instance, that is using OpenHands to, uh, remediate CVEs throughout their entire code base.

48. 9:21

    They have tens of thousands of developers, thousands and thousands of repositories. Um, and basically every time a new vulnerability gets announced in an open source project, they have to go through their entire code base, figure out which of their repos are vulnerable, uh, submit a pull request to that code base to, uh, actually, uh, you know, resolve

49. 9:37

    the CVE, update whatever dependency, fix breaking API changes. Uh, and they've seen a thirty X improvement on time to resolution for these CVEs by doing, uh, orchestration at scale.

50. 9:48

    Uh, they basically have a setup now where every time a new CVE gets announced, new vulnerability gets in, uh, they kick off an OpenHand session to scan a repo for that vulnerability, uh, make any code changes that are necessary, and open up a pull request, and all the downstream team has to do is click merge, uh, the

51. 10:06

    changes. Um, you can also do this for like automated documentation and release notes. Um, there's a bunch of modernization challenges that, uh, companies face. Um,

52. 10:18

    for instance, uh, you might want to add type annotations to your Python code base if you're working in Python 3. Um, you might want to split your Java, you know, monolith into microservices.

53. 10:29

    Uh, these are the sorts of tasks that are still gonna take a lot of, um, thought for an engineer. You know, you can't just like, again, one-shot it with Cloud Code and say like, uh, you know, "Refactor my monolith as microservices."

54. 10:41

    Uh, but it is still very rote work, right? You're still just kind of like copying and pasting a lot of code around. So if you thoughtfully orchestrate agents together, they can do this.

55. 10:49

    Um, a lot of migration stuff, so migrating from like old versions of Java to new versions of Java. We're working with, uh, one client to migrate a bunch of Spark 2, 2 jobs to Spark 3.

56. 10:58

    Um, we've, uh, used, uh, OpenHands to migrate our entire front end from React, uh, from Redux to Sustained. Uh, so you can do these very large migrations. Again, lots of very rote work.

57. 11:09

    Still takes a lot of, um, thinking from a human about how they're gonna orchestrate these agents. Uh, but then a lot of tech debt. Uh, detecting unused code, getting rid of that.

58. 11:19

    Um, you know, uh, we, we have one client who's using our SDK to basically scan their data.logs every time there's a new error pattern, go into the code base, and, uh, add error handling, fix whatever problem is, uh, is cropping up.

59. 11:32

    Uh, so lots of things that, you know, are a little too big for a single agent to just one-shot, um, but are super automatable, are good tasks to handle with agents as long as you're thoughtful about orchestrating them.

60. 11:45

    A bit about why these aren't one-shot-able tasks. Uh, some of them are technological problems, some of them are more like human psychological problems. On the technology side, you have a limited amount of context, uh, that you can give to the agent, so extremely long-running tasks or tasks that span like a very large code base.

61. 12:01

    Usually, you don't really have enough there. You might have to, uh, compact that context window to the point where the agent might get lost. Uh, we've all seen the laziness problem.

62. 12:10

    Uh, I've tried to one-shot some of these types of tasks, and the agent will say, "Okay, I've migrated three of your hundred services. Now you need to hire a team of six people to do the rest."

63. 12:19

    Um, uh, the agents often lack domain knowledge within your code base, right? They don't have the same intuition that you do for the problem. Um- Uh, and errors compound.

64. 12:28

    When you go on these really long trajectories with an agent, uh, a tiny error in the beginning is going to, uh, you know, compound over time. The agent's gonna basically repeat that error over and over and over again for every single step that it takes in its task.

65. 12:41

    Uh, and then on the human side, uh, you know, we do have this intuition for the problem that we can't convey. You know, say you want to break your monolith into microservices, you probably have a mental model of how that's going to work.

66. 12:51

    Uh, if you just tell the agent, "Break the monolith into microservices," it's just gonna take a shot in the dark based on patterns it's seen in the past without any real understanding of your code base.

67. 13:00

    Um, we have some difficulty decomposing tasks for agents and understanding, like, what an agent can actually get done, uh, in one shot. Um, uh, we also, like you, you, uh, do need this intermediate review, intermediate check-in from the human as the agent's doing its work.

68. 13:16

    We'll talk a little bit about what that loop looks like later. Um, but it's, again, not something you can just, like, tell an agent to do and expect the final result to come in.

69. 13:23

    You have to kind of approve things as the agent goes along.

70. 13:27

    Uh, and then not having a clear definition of done. I think, uh, if you don't really know what finished looks like for this project, it's hard to tell the agent.

71. 13:37

    Uh, on these types of orchestration tasks, I want to make it super clear that we don't expect every developer to be doing agent orchestration. Um, we think most developers are going to use a single agent locally, uh, for, you know, sort of ad hoc tasks that are common for engineers.

72. 13:50

    Building new features, uh, fixing a bug, things like that. I think running bot code locally, uh, in a familiar environment alongside an IDE is probably going to be a common workflow at least for the next couple of years.

73. 14:01

    Uh, what we're seeing is that a small percentage of engineers who are early adopters of agents, who are really excited about agents, are finding ways to orchestrate agents to tac- tackle, like, huge mountains of tech debt at scale, and get a much bigger lift in productivity for that smaller select set of tasks, right?

74. 14:18

    You're not gonna see a three thousand percent lift in productivity for all software engineering. You're probably gonna get more of that, you know, twenty percent lift that everybody's been reporting.

75. 14:26

    Uh, but for some select tasks like CVE remediation or code-based modernization, you can get a massive, massive lift. You can do, you know, engineer years of work in a, in a couple of weeks.

76. 14:39

    Uh, I wanna talk a little bit about what these workflows look like in practice. So this loop probably looks pretty familiar if you're used to working with local agents.

77. 14:47

    Uh, this is very typical loop. It looks a lot like the inner loop of development for, you know, non-AI coding as well. But basically, you know, you give the agent some prompts.

78. 14:56

    Uh, it does some work in the background. Maybe you babysit it and watch, you know, everything it's doing, and hit the Y key every time it wants to run a command.

79. 15:03

    Uh, then the agent finishes. You look at the output. Uh, you see if the tests are passing. You see if this actually satisfies, uh, what you've asked for. And then maybe you prompt the agent again to get it to get a little closer to the answer, or maybe you're satisfied with the result.

80. 15:15

    You, uh, you know, you commit the results and, and push.

81. 15:20

    For bigger orchestrated tasks, this becomes a little bit more complicated. Uh, basically, what you need to do is, uh, you or maybe, uh, hand in hand with Cod, you want to decompose your task into a series of tasks that can be executed individually by agents.

82. 15:37

    Uh, then you'll send off an agent for each one of those individual tasks, and you'll do one of those, one of those agents for each of the individual tasks.

83. 15:44

    And then finally, at the end, uh, you, maybe with the help of an agent, are gonna need to collate all the output together from all those individual agents into a single change, uh, and merge that into your code base.

84. 15:58

    Very importantly, there's still a lot of human in the loop here. Uh, you need to review not just the final output of the collated result, but, uh, the intermediate outputs for each agent.

85. 16:08

    Um, I like to tell folks the goal is not to automate this process a hundred percent. It's something like ninety percent automation. Uh, that's still, you know, a quarter of magnitude productivity lift.

86. 16:17

    Um, I think this is, this is really tricky to get right. This is where a lot of, like, thought comes into the process of, like, how am I gonna break the task down so that I can verify each individual step, uh, and so that, uh, I can actually, uh, automate this whole process without just ending up with

87. 16:33

    a high coded mess. Uh, this is a typical Git workflow that I like to use for tasks like this. Uh, typically, we'll start a new branch on our repository.

88. 16:45

    Uh, we might add some handles and context to that branch using like an agent's ID or an OpenHands view of the concept of a micro agent. Uh, but I-- Just some markdown explaining, you know, here's what we're doing here, uh, just so the agent knows, okay, we're migrating from Redux to Zustand, or, uh, we're gonna migrate these

89. 17:01

    Spark 2 jobs to Spark 3. Uh, you might want to put some kind of scaffolding in place. Uh, I'll talk a little bit more about examples of, of, uh, scaffolding later.

90. 17:10

    Uh, then you're gonna create a bunch of agents based on that, on that first branch. Uh, the idea is that they're going to be submitting their work into that branch, and it's basically gonna accumulate our work as we go along.

91. 17:21

    And then eventually, once we get to the end, we can rip out our scaffolding and merge that branch into main. Uh, now for, uh, if you're, you're kind of getting started with this, I would suggest limiting yourself to about three to five concurrent agents.

92. 17:33

    Uh, I find more than that, your brain starts to break. Uh, but for folks that have really adopted orchestration at scale, uh, we see them running hundreds, even thousands of agents concurrently.

93. 17:44

    Usually, a human is not, uh, in the loop for, you know, one human is not on the hook to review every single one, but maybe those agents are sending out form requests to individual teams, things like that.

94. 17:53

    Uh, so you can scale up very aggressively once you start to get a feel for how all this works and you feel like you have a very good way of getting that human input into the loop.

95. 18:04

    I'm gonna kick it off to, uh, my coworker Calvin here. He's gonna talk about, uh, a very, very large scale migration, uh, basically, uh, eliminating code smells from the OpenHands database that he did using our, uh, Refactor SDK, which we'll show up here.

96. 18:21

    Um, Calvin. OpenHands excels at solving atomic open tasks. Give it a focused problem, something like, "Fix my failing CI," and "Add and debug these endpoints," and it delivers. But like all agents, it can stumble when the scope grows too large.

97. 18:38

    Let's say I want to refactor an entire code base, maybe enforce certain type checking, update a major dependency, or even migrate from one framework to another.

98. 18:48

    These are not atomic tasks. They're sprawling, interconnected changes that can touch hundreds of files.

99. 18:54

    To tackle problems at this scale, we're using the OpenHands Agent SDK to build tools designed specifically to orchestrate collaboration between humans and multiple agents.

100. 19:06

     As an example, let's work to eliminate code smells from the OpenHands repository. Here's the repository structure. Just the core agent definition has about three hundred and eighty files, uh, spanning sixty thousand lines of code.

101. 19:20

     Says a lot about the volume of the code, but not much about the structure. So let's use our new tools to visualize the dependency graph of this chunk of the repository.

102. 19:30

     Here, each node represents a file. The edges show dependencies, who imports who. And as we keep zooming out, it becomes clear this tangled web is why refactoring at scale is hard.

103. 19:43

     To make this manageable, we need to break this graph up into human-sized chunks. Think PR-sized batches that an agent can handle and a human can understand.

104. 19:52

     There are many ways to batch based on what's important to you. Graph theory algorithms give strong guarantees about the structure of edges in between induced batches, but for our purposes, we can simply use the existing directory structure to make sure that semantically related files appear inside the same batch.

105. 20:08

     Navigating back to the dependency graph, we can see that the colors of the nodes are no longer randomly distributed. Instead, they correspond to the batch that each of those associated files exists in.

106. 20:18

     Zooming out and zooming back in, we easily find a cluster of adjacent nodes that are all the same color, which indicates that an agent is going to process all of those files simultaneously.

107. 20:28

     Of course, this graph is still large and incredibly tangled. To construct a simpler view, we'll build a new graph where nodes are batches and the edges between those nodes are dependencies that are inherited from the files within each of those batches.

108. 20:42

     This view is much simpler. We can see the entire structure on our screen at the same time.

109. 20:48

     But this is something we want to do after batches. Using a graph, we can identify batches that have no dependencies and then inspect the files that go into them.

110. 20:57

     This batch, for example, has sixteen files. Looks like it's an init file. It's probably empty. Let's check.

111. 21:04

     Now, this is a tool intended for human and AI collaboration, so once we know that this file is empty, we might determine that it's better to move it elsewhere.

112. 21:12

     Or maybe we're okay keeping it inside this batch, and all that we want to do is add a note to ourselves or review the batch so we know the contents.

113. 21:20

     Of course, when refactoring code, it's important to consider the complexity of the code that you're working with. This batch is trivial. Let's find one that's a little bit more complex.

114. 21:30

     Here's a batch that has four files that also reasonably chunky, and the complexity measures reflect this. These are useful to indicate to a human that we should be more careful when manipulating this batch and, for example, the first file we examine.

115. 21:44

     So how are we actually going to get rid of code smells? Well, it's a two-step process indicated by these buttons right here.

116. 21:51

     Before we can fix the code by removing code smells, we need to identify what's wrong in the first place. Enter the verifier. There are several different ways of defining the verifier based on what you care about.

117. 22:03

     You can set it to be programmatic, so it calls a batch command. This is useful if your verification is checking unit tests or running a linter or a type checker.

118. 22:14

     Instead, though, because I'm interested in code smells, I'm going to be using a language model that's going to be looking at the code and trying to identify any problematic patterns based on a set of rules that I've provided.

119. 22:24

     Now let's go back to our first batch and actually put this verifier to use. Remember, this batch is trivial, and fortunately, the verifier recognizes it as such. It comes back with an excellent report indicating which code smells it identified and which it didn't, and the status of this batch is turned to completed, green, good.

120. 22:42

     And this change in status is also reflected in the batch graph. Navigating back and toggling the color distillery, we can see that we have exactly one node out of many completed, and the rest are still yet to be handled.

121. 22:55

     But this already gives us a really good sense of the work that we've done and how it fits into the bigger picture.

122. 23:01

     So now our strategy for ensuring that there are no code smells in the entirety of our repository is straightforward. We just have to ensure that every single node on this batch graph turns green.

123. 23:10

     So let's go back to our batches and continue verifying until we run across a failure. We'll keep going in dependency order, making sure that we pick nodes that don't have any dependencies on other batches that we have yet to analyze.

124. 23:23

     This next batch is about as simple as the first, but because the init file is a little bit more complex, the report that gets generated is a little bit more verbose.

125. 23:32

     Continuing down the list, we come across the batch we identified earlier with some chunky files that have relatively high code complexity, and this batch happens to give us our first very green failure.

126. 23:42

     Notice that the status turns red instead of green.

127. 23:46

     Now, this batch has more files than what we've seen in the past, so the verification report is proportionally longer. Looking through, I see that it is listing file by file the code smells that it identified and which files.

128. 24:00

     I see one file is particularly egregious with its violations. We'll have to come back to that.

129. 24:07

     And if we zoom all the way back out to the batch graph and look at status indicators, we'll see the two green nodes representing the batches we've already successfully verified.

130. 24:16

     We'll also see the red node representing the batch that we just saw fail verification. Now, if our stated goal is to turn this entire graph green, this red node presents a little bit of an issue.

131. 24:26

     To convert this red node into a green node, we need to address the problems that the verifier found using the next step of the pipeline, the fixer.

132. 24:35

     Just like the verifier, the fixer can be defined in a number of different ways. The programmatic fixer can run a Bash command, or you can feed it out of Bash into a language model and hope it addresses the issues in a single step.

133. 24:47

     But by far, the most powerful fixer that we have uses the OpenHand agent SDK to make a clean copy of the code and spin up an agent that has access to all sorts of tools to run tests, examine the code, look at documentation on the internet, and do whatever it needs to, to address these issues.

134. 25:05

     So let's go back to the Spin up and Dash and run the fixer and see what happens.

135. 25:10

     Now, this part of the demo is sped up considerably, but because we're exploring these patches in dependency order, while we're waiting, we can continue to go down the list, run our verifiers, and spinning up new instances of the OpenHand agent using the SDK until we come across a node that's blocked because one of its upstream dependencies is

136. 25:28

     still complete. When the fixer is done, the status of the batch is reset. We'll need to rerun verification in the future to make sure that the associated open is reading in.

137. 25:38

     Looking at the report that the fixer has returned, there's not much information. Just the title of the PR. We've set this up so that every fixer produces a nice, tidy pull request ready for human approval.

138. 25:49

     Just because the refactor is automated doesn't mean it needs to be unviewed.

139. 25:54

     And here's the generated PR. The agent has done an excellent job of summarizing the code smells it identified, the changes made to address those, as well as any changes that it did not make.

140. 26:04

     It's also less than helpful, and that's for the reviewer, and it's a note for anybody working on this part of the code in future ages to request.

141. 26:12

     And when we look at the contents of this PR, we see it's very recent. All of the changes are tightly focused on addressing the code smells that we provided earlier, and we've only modified a couple hundred lines of code, the bulk of which is simply refactoring a deeply nested block into its own function call.

142. 26:28

     Not all of PRs are going to be this small, but our batching strategy and narrow instructions ensure that the scope of the changes are well constrained. This helps to improve agent performance, but it also will be easier to review code along.

143. 26:41

     From here, the full process for removing code smells from the entirety of the OpenHands code becomes clear. Use the verifier to identify problems. Use the fixer to spin up PRs that address those problems.

144. 26:53

     Review and merge those PRs to unblock new fixes, and repeat until that entire graph turns green.

145. 26:59

     We've already used this tool to make some pretty significant changes to the OpenHands code base, including strict typing and improving test coverage, and we could not have done it without the OpenHands agent SDK powering everything under the hood.

146. 27:14

     All right. So that's the, uh, OpenHands Refactor SDK, uh, powered by our OpenHands Agent SDK. Uh, we're gonna walk through a little bit later on in the workshop how to build something, uh, a little simpler, but, uh, very similar, where we get parallel agents working together to fix tasks that were, uh, discovered by an initial agent. [clears throat]

147. 27:34

     Uh, I wanna talk a little bit about strategy for both decomposing tasks and sharing context between these agents, as they're both really big important parts of agent orchestration. Uh, so effective task decomposition, uh, you're really looking to, uh, break down your very big problem into tasks that a single agent can solve, a single agent can one-shot.

148. 27:54

     Um, something that can fit in a single commit, single pull request. Um, super, super important because you don't wanna be, you know, constantly iterating with each of the sub-agents.

149. 28:03

     You want each one-- You want a, a pretty good guarantee that each one is just gonna one-shot the thing. You'll be able to rubber stamp it and get it merged into your ongoing branch.

150. 28:12

     Uh, you wanna look for things that can be parallelized. This is gonna be a huge way to increase the, uh, the speed of the task. Um, you know, if you're just executing a bunch of different agents serially, you might as well just have a single agent moving through the task serially.

151. 28:26

     Uh, the more you can parallelize, the more you can get many agents working at once, the faster you're gonna move through the task, uh, and iterate. Um, you want things that you can verify and inspect very easily and quickly.

152. 28:37

     Ideally, you'll have something where you can just, like, look at CI/CD status and have good confidence that if everything's green, you're good. Uh, maybe you'll need to click through the application itself, something like that, run, run a command yourself to verify that things look good to you.

153. 28:51

     Uh, but you wanna be able to very quickly understand whether an agent has done the work you asked it to or not. Uh, and you wanna have clear dependencies and ordering between tasks.

154. 29:00

     Uh, you'll notice these, these, uh, criteria are pretty similar to how you might break down work for an engineering team, right? You need to make sure that you have tasks that are maybe separable, tasks that, like, different people on your team can execute in parallel, and then collate the results together.

155. 29:14

     You wanna know, uh, once I get task A done, then that unlocks task B, C, and D, and then once those are done, we can do E. Uh, so very similar to breaking down work for a team of engineers.

156. 29:26

     Uh, there are a few different strategies for breaking down a very large refactor like the one we just saw Calvin just do. Uh, the simplest, like most likely you want, is to just go piece by piece.

157. 29:35

     You know, you might iterate through every file in your repository, every directory, maybe every function or class. Um, you know, this, this, uh, is a fairly straightforward way to do things.

158. 29:45

     It works well, uh, if those, um, dependencies are-- can be kinda executed, um, you know, without depending on one another too much. Um, so good examples might be like adding type annotations to our Python code base.

159. 30:00

     Um, uh, and then, you know, at the very end, once you've migrated every single file, say, you can collate all those results into a single PR.

160. 30:08

     A slightly more sophisticated thing would be to create a dependency tree. Um, and the idea here is to have some order into that piece-by-piece approach where, you know, you start, uh, as we saw Calvin do, you start with like the leaf nodes in your dependency graph, right?

161. 30:22

     You start with maybe your utility files, get those migrated over. Um, and then anything that depends on those, you know, it's gonna have those, those initial fixes in place, and the dependencies can, uh, can start working through, um, you know, their, their side of the process.

162. 30:35

     You can basically back your way up to whatever the entry point of the application is. Uh, this is often a, a, uh, better way to proceed. Um, it's more kind of like principle approach for how you're gonna order through these tasks.

163. 30:48

     Uh, another example is to create some kind of scaffolding that allows you to live in both the, like, pre-migrated and post-migrated worlds. Um, we did this, uh, for example, when migrating our React state management system.

164. 31:01

     Uh, we basically had an agent set up, uh, some scaffolding that would allow us to, to work with both Redux, Redux and Zestand at the same time. Um, uh, pretty ugly, not something you would actually really wanna do, uh, but it allowed us to test the application as each individual component got migrated from the old state management

165. 31:18

     system to the new state management system. Uh, and then we sent off parallel agents for each of the components, uh, got each component done, and then at the very end, once everything was using Zestand, we were able to rip out all of the, um, uh, all the scaffolding, so there was no more mention of Redux, uh, and

166. 31:35

     everything was working. But having that scaffolding in place allowed us to validate, you know, as each agent finished its work for just that one component, we could validate the application was still working, that component still works.

167. 31:45

     Uh, we didn't have to do everything all at once before we got some kind of human, uh, feedback from the agents.

168. 31:52

     Uh, next I wanna talk a bit about context sharing. Uh, as you go through a big, large scale project like this, uh, you're going to learn things, right? You're going to figure out, okay, what I-- my original mental model wasn't actually complete.

169. 32:04

     Uh, I didn't actually, uh, you know, understand the problem correctly. Um, your agents might, uh, run into-- You know, you might have a fleet of agents. You got ten agents running.

170. 32:13

     They're all hitting the exact same problem. You kinda wanna share the solution to that problem so they're not all getting stuck, right? Uh, there's a bunch of different strategies for doing this context sharing between agents.

171. 32:24

     Uh, one strategy, like the most naive thing you can do is share everything. Basically, every agent sees every other agent's context. Uh, so this is, uh, not great. Uh, it's basically the same thing as just having a single agent working iteratively through the task.

172. 32:35

     Uh, you're gonna prune your context window really quickly if you do something like this. Uh, so this is, this is not the way to go.

173. 32:43

     Uh, a, a better manual approach would be to have the human being just sort of manually enter information into the agents. Uh, if you have a chat message, a chat window with each agent, you can just paste in like, "Hey, use library one dot two dot three instead of one dot two dot two."

174. 32:57

     Um, the human can also launch either an agents.md or OpenHands micro agent to pass messages to these agents. Uh, but this does involve manual human effort. Um, uh, it involves a lot more, like, babysitting of the agents, so it's, it's not super scalable.

175. 33:14

     Uh, you can also have the agents basically share context with each other through a file, like agents.md. Uh, you can ha- allow the agents to actually modify this file themselves.

176. 33:23

     Uh, maybe they send a pull request into the file as they learn new things. Uh, downside here is that sometimes agents will try and learn unimportant things. Uh, they can get kind of aggressive about pushing information to this file, uh, so doing some kind of human review seems to help.

177. 33:37

     And then last, uh, this is probably the most, like, leading edge idea here, um, but you can basically give each agent a tool that allows it to send messages to other agents.

178. 33:47

     Uh, it could be like a broadcast message that goes out to all the other agents, uh, or it could be, uh, you know, point to point conversation. Uh, this is super, uh, fun to experiment with.

179. 33:56

     We're doing a lot, uh, to experiment with this now, uh, with our SDK. Um, but it's, uh, tricky to get right. It's, uh, you, you-- Once you get agents talking to each other, you're, like, increasing the, uh, uh, level of non-determinism in the system.

180. 34:09

     Uh, things can get a little blurry. Uh, I have a, an example here on the right of, uh, this is from a metadata support where they had two agents just talk to each other.

181. 34:17

     They just entered into a loop of wishing each other Zen perfection. Um,

182. 34:25

     cool. Uh, now I wanna work through an exercise. Uh, I would love it if you all, uh, wanna follow along. Um, you can access this presentation for, uh, copy-pasting purposes at, uh, dev.fh/openhands-workshop.

183. 34:41

     Uh, we'll, uh, we'll work through some coding exercises with the OpenHands SDK, specifically to, uh, do CVE remediation at scale. Uh, we're going to write a script that will take in a GitHub repository, scan it for, uh, open source vulnerabilities or CVEs, um, uh, and then set up a parallel agent for every single vulnerability we find to

184. 35:02

     solve that and open up a pull request. Uh, so dev.sh/openhands-workshop. Uh, let me know if anybody can't access it.

185. 35:11

     Uh, it's giving me the slideshow.

186. 35:14

     So it, so it, it should be the slideshow. If you wanna-- There will be, um,

187. 35:19

     uh, copy-pasteable prompts and, uh, links and stuff like that around slide twenty-nine.

188. 35:24

     Got it.

189. 35:25

     Uh, we'll get there. Uh, so in terms of how this process is gonna work,

190. 35:33

     uh, basically, we're going to start with one agent that runs a CVE scan on this repository. It's gonna scan for vulnerabilities. Uh, what's nice about using an agent for this is it can look at the, um, uh, the repository and decide, well, how am I gonna scan for vulnerabilities, right?

191. 35:48

     Am I gonna use Tribu to scan a Docker image? Uh, am I going to run npm audit on a package.json? Uh, so it can, it can basically detect the programming language and figure out, how am I gonna scan for CVEs here?

192. 36:00

     Uh, then once we have our list of vulnerabilities, we're going to run a separate agent for each individual vulnerability. Uh, each of these agents are gonna research whether or not it's solvable.

193. 36:10

     Uh, it's gonna update the relevant dependency, fix any breaking API changes throughout the code base, and then, uh, open up a pull request. Uh, what's nice about this is that we can merge those individual PRs once they're ready.

194. 36:21

     Can you show the link again? One more time.

195. 36:22

     Yeah. Uh, what's nice about running the solving in parallel is that, you know, we get, we get a bunch of different PRs, uh, so we can merge them as they're ready.

196. 36:33

     If one agent gets stuck, one of the vulnerabilities isn't solvable, all the other ones are still gonna work. Uh, maybe we get to ninety percent or ninety-five percent solved.

197. 36:42

     Uh, we don't have to get to a hundred percent in order to have any value here.

198. 36:47

     Uh, just a quick pseudocode of what this is gonna look like.

199. 36:50

     Uh, so this is an example using the OpenHands SDK of how to create an agent. You can see we create a large language model. Um, we then pass that large language model to an agent object, along with some tools, uh, like terminal, a file editor, a task tracker for planning.

200. 37:06

     Uh, we give it a workspace, and then we just tell it what we want it to do and hit run. Uh, this is a pretty, like, naive hello world example.

201. 37:14

     We'll see how it gets a little bit more complicated as we progress through this particular task.

202. 37:19

     Uh, but then once that first agent is done, we're gonna iterate through all of the vulnerabilities we get back out. Um, and then for each one, we'll send off a new agent, uh, asking it to solve that particular CVE.

203. 37:35

     All right. So, uh, to get started here, uh, I would say create a new GitHub repository. Uh, we can start to save our work there. Uh, you're also going to need both a GitHub token and an LLM token.

204. 37:49

     Uh, I would... Uh, if you sign up for, for OpenHands at app.openhands.dev, you can get a ten dollar free credit, um, LLM credits there. Um, if you're already an existing user, let me know and I can, I can bump up your, your existing credits for the pur-purpose of this exercise.

205. 38:05

     Um, then we're going to start, uh, an agent server. Uh, this is a, um, uh, basically like a Docker container that's going to house all the work that our agents are doing.

206. 38:17

     Uh, this is a great way, again, to run agents securely and more scalably. So instead of running the agents on our local machine to solve all these CVEs, uh, we're gonna run them inside of a container.

207. 38:28

     Uh, hypothetically, if we were doing thousands of CVEs, we could run this in like a Kubernetes cluster so that, you know, we have as many workstations as we want for our agents.

208. 38:36

     But for the purposes of this exercise, we'll just run one, one Docker container as a, as a home for our agents. Uh, then we can create, uh, an agent inside of the OpenHands Micro Agent to, uh, you know, start working through this task.

209. 38:48

     I'm gonna be using the OpenHands CLI as we go here. Um, you're welcome to check out the OpenHands CLI. You can also use Cursor or Cloud Code or whatever you're used to using, uh, as we, uh, kind of byte code our way through a CVE remediation process with OpenHands.

210. 39:03

     Uh, I'm gonna give it a couple minutes. I'm gonna walk through creating my GitHub repo, getting my GitHub token, et cetera. Um, uh, if y'all have any trouble, feel free to raise your hand.

211. 39:13

     I can come around and, uh, help, you know, getting in all the tokens set up.

212. 39:37

     You said app.allhands.dev?

213. 39:54

     App.allhands.dev. Yeah. So I've got my new GitHub repo here.

214. 40:11

     Uh, so I'm gonna add a quick OpenHands Micro Agent here.

215. 40:22

     Create new file. And I'm just gonna tell OpenHands, "Building a new,

216. 40:48

     uh, process for remediating CVEs with agents." Relevant docs for the OpenHands SDK are at

217. 41:04

     OpenHands.dev/openhands-sdk/. Uh, so this will give OpenHands a little bit of context.

218. 41:19

     So we're gonna use the, um, we now have this repo to actually, uh, to get a token. Uh, I'm actually gonna do it here so that it shows my token, but you can go to GitHub settings, uh, your profile,

219. 41:35

     then developer settings. Personal access tokens. Uh, I like to do classic tokens.

220. 41:45

     Uh, create classic token, give it a name, and then, uh, the repo scope is really what you'll need. Uh, that way we can open up pull requests, uh, to solve each of the CVEs involved.

221. 42:02

     We do have a classic token, not a, not the new fancy ones.

222. 42:07

     I haven't even gotten the link. You're welcome to do... I guess you could create a new repository.

223. 42:15

     I haven't gotten used to them either, so [laughs] I'm with you. [laughs]

224. 42:20

     Stuck in the old ways.

225. 42:22

     So what permissions do we need to-

226. 42:24

     Uh, just a repo permission. I'm also just gonna show, if you sign up for, uh, app.allhands.dev.

227. 42:39

     Um, if you go to the API keys under your profile here,

228. 42:46

     you can get your OpenHands API key, your LLM key here. I won't show it, but this will allow you to use in our LLM proxy.

229. 42:57

     Okay.

230. 43:00

     What other steps are... All right. Last, I'm gonna start up some agent server here. You'll probably want to copy and paste this out of the presentation.

231. 43:17

     See. Um, yeah. [clears throat]

232. 43:34

     Got my repo clause. Run the agent server container.

233. 44:23

     Uh, back here, if you do wanna work with the OpenHands CLI.

234. 44:29

     Um, give me tool install OpenHands. Gonna give me... [clears throat]

235. 45:08

     I'm gonna start up the OpenHands CLI. Again, you can use Podcode, Cursor, whatever else if you want. Uh, you folks need a little more time with the setup? The LLM key, get a token set up?

236. 45:23

     Sorry, check here. Uh, so I'm gonna start with, uh, this first prompt. Uh, basically what we're going to do is we're going to point our agent, uh, at the OpenHands SDK, uh, point it at the documentation, uh, and just ask it to basically check that our LLM API key is working, that it can actually do an LLM

237. 45:45

     execution. This will be like a very basic hello world, so just to kinda get started here. Uh, I'm gonna tell it, uh, I'm using, I'm using the OpenHands, uh, key that I generated at app dot allhands dot dev.

238. 45:58

     Um, so I'm telling it to use this OpenHands/bot sonnet-four model. Uh, you can replace this with Anthropic if you wanna use just, like a regular Anthropic API key. Uh, you may need to set this model a bit differently depending on if you're using OpenAI, uh, using Light LLMs. [laughs]

239. 46:14

     So you can look at the Light LLM docs to figure out if you, if you have an open API key or an OpenAI key, uh, you can look at the Light LLM docs to figure out which model by looking for the string.

240. 46:24

     Uh, but I'm just gonna copy and paste this as is. [sneezes]

241. 46:39

     Sorry, what's the step for, um, agents dot md or the one for OpenHands?

242. 46:44

     So I would say just create a, uh, a file, either the agents dot md if you're working with a, a tools compatible with that, or, um, uh, for OpenHands, we have what's called a micro agent.

243. 46:58

     Let me get to it. Uh, so OpenHands slash-- dot OpenHands slash micro-agent slash dot, if I can mention, repo dot md is the description of the repository you're in.

244. 47:07

     Um, and I just gave it a couple links to the SDK documentation, uh, at the repository for the SDK, so that it has access to, you know, basically the, the API docs there.

245. 47:21

     This is kind of an optional step. It'll make things a little easier, though.

246. 47:29

     Docker. [laughs] W agent's doing. All right. It thinks it's got something good. So let's see what's gone on.

247. 47:42

     Python CVE solver. Need environment variables.

248. 48:10

     Uh, we're using Thera to set my environment variables here.

249. 48:15

     Make sure I don't check those in. One more time. [clears throat]

250. 48:33

     Got a small error. Looks like the agent didn't quite get the API docs right.

251. 48:40

     Let's, uh, paste the error back, see what happens.

252. 49:01

     Let's try again. Of course, lab dev, never, never fail. [clears throat]

253. 49:51

     Got an author. [coughs] She's working. [clears throat] [coughs]

254. 50:49

     Version. UV. [background noise] Let's use Cloud. [laughing] [background conversation] It's just weird. [laughing]

255. 51:08

     When you, when you do this. Yeah. UV makes it so bigger.

256. 51:15

     UV tool OpenHands. That breaks?

257. 51:19

     Yeah.

258. 51:19

     You know what version of UV you're on?

259. 51:21

     On, uh, oh nine six. Oh dot nine dot six.

260. 51:26

     What error are you getting?

261. 51:27

     Nine eleven.

262. 51:28

     I don't know why.

263. 51:29

     Go on to Google. It seems very [beep] No executables are provided by package OpenHands. Removing tool. Error failed to install entry points. Oh.

264. 51:38

     I'm new-ish to the Python world, so I assumed I was doing something silly.

265. 51:42

     You could try updating to seven oh nine to eleven, which is what I'm on, but.

266. 51:45

     Okay. Yeah, I'll try.

267. 51:48

     I have a question.

268. 51:49

     Yeah.

269. 51:49

     Um, so I was able to... I see you're running through the CLI. I was able to run this on the, like, AllHand- AllHands dot dev.

270. 51:57

     Yeah, cool.

271. 51:57

     And it submitted a PR, created it. Looks good.

272. 52:00

     Awesome.

273. 52:00

     Why are you doing it through the CLI?

274. 52:02

     Uh, really just for [clears throat], uh, uh... Normally, I actually prefer to work through the web UI here. Um,

275. 52:13

     I think, uh, being able to, like, run and show the script is working locally, uh, is, like, a little bit b-better to hand off. I actually like to work through the web UI normally, and then have the agent push, and I pull locally if I really wanna work locally.

276. 52:25

     Yeah.

277. 52:25

     But I figured that was just extra, extra steps for presenting purposes. Uh, but yeah, feel free to use the, the web or the tool.

278. 52:41

     Looks like I got my bright API key set up here. Just gotta fix the output. [sighs] [background conversation] [clears throat]

279. 53:59

     Should we get, uh, two hundred from that?

280. 54:01

     What's that?

281. 54:03

     Should we get two hundred for clicks and good setup?

282. 54:08

     I'm just asking for clarity.

283. 54:09

     Uh, yeah, you should get something like this. Uh, like, I just got finally, uh, where the LLM says, "Hello, OpenHands."

284. 54:21

     Uh, just gonna, uh, call this LLM section.

285. 54:37

     Anybody manage to get the LLM connection working?

286. 54:41

     I think so. I've created the file.

287. 54:43

     Nice. Uh, just so a quick view of what this looks like. Yeah, I have it in the [background conversation]

288. 54:55

     Uh, basically you can see we create an LLM. Tell it what model we wanna use. Tell it what the API key we want, we want it to use. Um, and then just send a, a quick message to, to the LLM to make sure it's actually working. [sighs]

289. 55:10

     Uh, all right. For the sake of time, I'm gonna move towards prompt two. Uh, so here we're going to actually start to do some work with agents. Uh, so we're gonna tell, um, you know, the agent we're working with, uh, we want to use the SDK to create a new agent, uh, that's going to take in a

290. 55:28

     GitHub repository. Uh, it's going to connect to a remote workspace, uh, running at localhost:eight thousand. Again, that's the, the Docker start command from before. If you haven't already run that, now's a good time to get Docker running.

291. 55:41

     Uh, Docker run this agent server. Uh, [clears throat] it's going to, uh, clone our repository into that Docker container. Uh, we're gonna create an agent that's gonna work inside that Docker container, and we're gonna tell that agent to scan this repository for any vulnerabilities.

292. 56:44

     With, uh, the OpenHands CLI, is there a way to interrupt and get it to stop?

293. 56:48

     Uh, I think Control P for pause. Yeah.

294. 56:53

     And then can I insert my corrections or-

295. 56:55

     Yeah, then you can type a new message or just type continue.

296. 57:02

     Thank you.

297. 57:08

     Yeah.

298. 57:13

     So I got the CLI to install, but I had to add dash AI. [background chatter]

299. 57:19

     It seems on PyPi that there's a dash AI version, but then it says in the docs that it's [stutters] I don't-

300. 57:26

     Should be... I think the AI one is deprecated, but it is, it is a usable CLI if you wanna use that. I'm gonna surface that one up with our team.

301. 57:40

     Did you get the dash AI one to work? Because as soon as I tried to run it, it crashed.

302. 57:45

     Oh. [laughing] [laughing]

303. 57:46

     Oops. It installed. I was so happy.

304. 57:49

     Yeah, yeah. It installed, and then it, it didn't work.

305. 57:52

     There was a deprecation warning when I go to version, so yeah. Fair enough.

306. 57:55

     There, there is a... If you wanna download an executable binary on our release page-

307. 57:59

     Okay.

308. 57:59

     That might be straightforward, or you can also run it in a Docker [background coughing]

309. 58:03

     container. Um, if you look at our CLI docs.

310. 58:09

     Uh, I think there's a UV run as well. [clears throat]

311. 58:58

     Try UV, UV run.

312. 59:03

     And... Got the version.

313. 59:11

     You have the version?

314. 59:12

     Yeah. Interesting. [sneezes] Think it was in one.

315. 59:18

     Oh, interesting. All right, so you can try adding a kind of version. That's for the Open, not OpenHands AI.

316. 59:25

     The regular OpenHands.

317. 59:25

     The regular OpenHands.

318. 59:27

     Okay. Cool. Thank you.

319. 59:33

     All right. Supposedly have an agent working here. Let's see. I'm gonna run it with [background coughing] Paris repo. Uh, which should have a few CVEs in it. Uh, let's see if we find any vulnerable.

320. 59:46

     Uh, by default, OpenHands will, uh... We'll visualize the output here, so we can see the agent working, uh, even with the SDK, very similar to how we saw the, um,

321. 1:00:01

     uh, CLI working. [background coughing] Uh, you can see it's updating its task list.

322. 1:00:07

     It's, uh, cloning the repository. It's, uh... It doesn't have Trivy installed, so it's installing Trivy. It's basically doing what we would expect an agent to do, uh, given the task we fed into it.

323. 1:00:49

     So we're running Trivy now.

324. 1:01:00

     Perfect.

325. 1:01:04

     Show a bit about what this, what this generated code looks like. Uh, you can see, so we, we instantiated our LLM in the first step.

326. 1:01:14

     Now we're actually passing this LLM to an agent. We're also giving it a terminal tool and file editor tool. Uh, we're creating this remote workspace that's connected to our Docker container so that agent can start working in its own environment.

327. 1:01:26

     Uh, we create what's called a conversation, which is basically one chunk of context that the agent's gonna manage as it goes about its, its work. Uh, we pass it a task with some clear instructions for what it's supposed to do, uh, and then set that, and then set that task to the conversation.

328. 1:01:46

     Looks like that initial scanner agent is almost done.

329. 1:01:57

     It looks like that agent ran just fine with these results.

330. 1:02:07

     In the meantime, I'll keep, uh, keep plugging along here. Now we've got an agent that's, uh, scanning for vulnerabilities.

331. 1:02:18

     Uh, so the next thing I'm gonna ask this to do is basically we're gonna reach into the agent's environment and get the vulnerability list out from it. Um, the idea is we're going to have it save just the vulnerabilities to a JSON file.

332. 1:02:31

     Uh, then we can, on that workspace object inside of the Docker container, we can run an execute command in order to get those vulnerabilities back out. Uh, we also have some, some, uh, functions for, like, manipulating files, uh, within the workspace.

333. 1:02:44

     Uh, then for now, we're just gonna iterate over the vulnerabilities to that JSON file, print it out just so we can see if we were able to reach into this workspace and get some information back out.

334. 1:03:03

     All

335. 1:04:50

     right. Supposedly good to go. See what happens.

336. 1:04:56

     Should run it.

337. 1:06:04

     Actually got some vulnerability results.

338. 1:06:33

     Agent's finished. Let's see if our script can get

339. 1:06:37

     results back. Error parsing JSON.[clears throat] [background noise]

340. 1:08:13

     One more time. [background noise] What, what is the observation event?

341. 1:08:28

     So for every [background noise] states, uh, there's a, there's an action and then an observation. So it might be run this command, and then an observation goes back with the output of that command.

342. 1:08:38

     So it's a hook [background noise].

343. 1:08:40

     Uh, it's more core than a hook. It's, it's, it's the,

344. 1:08:45

     uh, basically the entire trajectory the agent takes to stand up to events, and then, uh, there's two kinds of events: actions and observations. So OpenHands, whenever we make calls with the LLM, it comes back with an action to take, or basically a tool call, uh, and then the observation's like a tool calls. [background noise]

345. 1:09:23

     If anyone's stuck on anything, I'm happy to come around to you. Feel free to raise a hand. [background noise]

346. 1:09:33

     What's that? [background noise]

347. 1:09:37

     Number four.

348. 1:09:38

     Uh, I just did number three. [background noise] Nice. Yeah. It looks like it's printing the CVE list.

349. 1:09:55

     Perfect.

350. 1:09:55

     Yeah, that looks good. Jump ahead. [background noise] So the, the

351. 1:10:21

     process we're going through here with the five, the five prompts, this is really, uh, to demonstrate what it would feel like to actually, like, build with our SDK, right?

352. 1:10:31

     Uh, this is not the way that I would-- This is the way I, I would maybe like to work if I was actively working on a problem. You know, I, I could have just given you this, this whole fully packaged Kubernetes pre-built, right?

353. 1:10:41

     Yeah.

354. 1:10:41

     That had all this built. But, uh, is, is that what you're asking? Like, why are we, why are we pasting these prompts in one by one?

355. 1:10:48

     Eventually, we get a very large script, like-

356. 1:10:52

     Right.

357. 1:10:52

     -we should break it into several files or sections or-

358. 1:10:56

     Sure.

359. 1:10:56

     -choose. I don't know.

360. 1:10:57

     Yeah, yeah, yeah. No, I think there's, there's definitely better ways to organize this code than to have one single script. Just, uh, easier for demo purposes.

361. 1:11:05

     Okay.

362. 1:11:05

     Yes. I do have a, I do have a demo repo, um, I think it's OpenHands/cve-demo that uses special classes. There's a single, you know, CVE agent subclass that uses the same LLM.

363. 1:11:17

     It's a little bit more better organized than just this one script. [background noise][background noise]