How Coding Agents Change Software Development Forever - Hailong Zhang

Talk transcript

1. 0:01

   Hi, everyone. In this presentation, I'm going to talk about coding agents and how to work with coding agents.

2. 0:09

   This is my agenda. First, I'm going to talk about the future of software development workflow, and then I'm going to show you example of coding agent, and this agent will boost the unit test.

3. 0:21

   And finally, I'm going to dive deeper into how we solved real-life problems with agents, how we build the agents.

4. 0:32

   Generative AI has shifted a lot of things, and the people believe AI coding is going to be the future. And we believe a lot of the routine works will be handled by AI, including entry-level coding tasks, debugging, testing, and operating.

5. 0:49

   But still, I think the creative works, including product design, architecture design, and the difficult issues, still need to be handled by humans. So it must be a human and AI agents collaborate together to solve problems in same workflow in the future.

6. 1:11

   The collaboration has two types. First one is synchronous and another one is asynchronous. I think people may pretty familiar with the synchronous one, GitHub Copilot or Cursor. This is AI live inside your IDE, and this AI works simultaneously with human.

7. 1:29

   When you are typing, it's working, right? So

8. 1:33

   we have already had this kind of product for a long time, since twenty twenty, and it's getting mature since twenty twenty-three, and the rapidly grows twenty twenty-four, and it's widely adopted today.

9. 1:47

   But the asynchronous one is pretty new, and it just started twenty twenty-four. The asynchronized one is more like a bot inside your workflow. For example, GitHub bot, and this bot can be triggered either manually or automatically.

10. 2:02

    And this bot will complete the task without human's attention, fully autonomous, and will submit a deliverable once it's done. So it's totally different experience, and these two types of agents are all important.

11. 2:17

    And I think people need all these kind of agents to solve the real-life problems.

12. 2:26

    Imagine this is your workflow in the future,

13. 2:29

    and the workflow will have a lot of AI agents, small agents live inside your repos, workflows in different stages to solve different problems, including unit test, including fixing bugs, writing documents, submit code reviews, and releasing from these kind of tasks, and so human can focus on more creative part.

14. 2:55

    Next, I'm going to show you a detailed coding agent, which will boost unit test.

15. 3:02

    I think a unit test is more important today than before because in the AI era, a lot of code are generated by AI. As we just talked, if you are using Cursor, you must be familiar with tab.

16. 3:15

    For each of the tab, multiple lines of code in multiple parts of the file will be generated, and it's very hard to... for you to focus on every generation, and you may overlook something.

17. 3:26

    The coding speed is really, really fast, but it also easy to generate bugs. So one of the way to solve the issue or prevent the issue is to write a unit test.

18. 3:38

    People talk about unit test. People think unit test is important. But when writing unit test, people hates it. Yeah, developers, every developers hates to write unit test. So we build a Guru, an AI agent to help developers to write a unit test and managing unit test.

19. 3:58

    This is a screenshot that pull request of unit test generated by Guru. And the process is triggered either manually or automatically by a pull request. When human submit a pull request, Guru will detect the change of the code and determine if this change needs more unit test or change existing unit test.

20. 4:21

    And the Guru will do the coding work, writing the test, and to run the test to make sure everything works, and then prepare and submit a pull request, including all the information showing in this screenshot, the summary of the test and the coverage improvement of the test.

21. 4:37

    Now, human need to review this pull request and determine if this unit test is good or not, uh, to merge into the, to the repo.

22. 4:49

    Let's look at the, uh, performance of Guru in production. This is also a screenshot from GitHub. It's a list of pull requests. And, uh, as you can see, over fifty percent of the pull requests are merged and accepted by humans.

23. 5:04

    So fifty percent is not a very large figure, and there are a lot of rooms to... for improvement. But I think in real life, fifty percent is already meaningful in production and help solve a lot of problems.

24. 5:19

    And also, Guru handles around eighty percent of the unit test itself in its own repo. We are using Guru every day, and then most of our tests are generated by Guru.

25. 5:33

    From the commits, you can see Guru is already in, in the counts of commits, Guru is already the first contributor in our team, and we believe more and more agents will become contributors in people's repo this year, twenty twenty-five.

26. 5:50

    Next, I'm going to dive a little deeper into how we build Guru to solve real-life problems.

27. 5:58

    To build an agent, I think the first and the most important thing is to define the problem itself. A clear, concrete, and a doable problem is crucial For example, unit test is a problem, but software engineering is not a problem.

28. 6:16

    And once we have the problem, we need to build the datasets for the evaluation purpose and also the evaluation harness. And then we work on LLMs, building context. And the last, we need Agent OS or framework to orchestrate everything together.

29. 6:36

    As we all know, agents work on top of LLMs and Gru work on all frontier models, either it's from OpenAI, Anthropic, DeepSeek, Google, et cetera. And we evaluate the models on the different scenarios, try to find the best model for each of the scenario.

30. 6:53

    That means even within the same job, Gru may use different LLMs for different stages. And also, we fine-tune models to improve the performance. For example, we fine-tune GPT-4o with human-labeled unit-test code to improve the generation of the test code.

31. 7:18

    Building context in agents is really important, and, uh, we think it's, it's necessary to build the context for specific tasks. For example, here in unit test, we build the context for each of the languages and the frameworks, and we also gather all the information from environment.

32. 7:39

    For example, the GitHub issues, uh, code reviews, commits, pull requests, README code itself, and all this information need to be gathered together and to be filtered and to be selected to fit into the context.

33. 7:57

    We talk about unit test a lot, but our vision is not only unit test. We want to build agents for different software engineering tasks. For example, the, the refactor task, the E2E test task.

34. 8:10

    It is almost import... impossible for us to build every of the agents from scratch, so we build the framework, or we can call it agent operating system, Agent OS.

35. 8:22

    Different tasks may share similar runtime, similar tools, similar context, so we abstract the common infrastructure, and this operating system will enable us to build the agents in this domain in a really fast pace.

36. 8:42

    The agent era is coming. Let's embrace agents in workflow. Thanks for watching.