The Last Human Code Review: Building Trust in AI-Generated Code

Talk transcript

1. 0:00

   [upbeat music] Are you all set up with your AI factory? Everything is smooth?

2. 0:18

   As you're sitting here, your code is being deployed, ARR growing, right? Is- if it's like that, raise your hand. No? Oh, okay. Two people, great. Um, let us know, like, before and after this talk what the ARR and, and

3. 0:33

   do you feel like there is a bottleneck, a new bottleneck that is not on writing code, rather somewhere else, else in the SDLC? Is that your biggest thing to tackle if, if you are tackling that right now?

4. 0:47

   Okay, so you're in the right place. And what about, like, code review, verifying that the code work according to your intent, according to your architecture, standards, best practices, et cetera?

5. 0:59

   Is this something that you're tackling day to day or week by week? Okay, so you're, you're in the right place. So I'm Itamar Friedman, the CEO and co-founder of Qodo.

6. 1:09

   Um, I don't share it too much, but Qodo stands for quality of development optimization. Our mission and, and, uh, is to help you all have a code governance, code review platform that understand your code base, your tribal knowledge, your best practices, and that does not come off the shelf from a model.

7. 1:32

   It requires a system that optimize for you as we go, and that's, uh, why we decided to name our company, uh, this way. Uh, so, uh, I'm gonna talk about the last human code review and what do we need to do in order to get there.

8. 1:48

   Okay? And, um, c- can you switch, instead of seeing myself, seeing the presentation here? [laughs] Thank you. Um, so I, I think like first of all, we need to agree

9. 1:59

   on why do we have code review at all. So I think basically these are two buckets that we should agree on, and I think, I think it's quite common.

10. 2:08

    One is we wanna validate the code that is in high quality, safe, maintainable, the right architecture, according to our best practice, et cetera. The second re- reason is actually alignment and learning, right?

11. 2:21

    Like where senior developers, for example, have o- one last chance, a gateway, a gatekeeper before, uh, it's being, uh, uh, code is being pushed to production to have that alignment and teaching, right?

12. 2:34

    So these are the two reasons, and then that's what you need to think if you're trying to automate a code review. Can human still do that? And this, if this is the right place, the, the code review process, the pull request process is that the right place to still doing these two things?

13. 2:51

    If you have, uh, new tools, new processes that will help you unblock this bottleneck, but let you still do these two buckets of tasks, then you're on the right path.

14. 3:04

    Uh, so we're here to ask yes, no. Is human code review still optional end of twenty twenty-six? Uh, is it becoming optional or is it still mandatory? Okay? That's, that we're here to answer.

15. 3:19

    Do you think like every PR or the majority of PRs are gonna be reviewed line by line or, you know, bucket by bucket in the, in the chunks, in the hunks by your developers?

16. 3:30

    Raise your hand. Or do you think, do you think by end of this year your developers are still going to review diff by diff?

17. 3:38

    Okay. So try to think why, why is that happening. Now I, I, I wanted to share with you that yesterday night we did a drone show and I had to, uh, opportunity to talk to different people during that drone show, different, uh, engineering leaders, and what I can tell you is that I saw two very different groups

18. 4:00

    of school of thoughts. Okay? They both agree that bugs [laughs] are coming in different shapes. For some this is fine, [laughs] and we're just going to fix that quickly after it hits the production, and for some not.

19. 4:15

    So, so actually we do see two teams. Those that are thinking about the room is split into two. The other are thinking like, "Hey, we have to let, make sure that every piece of line is, is trusted, uh, and, and a human must review that."

20. 4:31

    Or the other group, somewhat [laughs] reckless or so, saying, "Let's, let's like push those bugs into production and we quickly fix that and that's how we actually do things because it's much faster.

21. 4:47

    Velocity is more important than getting, getting things, uh, right." And I think like you need to think like where do you sit? Of course I put it in two, two sides of the spectrum, but there's some, somewhere in, in between and you, you need to think what, what's your philosophy because that will lead you to different milestones

22. 5:06

    or different tools that you need to use in order to get that, uh, uh, confidence that you can skip over a, a human review in the pull request, in the code review.

23. 5:17

    So let's start talking about what is the process, how do we need to start thinking about it. So I claim that models are not, not a barrier anymore. Like it's not a matter of, of the, you know, a model doing a good re- good, good job or not, having the right reasoning or not.

24. 5:35

    Basically the models are improving, but I'm telling you I just came from one of the leading labs [laughs] where we are inspecting how benchmarks for code review did not change a lot throughout the latest model.

25. 5:50

    The, the key here is actually context. Okay? Like the models, if you give them the right context, and what is that right context we're gonna talk about, they could already reason pretty well over what- Is the issues that we-- What are the issues that we need to surface for a certain change in the code?

26. 6:09

    Otherwise, if you don't have the context, even the best model out there, they will give you different types of, of, uh, bugs and issues. Some of them are really good, but in many cases, they will simply tell you, "Hey, did you consider error handling?"

27. 6:22

    Uh, uh, like, or not. By the way, error handling could be like a really good thing to handle. Depends. In some cases, it's critical, in some cases are not.

28. 6:31

    And again, the context is what, what matters. Right now, our context is like spread all across. Like, we have agents MDs, Claude MDs, skills MDs, and, and the thing is that each, each one of them has like different standards, uh, different org-organization and sub-organization are dealing with diff-differently.

29. 6:52

    Even within a certain team, you might be using that, that differently. And you're actually maybe using like the same, uh... One, one team is using, uh, the same agent to do code review and-- Sorry.

30. 7:05

    Co-coding and code review. The other might be using something else. And all of that does not bring you the trust and consistency that you're looking, uh, towards. Um, by the way, like you also might have like coding agents that are running in your IDE, but you're probably building the AI factory that running those agents like in workflows

31. 7:26

    to automate some of the coding. I see teams that are already, uh, having more lines of code being, uh, shipped that are not generated from the CLI or, or the IDE.

32. 7:37

    So how do you control all that? Like that's, that's missing, uh, like in-- right now, like in our, in our, uh, tools, in our infrastructure. A-and let alone if you add those MCPs and, and, and RAG like, uh, style, uh, context, I, I don't know if you have like great visibility.

33. 7:57

    There are ways to tackle that. There's great talks out there. Check how you could have like MCP versioning and have datasets for every M-- Uh, like a benchmark for every MCP change, but that's hard to manage.

34. 8:09

    We're missing like a governance layer for us to move to the next level, okay? Like, where we can actually trust the code without human reviewing it. So where, where is that context?

35. 8:21

    Where is that context? Basically, I would say experience, tribal knowledge,

36. 8:28

    wisdom of your developers is a lot in their heads. They are in some documents. There are documents, infrastructure documents, um, but a lot-- By the way, a lot of them is our Slacks or, or Teams or, or so, um, the, the data is there, but a lot of the information are, are in your developer heads, and we

37. 8:47

    need like to, to codify them. Now, um,

38. 8:53

    I think like, basically, another thing that I'm saying, like when I'm saying that we need to codify human knowledge, what we're actually saying is that we're trying to build an interface for agents and interface for humans to collaborate each other on that, on that knowledge.

39. 9:11

    And that's a very important conte-- Uh, um, a very important point. When you want to extract that information, the tribal knowledge from your s-- Uh, like, uh, senior developers, et cetera, and codify that, do you codify that only in agent's language, uh, which is very maybe verbose and structured, or you wanna codify that in a wiki style,

40. 9:34

    uh, get started and all that, what developers love, love doing? And the answer is that you probably need to build your context lake, your context engine. As I mentioned, that's the s-- The, the, the real like goldmine here to, to get the code review, uh, like auto-automated.

41. 9:51

    You have to have it fitting for both. So what you're seeing here, for example, is that Qodo, as an example, but you can use other tools, help you collect all the rules and standards that your team own, that your team is using day to day, and then it will provide that information during the review for humans.

42. 10:16

    Hey, notice that Qodo used four rules... Uh, sorry. Uh, used, uh, uh, many rules and four are violated, and that includes a link to all the rules that are, were being used.

43. 10:29

    That's for human in order to trust, okay? In order to trust the results that are coming from your code review tool, et cetera, you have to build that interface for, for human.

44. 10:40

    You have to accumulate that knowledge and have an interface for human. But you also want to have an interface that is dedicated for agents. What you're seeing here, uh, for example, is a com-comment, for example, by Qodo, that is speaking to another agent.

45. 10:56

    "Hey, dear agent, Qodo just reviewed this, uh, PR and has found five different issues." Qodo already spent, uh, like, uh, some back-background task and use Claude Code, for example, Harness, in order to do fixes, and there is a closed PR, like you can see here on the top right, there is some closed PR with all the fixes.

46. 11:20

    And now, when an agent is coming to review this PR once again, then it has like a cherry-picking moment, uh, with everything that all the code that is actually is passing your rules, your standard, uh, and everything like we're, like more a-architectural decision that we're gonna talk about.

47. 11:39

    So what, what I said so far is that

48. 11:43

    just to, to ca-- Uh, like recap so far, if you want to trust and you want to get to a point where you're trusting the code is being shipped, you need to have the right context that is being gathered and being used during the code review process.

49. 11:59

    Then you will see that this code review process have links and information for human and have links and information for, for agents, okay? And when that is in place-

50. 12:11

    You will see that developers are writing less and less comments in the pull request, and then after one hundred of these pull requests, there's human-- no hu- more human review, you know that you're ready for, for automation.

51. 12:24

    Okay? Now, back to, back to the context. So far, I talked about relatively simple context. I talked about, uh, rules and standards and skills, but actual, the human knowledge that the tribal knowledge in your organization sits in understanding the system architecture.

52. 12:43

    What are the P-zeros, the, the bugs that actually made an outage, outage for, for you, unfortunately, in the last like three months or so? When a microservice, uh, one changed its contract and broke a microservice two, right?

53. 12:57

    That is-- does not exist in most code review and, and like if you try to build yourself, it's really hard to build, but it is available in some of the, those like code review, uh, solutions that is dedicated for that.

54. 13:09

    For example, what you're seeing here is the graph being built for a certain microservice and all the repos and, and their connection. And in each node and edge, there is-- what is the con-- if it's an edge, what is the contract between two, uh, piece of your software?

55. 13:26

    But also links to history of discussions between developers that they had when they fixed an issue because a root cause analysis, and now you need to codify that, okay?

56. 13:38

    And when you get to, to that level of a context engine, now you're ready to start approving and blocking PRs automatically. And you want to do that not just by letting AI some like choose by yourself, rather giving some semantic rules that, that, for example, when you-- when do you guys

57. 14:03

    approve or, or, or block a PR? And that knowledge also needs to be accumulated as part of your, your context. What I'm actually saying is that software development, at least code governance, is going to change from reviewing your pull request to actually reviewing your entire software development from a graph obstruction,

58. 14:28

    where you're seeing your PRs as bubbles with all the issues that might, uh, happen, even with three different PRs are in on the fly, when they-- which, which contract they might bridge, they, they might ruin,

59. 14:43

    okay? And that's how the software development, uh, uh, future is gonna look like. What you need to do in order to get there is to codify your standards. It needs to be built in a way that humans can trust and audit and control.

60. 15:01

    You need to build real-time, self-learning context, learning from peer history, learning from accepted and unaccept, learning from discussions between, between developers, learning between, uh, learning like from those cases that broke your production.

61. 15:18

    And that context needs to be not just like thrown into files. It needs to f- to, to, to sit and located in a place that agent understand where is that context fitting.

62. 15:30

    And then you need the governance infrastructure that gives you that visibility of what's happening. Rather the graph I presented, but there is more to that. Okay, I'm gonna show you very soon another vi- uh, visualization that helps you understand the overall status of your PRs and software.

63. 15:47

    If you are already shipping AI-generated code faster than your human can review, I'm actually saying that you are in the problem. You're not like ahead of the problem. You, you-- if you put the infrastructure, if you gather the context, if you start accumulating how code review can be automated for you, then that's where you're gonna get the

64. 16:08

    ten x velocity that you're being promising your CEO or yourself or, or your developers, because otherwise it's a bottleneck. You need to own your rules and standards and codify them and get analytics and statistics about each one of them.

65. 16:23

    How many times they're being caught? Which rules and standards and skill is actually being used during the review process? And is it useful or not, or does it need to get an update?

66. 16:33

    You need to have visi- full visualization of your software graph. What is the connection? What are the contracts? What is working? What is not working? What are two PRs are gonna crash very soon because they're touching the same agent and they don't, and they don't know?

67. 16:48

    And you need to start learning how to auto-approve and auto-block. That won't happen in like immediately. That needs to gradually being automated for you step by step by adding more rules for blocking and more rules for, for, for approving over time.

68. 17:11

    So trustworthy automated review, your rules, your standards in your software graph placed in the right edge and the right node of your software. Learning from the tribal knowledge of discussions, of history of your software and codifying that.

69. 17:30

    There's so much tribal knowledge what to do and not to do, and that needs to be codified as well, both for the agents and for the humans. And basically, you need to accumulate that experience and codify that.

70. 17:44

    And sorry for the buzz or whatever you wanna call it. That's moving from artificial intelligence to artificial wisdom. Because right now, you know better than I that your developer holds the judgment of what's bad and what's good.

71. 17:59

    It's not your, your software, not your AI tools. If you wanna get to a point where judgment is moving to your AI tools, that's AI wisdom, where that experience needs to be codified in the right way, the right place for agents and humans.

72. 18:16

    And that's why we are here, Qodo, to help you, and our

73. 18:21

    vision and mission is not far away from now. We wanna reach in twenty twenty-seven to a place where you have zero outages, zero bugs in production, at least the critical and the high ones.

74. 18:33

    That's why we're here. Thank you so much for having me. [audience clapping] [upbeat music]