The State of AI Code Quality: Hype vs. Reality

Talk transcript

1. 0:00

   [on-hold music] I'm really excited being here.

2. 0:22

   So many-- so much pragmatic and insight and suggestions. I was sitting there, uh, just, just before. So I'm Itamar Friedman, the CEO and co-founder of Qodo. Qodo stands for quality of development, and I'm gonna share, uh, our reports and other companies' reports about state of AI code quality.

3. 0:39

   Uh, you know, trying to, uh, talk about the hype versus reality, which was, uh, like, one of the, uh, points that were discussed here quite a lot, which is awesome.

4. 0:49

   So in the last three weeks, four weeks, we saw like, three outages in the clouds, unfortunately, right? And these are f- coming from companies that really care about moving fast, right?

5. 1:00

   They're, they're, they're saying themself that they're using AI to generate code ten percent, thirty percent, fifty percent. At the same time, they care about quality. So how did that happen?

6. 1:10

   And is it, is it related? I don't know, but let's have some... Uh, I'm gonna share some guess. So by the way, [REDACTED:age] percent of developers say that the, like, quarter of their code is either generated by AI or in f- in f- like, uh, uh, shaped by AI, and fifteen percent say that even more than eighty,

7. 1:27

   eighty percent of their code, uh, is basically generated or sh- or shaped by AI. Now, people are using AI to do vibe coding, but actually they're even doing it for vibe checking, vibe reviewing.

8. 1:42

   This is the command of Clau-- this is the prompt for the command of Claude Code for security review. It was hyped like two months ago. Do you know what I'm talking about?

9. 1:52

   Now it says there, I don't know if you see it, uh, "You are a senior security engineer." Good. And then like somewhere there, uh, down the line, it says, "Please exclude denial of service."

10. 2:04

    Don't, don't, uh, catch denial of service issues. Maybe that's part of the, [chuckles] part of the reason like we're, we're having, uh, cloud outages. Probably not just that, but you get the point.

11. 2:15

    Like, we need to be rigorous about how we deal with quality. It's not just like vibe quality or, or so, like we're doing vibe coding sometimes. Uh, let's go to another example, okay?

12. 2:27

    Cursor, I, I guess like... or, or Copilot, most of you use rules, right? We're gonna talk about it. You invest in code generation. After a while, you understand if you invest, you'll get more out of it.

13. 2:38

    And, uh, we, we asked like a bunch of, of developers, and I'm asking you as well, think, think for a second for all the developers there in, in the audience, like, when you write Cursor rules or Copilot rules, et cetera, do you feel they're completely followed or it's like mostly followed?

14. 2:54

    Do you know how much they are followed? And what extent are they followed? It's rigorously, like how technical deep they're, they're being followed. So the... what we get back, like the answer from what you see here on the screen is mostly like B, C, and D.

15. 3:07

    They are followed, but they're not completely followed. Okay? So that means like we are generating code, trying to push it to the standards, but it not necessarily still like getting to the quality we wanted.

16. 3:19

    I'm gonna share a bit more statistics and, and information and some insight from three reports. One done by Qodo, uh, another by... done by Sonar, another by Pharos, and all of them are, are focused on code, code quality, review, et cetera.

17. 3:35

    The sample size is thousands of developers, in some cases even more, millions of pull request, and, and a billion of, of lines, lines of code that were, uh, uh, were being checked.

18. 3:46

    Like, for example, if you think about, uh, Sonar, this is a company, yeah, that like coming from pre-AI, but they see code at scale, and you-- they're doing like a lot of, uh, checks in code that are not necessarily AI-focused, but are necessary in order to check, uh, your, your software from all possible direction, and that's why

19. 4:09

    they're scaling and the scale of the code that they're seeing is, is immense. Okay? So for example, we took information from, from their report, and eventually my purpose here is to break down the different dimension of what, uh, code quality means and give you some-- share some stats and, and insights.

20. 4:25

    I want to start with the end. [chuckles] Okay? This is the takeaway I want you all, all, like to take from, from the next thirteen minutes that I have. We started with code generation.

21. 4:38

    We, like out of the box, use it for autocomplete, et cetera, and you invest in it, and you can get more out of it. But there is a glass ceiling for how much productivity you can get from code generation.

22. 4:49

    And then we move to the agentic code generation, right? Uh, let's call it Gen two point oh. And that's a higher glass ceiling. It could do much more productivity, and especially if you invest in it, for example, rules, et cetera.

23. 5:02

    Then with AI breaking outside of the IDE, we can start using AI also for code-- for Agentic quality workflows. It could be inside the IDE, but the, the truth is that if you think about all the workflows you have in your organization, especially if you're more than one hundred developers or so, you probably have a lot of

24. 5:23

    workflow that you... are related to quality that you need to auto-automate. And that's where you start like, uh, breaking through the glass ceiling of, of productivity if you invest in it.

25. 5:33

    And finally, I, I claim that you need those agentic workflows, keep learning, and we might touch a little bit of that like l-later, later on. Okay? Like, because quality is something dynamic, so you'll only finally break, break the glass ceiling if, if you really have those quality workflows and rules and standard being, being dynamic.

26. 5:54

    And then, then you will see the promised two X, let alone the ten X that you were promised the hype and, and you, you heard from McKinsey and from Stanford, you're not getting that.

27. 6:03

    I don't need to tell you the two X, ten X for the entire software development, uh, lifecycle.

28. 6:09

    So a bit about more about, uh, the market adoption. Uh, one of the report say that, uh, 82% of, uh, adoption already for AI, uh, dev tools are being used daily or weekly.

29. 6:21

    Uh, some people, at 60 per- 60%, fifty-nine report that they're using more than three, and 20% saying they're using more than five code generation tools. If you think about it for a second, uh, don't only take like Cursor, Copilot, Codex, uh, Claude Code, et cetera, sorry from insulting anyone in the, in the...

30. 6:37

    That I forgot their tool, but there's also the lovable et cetera. They also generate code, and by the way, you're gonna get to 10. I'm-- Count on me, you're gonna get to 10 tools in two, three years that generate code for you, okay?

31. 6:49

    Come to talk to me about it later, I-I'll try to convince you. And, and the thing is that it-it's coming from bottom up, like 50% of the usage is coming from less than 10 of...

32. 6:58

    A team that are less than 10 developers, but it is propagating also to the enterprise. Again, I'm sure you know, I mean, talk-- propagating to the enterprise at, at scale, like not just like five developers.

33. 7:08

    In the last year, we're seeing like more and more enterprise using co-code generation. Uh, so, uh, and if like, um, an average w-within reports, we saw 82 to 92% using weekly to a monthly, uh, code generation tools.

34. 7:23

    And in some cases, maybe extreme, maybe not, we're gonna talk about it, we saw 3X productivity boost in writing code, okay? But that doesn't mean that if you have, uh, 3X productivity in writing code, that you actually guarantee any quality like I presented before.

35. 7:40

    So actually, 67% of the developer that we asked, asked have serious quality concerns about all the AI generated-- all the generated code, uh, uh, like, uh, code generated by AI or influenced by AI.

36. 7:54

    And they're claiming that are missing the framework, how to deal with quality, how to measure quality. It's a big question. What is quality? I'm gonna talk about it in the next few slides, okay?

37. 8:04

    Think about it for a second before I bre-break it down. What, what is quality? Um, so what we're actually saying that the crisis with vibe coding, uh, viable coding, we're seeing it shifting and, and evolving, is that you're getting like more task being done, like 20 some report, 20% more task, you know, velocity.

38. 8:24

    And like ninety-seven or more percent or so of PR is being opened. And eventually it takes more time to review PR, like 90% more time to review PR. And by the way, like there's a lot of statistics about AI generating code.

39. 8:39

    A-at least there's not less amount of bugs per line of code. I'm not claiming that there are more, but even if there's not less bugs per line of code, you have much more bugs because there are much more PRs, much more code being generated, et cetera, right?

40. 8:53

    So that, that's a problem for the reviewer. So it's somebody surprised it takes more time to review these, especially in the age of agents, right? When five minutes calling to Claude Code, I have one thousand line of code after five minutes.

41. 9:05

    Once upon a time, it took me like hours to write ten proper line of code, right? Now, let's zoom out for a second. Code generation is magnificent, okay? Like it, it's a game changer when you're talking about greenfield.

42. 9:18

    You saw, uh, people talk about it a few slides, a few, uh, other minutes before me. Uh, it, it revolutionized how we do p- proof of concept, uh, project, et cetera.

43. 9:28

    But when you're dealing with heavy-duty software, then you, you like it or not, we are dealing with a lot of things when, uh, w-when you serve millions of clients, you have financial transactions.

44. 9:40

    When you're doing transportation, you're dealing with code integrity, if you like code governance, uh, review standards, testing, reliability, et cetera. That's what we need to, uh, uh, to deal with.

45. 9:51

    Now, let's break that under the surface part of the glacier into two dimensions. This is one dimension. You can look on the qua-quality issues, uh, in-- throughout the software development lifecycle, like planning and then development, writing code, code review.

46. 10:08

    Code review is a bit of a process, but like what you're s- like checking quality, that's part of the process of code review. Testing, which is another part of, of quality, and, and deployment, and I know I didn't cover the entire like, uh, uh, software development lifecycle, but just to give an example.

47. 10:25

    And each one of them like possess, like introduce new problems that are coming because you're using more and more AI-generated code. Um, now another dimension to look at it is actually code-level problems and process-level problems, okay?

48. 10:40

    I'm not, I'm not opening the, you know, lists of functional, just opening the list of non-functional. You're talking about security, inefficiency that are not necessarily, uh, functional. Use-- I'll show you some statistics about that.

49. 10:55

    And then process level is, for example, learning. Hey, if you will have

50. 11:02

    a, a, a bad outage because of AI-generated code, who is responsible? Is it the AI or, or the team that own that, okay? Like you need to learn and own the code eventually.

51. 11:13

    That's a process that n-needs to be done. Verification, porting, guardrails, standards, uh, et cetera. So, so, so all of those issues, when they're introduced to thousands of developer that we ask them, do you think like actually AI helped to reduce with those problems or, or actually made m-more like more challenging?

52. 11:34

    Forty-two, uh, people reported that they spend forty-two more of the development time on solving issues, on fixing bugs, et cetera, and, and they saw thirty-five, uh, uh, percent project delays.

53. 11:48

    We're talking about, we're talking about m-maybe gains, they're talking about like delays. Okay, there's some bias. We told them, we talked about problem with quality and what's the impact, e-et cetera.

54. 11:58

    Um, but that's what they, they sh- they present, uh, to... When they, they answer, uh, when, when you're talking about like when you're mess- using AI code-- AI-generated code.

55. 12:08

    And we see reports, uh, some of the reports talking about three X more security incide- incidents. By the way, it makes sense. You remember we had a slide saying three X more writing code, [chuckles] so three X more security incidents, like the same amount of line of code, the same amount of, uh, uh, problems, the correlation.

56. 12:23

    So what to do with that? Like, I talk about problems and problems and problems. Okay. Help, help me deal with it. Uh, like, let's, let's spend a few minutes on, on that.

57. 12:32

    So one, one suspect, of course, is testing. And actually, really interesting, we ask a couple of question about testing, and one really relevant saying is that people said that when they heavily use AI to...

58. 12:46

    on testing, use AI to do testing, they actually double their trust in the AI-generated code. Okay, that's one thing. The ne- next suspect to help us with the quality is code review.

59. 12:59

    What really interesting about code review, that it's a process that helps almost with all the process level and the code level, like, issues. For example, you can set your AI code review tool to tell you, uh, block this PR if it doesn't cover certain level of test coverage.

60. 13:17

    So through the PR, you take care of the testing process problem, okay? So code, like code review with AI is actually one of, one of the major things you, you, you can do.

61. 13:29

    And people that are, uh, developers that are using AI code review tool, they're saying that they're seeing, they're seeing double the quality gain. And they're saying that actually it's- it helps them to, uh, uh, im-impr-impr-improve forty-seven percent in productivity of writing code, okay?

62. 13:47

    Now, a bit statistics from our own, uh, AI code review tool. We scan a million of PR's a month, and we took one mill- million of those PR's, and we noticed that seventeen percent include like high-severity issues.

63. 14:01

    By the way, we're now analyzing, uh, before and after using AI. I don't have that statistics yet. But we're noticing since we're starting, uh, most of the companies we serve, they use AI-generated code.

64. 14:11

    So that's why, uh, I don't have before. We need to go scan backwards. Uh, and that's like a, a really big, a big number. Another thing I wanted to talk to you like about, uh, when you're trying to improve on quality is, is the foundation of having the right context that is brought to the, uh, code generation

65. 14:30

    tool, that is brought to the AI code review tool. Better context, better quality across the board, wherever you're using AI. Uh, so when we ask developers when, when you ha- when you don't trust AI-generated code, like you remember like [REDACTED:age]-seven percent, uh, said, uh, like are really worried about that.

66. 14:48

    They said eighty, eighty percent of the time, they don't trust the context that the LLM have, okay? And, and, and, uh, when we ask developers, "What would you like to be improved in your AI-generated code, in your AI code review tool?"

67. 15:02

    They said the number one was context. It was number one at thirty-three percent. They can choose like among many things to, to improve. So context is extremely important. I can tell you that at Qodo, one of our technology moats, uh, is, is around context, and when you connect our context engine, we're seeing it as the number one

68. 15:20

    tool that is being used. Like [REDACTED:age] percent of code generator or code review tools, [REDACTED:age] percent of their calls to an MCP would be to a context MCP, okay?

69. 15:31

    And just to tell you, the context doesn't necessarily need to include only your code. It could also include context to your standards, to your best practices. We're seeing in our AI code review that eight percent of the context usage, usage is actually from files that are related to standards and, and best practices, et cetera.

70. 15:49

    Okay, I have to, uh, CEO of Qodo, like marketing will be mad at me if I don't brag a little bit, right? So this is, uh, our kind of like our architecture of our context engine being presented by Jensen on GTC keynotes.

71. 16:01

    And he-- notice he didn't talk about our

72. 16:04

    co-co- code review capabilities, about our testing capabilities. He talked about our context engine that NVIDIA checked because there is a realization that AI quality, AI-generated, whatever review testing will come from bringing the right context.

73. 16:17

    So invest in that. You need to, to build your context, buy a solution and invest in it, build your solution, uh, et cetera. And the context needs to include code, uh, uh, versioning, peer history, uh, organization logs, et cetera.

74. 16:33

    That's where all the context sits. It's not just in the last branch of your code base, okay? So I'm, I'm zooming out, starting to talk about like recommendations and, uh, and like, uh, the takeaways.

75. 16:46

    So what, what, what's next? So automated g- uh, quality gateways, invest in that. People talked throughout the morning about parallel agents. You know what I'm talking about? Like background agents.

76. 16:58

    You can use a lot of those like tools and capabilities to build, build your quality gates. Uh, use intelligent code review testing, and you need a li- living and breathing like documentation and, and...

77. 17:12

    What documentation means is, is a story by itself. Uh, I'm not gonna double-click on it. And, and this is how I present for three years now, and I think I'm gonna go all the way until age of [REDACTED:age] with this slide, of how I think the future of software development looks like, okay?

78. 17:31

    So basically, you have your specification and you have your code, right? And you have multiple agents, parallel agents, that are helping you to improve your spec, write your spec, improve your code, transfer, transfer from your spec to your, to your code, uh, uh, make tests which are executable specs, right?

79. 17:51

    Uh, and, and then you're gonna have your context engine, the software development database, and you will build your tools, especially MCPs around quality and verification. And you'll make sure you have environments Stable, secured sandboxes where those agents can run and, and run validation and quality, uh, workflows.

80. 18:11

    So don't, don't forget, like the path forward is quality is your competitive edge over your [laughs] competition. AI is a tool. It's not, it's not a solution, okay? And don't like only think about code generation as the only thing.

81. 18:27

    Look on the entire SDLC or product development lifecycle. I saw one of the, uh, uh, people talk to, uh, uh, on s- speakers, and it- iterate with everything we talked about today.

82. 18:39

    I have t- uh, I want to tell you that you will gain value from it. We're seeing in the reports, people seeing like security vulnerability being reduced, faster code review.

83. 18:49

    You... we just got a hit on that because of AI-generated code, and test coverage in a month can, can triple, uh, depends on, on the project, et cetera. With, with the last minute, I wanna show like a really small piece of what you can do with Qodo.

84. 19:03

    Uh, you can go into Qodo and define your own rule. For example, almost the same rule you'll put in Cursor of, "I don't like nested ifs," if this is a problem that you have.

85. 19:14

    But then Qodo will look on your context, build the good example, the bad example, and then start giving, like building a workflow that is specifically to catch that issue and give you statistics over time when it's being accepted and when not, so you can adjust that rule and really know and have visibility to, to your standards, okay?

86. 19:37

    So when an, a PR is written with a few ifs and else, although it was written with Cursor Copilot that had a rule do not do nested ifs, et cetera, then eventually when you open a PR, you'll get, uh, Qodo, uh, uh, catching that and giving a suggestion according to the good and the bad example.

87. 19:56

    Qodo will also make a graph, give you a CLI checks, like check each one of the rules, and eventually tell you the nest- nested if, and then will record and learn what you did or did not do with that suggestion in order to adapt the standard and, uh, of the, of the quality.

88. 20:13

    Um, there were also automated like suggestion. You don't need to write your own. It learns your, your s- your, uh, standards and quality and offer that to you. And that's it.

89. 20:22

    I'm, I'm really, really excited about like breaking the glass ceiling, [laughs] okay, with what we did with code generation and then agentic code generation. Now we're turning into the era of putting AI into work and through the entire SDLC.

90. 20:37

    The most important part is related to quality. You would need to invest in that. It's not out of the box, okay? And then you would see eventually the promised 2x te- [laughs] that, that, that probably, uh, promised to the CEO or something like that once they give you the budget for, for the relevant tools.

91. 20:55

    Thank you so much. [upbeat music]