1 00:00:00,161 --> 00:00:01,161 Morning, everybody. 2 00:00:02,781 --> 00:00:04,341 My name is Tariq Shaukat. 3 00:00:04,421 --> 00:00:06,601 I am the CEO of Sonar, as you just heard. 4 00:00:06,801 --> 00:00:13,361 And we're here to talk today about how we have been helping companies build software 5 00:00:13,361 --> 00:00:15,641 factories that actually work for enterprises. 6 00:00:15,641 --> 00:00:21,701 And I thought it'd be an interesting way to start looking at actual factories, because 7 00:00:21,701 --> 00:00:23,441 we know how to build actual factories. 8 00:00:23,441 --> 00:00:28,281 And for those of you who have a little bit of the gray hair and all of this sort of thing, 9 00:00:28,381 --> 00:00:30,361 you may remember the 1980s. 10 00:00:30,461 --> 00:00:34,161 And in the 1980s, you could find cars that were really good cars. 11 00:00:34,501 --> 00:00:37,761 But if you look at the picture here and if you remember those times, 12 00:00:37,901 --> 00:00:40,301 the cars didn't quite fit together perfectly. 13 00:00:40,941 --> 00:00:44,841 You'd have different pieces that would leave big gaps. 14 00:00:44,841 --> 00:00:49,521 It would be really good cars that just didn't kind of work the way they were supposed to. 15 00:00:49,521 --> 00:01:00,501 And then Toyota, and particularly Lexus, came out with a commercial, and it showed something very simple, which was a ball bearing running across the seams of this car. 16 00:01:00,561 --> 00:01:06,821 And it was kind of revolutionary at the time because it showed that you could actually build a car that fit together perfectly. 17 00:01:07,061 --> 00:01:18,681 And instead of building for volume and building and then fixing the old mindset, you could actually have a mindset focused on ship it right, on no rework, on customer happiness. 18 00:01:18,681 --> 00:01:26,961 and there was not this was not a single change what happened was that people really started 19 00:01:26,961 --> 00:01:33,261 thinking about the factories differently they started engineering building factories that 20 00:01:33,261 --> 00:01:39,341 were engineered to deliver quality and there's a number of different pieces that i think still 21 00:01:39,341 --> 00:01:46,681 resonate today the first is you prioritize quality instead of volume the second is that you fix the 22 00:01:46,681 --> 00:01:54,601 system, not just individual symptoms that you see. The third is that you design for something that 23 00:01:54,601 --> 00:01:59,821 can be built as opposed to designing the wheels and designing the washer bottles and whatever it 24 00:01:59,821 --> 00:02:04,881 is and then hoping it fits together. You solve problems as they occur as opposed to kicking the 25 00:02:04,881 --> 00:02:09,661 can down the road and fixing it at the end. And you empower everybody. Anyone could stop the line, 26 00:02:10,041 --> 00:02:15,701 not just the boss. Now, why am I talking about this? It's actually because modern software 27 00:02:15,701 --> 00:02:20,421 development actually embraced all of these principles. If you think about how we build 28 00:02:20,421 --> 00:02:27,461 software today, what is considered best practice, a lot of the terms and the practices came 29 00:02:27,461 --> 00:02:33,041 from that idea of a Toyota production system. Agile, Kanban, error budgets, if you're in 30 00:02:33,041 --> 00:02:39,741 the SRE world, continuous integration. These ideas all kind of sprung from this notion 31 00:02:39,741 --> 00:02:44,121 of how do we build factories that work. And it's worked really well for us in software 32 00:02:44,121 --> 00:02:48,441 development so far. But agents, as we all know, you wouldn't be sitting here if you 33 00:02:48,441 --> 00:02:53,821 didn't understand that agents are changing the game. And as agents are changing the game, 34 00:02:54,021 --> 00:03:01,721 weird things start to happen. First of all, this is research that was published last month 35 00:03:01,721 --> 00:03:07,381 of the effect of implementing asynchronous agents in your development lifecycle. 36 00:03:07,901 --> 00:03:12,141 And I'm sure many of you have seen this. You have an explosion of the number of lines of 37 00:03:12,141 --> 00:03:19,501 code written up 2,450%, you have an increase in the number of commits. So the volume code 38 00:03:19,501 --> 00:03:25,641 creation is now easy and it is now abundant. And by the way, more and more people, I'm 39 00:03:25,641 --> 00:03:29,821 sure many people in this room are saying, okay, if it's easy and abundant, how do I 40 00:03:29,821 --> 00:03:35,161 automate it? How do I build factories, agentic software factories that help automate and 41 00:03:35,161 --> 00:03:40,321 scale this? And today we just finished our what we call state of code developer survey. 42 00:03:40,981 --> 00:03:45,861 it'll be out next month but this is a little preview today 5% of developers 43 00:03:45,861 --> 00:03:49,181 that we spoke to say that they're running agentic software factories but 44 00:03:49,181 --> 00:03:55,781 34% said in a year they expect to and 39% in financial services so people are 45 00:03:55,781 --> 00:04:01,121 thinking about how do you actually build these next generation factories and 46 00:04:01,121 --> 00:04:05,621 there's a couple of gotchas in here security risk the agents can work really 47 00:04:05,621 --> 00:04:10,081 well to help you build stuff it can also work really well to help you hack stuff 48 00:04:10,081 --> 00:04:13,661 help you attack stuff, find vulnerabilities, find quality issues, etc. 49 00:04:14,361 --> 00:04:19,381 And so as I was writing this presentation kind of over the weekend, 50 00:04:19,661 --> 00:04:22,761 a little just in time, again from the Toyota production system, 51 00:04:23,481 --> 00:04:25,601 Satya Nadella posted this article on X. 52 00:04:25,621 --> 00:04:27,461 And if you haven't read it, I think it's a great article 53 00:04:27,461 --> 00:04:30,161 that I think everyone should internalize. 54 00:04:30,301 --> 00:04:33,621 But he was talking about insider risk of models inside of companies. 55 00:04:34,181 --> 00:04:36,361 And one of the things that he talked about 56 00:04:36,361 --> 00:04:40,401 was the need for an engineering approach to containment and governance 57 00:04:40,401 --> 00:04:43,661 built around strong deterministic system design, 58 00:04:44,061 --> 00:04:46,721 human controls, and reliable operating procedures. 59 00:04:47,241 --> 00:04:50,081 And this really resonated with me 60 00:04:50,081 --> 00:04:54,301 because what we are seeing is a paradigm shift 61 00:04:54,301 --> 00:04:58,301 in software factory design or software development, 62 00:04:58,341 --> 00:05:00,021 let's call it more broadly, that 63 00:05:00,061 --> 00:05:07,361 we need to engineer, we need to build systems that engineer confidence and quality into agentic software factories. 64 00:05:08,441 --> 00:05:22,061 So to start doing that, I think it's important to understand the limitations that we're facing. So what's the hard thing about this? And there's five that I'm going to call out. The first is that human in the loop doesn't scale. 65 00:05:22,061 --> 00:05:24,781 and this may be, I don't know, to this room 66 00:05:24,781 --> 00:05:26,361 it may not be all that controversial 67 00:05:26,361 --> 00:05:29,121 but virtually every company I talk to 68 00:05:29,121 --> 00:05:30,781 I say, what's your AI governance strategy? 69 00:05:30,981 --> 00:05:32,521 And they give me all this stuff and they say 70 00:05:32,521 --> 00:05:34,081 but don't worry, we have humans in the loop 71 00:05:34,081 --> 00:05:36,061 and the problem with humans in the loop 72 00:05:36,061 --> 00:05:40,941 as I mentioned earlier, 2,450% increase in lines of code 73 00:05:41,681 --> 00:05:43,621 240% increase in commits 74 00:05:43,621 --> 00:05:48,521 but the same study showed only a 30% increase in releases 75 00:05:48,521 --> 00:05:50,321 we haven't figured out how to scale 76 00:05:50,321 --> 00:05:52,941 from producing stuff to shipping stuff. 77 00:05:54,401 --> 00:05:56,641 There's automation needed, there's new tools needed, 78 00:05:56,641 --> 00:06:01,421 and really importantly, this is a concept that really, 79 00:06:01,661 --> 00:06:05,001 I see it every day now, and I'm sure many of you do as well, 80 00:06:05,441 --> 00:06:06,781 called cognitive surrender. 81 00:06:07,381 --> 00:06:09,501 And this study was done just a couple months ago, 82 00:06:09,581 --> 00:06:14,401 and it looked at how often do humans agree with AI. 83 00:06:14,401 --> 00:06:18,841 And it turns out they agree with AI when AI is right 93% of the time. 84 00:06:20,321 --> 00:06:23,361 And four out of five times they agree with it when the AI is wrong. 85 00:06:24,701 --> 00:06:29,401 So it's not that human in the loop is, we just need to make things faster. 86 00:06:29,481 --> 00:06:34,081 It's actually that it doesn't work in the agentic era. 87 00:06:34,821 --> 00:06:35,961 So that's problem one. 88 00:06:36,061 --> 00:06:40,501 Problem two is that agents struggle with large complex code bases. 89 00:06:40,781 --> 00:06:47,901 They are very, very smart, but you need to understand what happens in a code base in order to work on it. 90 00:06:47,901 --> 00:06:50,961 The left-hand side is a study that was done on code translation. 91 00:06:51,241 --> 00:06:56,801 And what it shows is that the agents were able to get it right 91% of the time 92 00:06:56,801 --> 00:07:00,901 when the code base was 10,000 lines of code. 93 00:07:01,541 --> 00:07:04,701 And by the time it got to even 50,000, 94 00:07:04,761 --> 00:07:07,961 and I'm sure everyone in this room operates off of much bigger code bases than this, 95 00:07:08,181 --> 00:07:11,661 but when it was trying to translate 50,000, the accuracy collapsed. 96 00:07:13,001 --> 00:07:19,561 Similarly, another study looked at how many tokens does it take on a large complex codebase 97 00:07:19,561 --> 00:07:20,921 to complete the same task. 98 00:07:21,101 --> 00:07:23,001 And the variance was 30x. 99 00:07:23,161 --> 00:07:25,141 Not 30%, but 30x. 100 00:07:25,301 --> 00:07:31,161 And the reason is because the agent spends a huge amount of their tokens trying to understand 101 00:07:31,161 --> 00:07:34,761 how to navigate the codebase, the structure of the codebase, things like this. 102 00:07:34,761 --> 00:07:43,321 And it's kind of like if I'm trying to go from L.A. to here, and if I pick a random starting point. 103 00:07:43,361 --> 00:07:46,741 If I happen to pick White Plains, I will get to New York pretty quickly. 104 00:07:46,821 --> 00:07:49,801 If I pick Albuquerque, it's going to take me a long time and a lot of cost. 105 00:07:50,041 --> 00:07:52,161 Same thing happens with agents. 106 00:07:53,861 --> 00:07:54,661 Errors compound. 107 00:07:55,281 --> 00:07:57,641 This is another important concept. 108 00:07:57,821 --> 00:07:59,641 This is a study out of Meter. 109 00:08:00,481 --> 00:08:03,841 And Meter looks at, and you can find this on their website, 110 00:08:03,841 --> 00:08:10,261 they look at how effective are agents at completing tasks that humans can do. 111 00:08:10,501 --> 00:08:15,081 And the way they look at this is can you complete a task that takes X number of hours. 112 00:08:15,201 --> 00:08:19,541 So you can see here the Claude Mythos preview, which is the latest version that they've published, 113 00:08:20,061 --> 00:08:23,441 was good at completing tasks that take a human 16 hours. 114 00:08:24,221 --> 00:08:25,541 So that's really good. 115 00:08:25,541 --> 00:08:31,341 But there's an important caveat in the fine print here, which is this is at 50% accuracy. 116 00:08:31,341 --> 00:08:38,201 When you dial up the accuracy to 80%, the curve flattens substantially. 117 00:08:38,201 --> 00:08:41,901 Because as you do long-running tasks, errors compound. 118 00:08:42,101 --> 00:08:49,861 If you have 95% accuracy on each task, by the time you're at 20 steps, you have 36% accuracy. 119 00:08:50,681 --> 00:08:55,901 And so that is another issue we have to deal with in software factories. 120 00:08:55,901 --> 00:09:01,661 probably the most aggressive headline I put in here is is this fourth point that LLMs are 121 00:09:01,661 --> 00:09:07,561 unreliable reward-seeking judges you know so the question that we always get is well can't you just 122 00:09:07,561 --> 00:09:13,381 ask an LLM and so people have been studying this as well and this is all again the first on the 123 00:09:13,381 --> 00:09:20,561 left-hand side is from PayPal AI's research lab they published a paper this month actually and 124 00:09:20,561 --> 00:09:23,201 and found that depending on the model and the situation, 125 00:09:23,381 --> 00:09:29,781 between 7% and 17% of verdicts change across what they call identical replications. 126 00:09:30,041 --> 00:09:34,361 Ask the same question, get a different answer between 7% and 17% of the time. 127 00:09:35,201 --> 00:09:39,021 And if you start challenging, this is a study out of Meta now, 128 00:09:39,301 --> 00:09:42,001 if you start challenging the LLM, are you sure? 129 00:09:42,241 --> 00:09:43,201 Very simple challenge. 130 00:09:43,401 --> 00:09:48,161 It'll actually change its mind 7% to 34%, again, depending on the model. 131 00:09:48,161 --> 00:09:52,641 So you ask the AI judge twice and you may get a different answer. 132 00:09:53,721 --> 00:09:59,001 And then lastly, and I won't dig into this one too much because I think it's probably self-evident to everyone, 133 00:09:59,201 --> 00:10:05,981 as a time to exploit near zero, the ability for us to respond is incredibly challenged. 134 00:10:06,181 --> 00:10:14,361 And so prevention is something that we believe the software factory has to bake in to the design, has to engineer into the factory. 135 00:10:15,221 --> 00:10:18,301 So as we've thought through this, 136 00:10:19,221 --> 00:10:24,241 we have written down what we call the code confidence formula, 137 00:10:24,241 --> 00:10:26,021 which is basically how do you build, 138 00:10:26,221 --> 00:10:28,921 how do you think about a software factory 139 00:10:28,921 --> 00:10:30,461 that delivers what we all want, 140 00:10:30,481 --> 00:10:34,201 which is outcomes that we can feel confident in. 141 00:10:34,401 --> 00:10:36,021 Not outcomes that we can trust, 142 00:10:36,261 --> 00:10:37,981 not blind trust or anything like that, 143 00:10:38,101 --> 00:10:40,001 but outcomes that we can feel comfortable with. 144 00:10:40,081 --> 00:10:43,501 We can sit and tell our executives, our customers, 145 00:10:43,501 --> 00:10:48,501 our shareholders, our CEOs, our boards, that yes, I believe this factory is working well. 146 00:10:48,701 --> 00:10:56,441 And we think there's four components to this. The first is being crystal clear about the intent. 147 00:10:57,101 --> 00:11:00,561 If you don't know what you're trying to build, you can't tell the agent what to build, 148 00:11:00,561 --> 00:11:06,801 and you will not get the outcome you want. The second is guide rails. And we very specifically 149 00:11:06,801 --> 00:11:12,801 talk about guide rails, not guard rails. Guard rails prevent something from going off the road. 150 00:11:12,801 --> 00:11:16,681 guide rails help keep you on it and I think what agents need are the small 151 00:11:16,681 --> 00:11:22,981 nudges to guide and govern those agents the third part is verification that needs 152 00:11:22,981 --> 00:11:28,201 to be independent thorough and issues fixed as opposed to just cans kicked 153 00:11:28,201 --> 00:11:34,321 down the road and then lastly evidence trust but verify trust but understand 154 00:11:34,321 --> 00:11:39,481 that it was done with proof that is consistent auditable and repeatable and 155 00:11:39,481 --> 00:11:44,781 And we multiply these together because we've seen a number of situations where one of these 156 00:11:44,781 --> 00:11:45,621 elements is missing. 157 00:11:45,901 --> 00:11:48,281 And if it's zero, the confidence goes to zero. 158 00:11:48,421 --> 00:11:52,421 If you do all of this but don't have evidence, you go to zero. 159 00:11:52,421 --> 00:11:54,701 You don't have confidence in the outcome. 160 00:11:55,141 --> 00:12:01,341 So we really think about factory design as requiring these four elements. 161 00:12:02,201 --> 00:12:06,481 And as we put together factories, as we help customers, I'm not going to walk through every 162 00:12:06,481 --> 00:12:12,101 box on this page but as we help our customers and partners build these software factories 163 00:12:12,101 --> 00:12:16,421 we're very deliberate about how do you get the intent bottom left hand side in purple 164 00:12:16,421 --> 00:12:22,361 how do you put the guardrails in place how do you do verification in a way that's thorough and 165 00:12:22,361 --> 00:12:29,041 independent and how do you produce the evidence and by doing this we see amazing results so these 166 00:12:29,041 --> 00:12:35,461 are real results now from our from our customers the first one is focusing on intent and guide 167 00:12:35,461 --> 00:12:40,541 rails and intent in guide rails we have a product called sonar vortex which gives intent guide 168 00:12:40,541 --> 00:12:45,701 rails and in the agentic loop starts to do verifications again nudging the agents and what 169 00:12:45,701 --> 00:12:52,961 we see when we provide agents in long running in large complex code bases with long running tasks 170 00:12:52,961 --> 00:13:00,401 with this information is a 36 percent reduction in token cost because you're putting it in white 171 00:13:00,401 --> 00:13:03,721 planes you actually you're putting it close to the answer you're helping it understand 172 00:13:03,721 --> 00:13:10,781 the map. You don't have to burn tokens on things we already know. And 92% reduction in issues in 173 00:13:10,781 --> 00:13:17,441 the commit. So it's cheaper and we can get a better outcome to go into that verification stage. 174 00:13:17,641 --> 00:13:21,561 This is again all because we're nudging the agent to keep on track. 175 00:13:22,921 --> 00:13:29,621 Then you get to verification. And we started life as a verification company. This is how we see the 176 00:13:29,621 --> 00:13:33,261 world is through a verification lens. And we have a very strong point of view that independent 177 00:13:33,261 --> 00:13:41,161 verification has to cover the full range of issues that is both precise and cost effective. 178 00:13:41,421 --> 00:13:49,301 Use CPUs when you can use CPUs, use GPUs when you need GPUs. So we look at deterministic mixture of 179 00:13:49,301 --> 00:13:54,601 methods, and then you add in agentic mixture of methods, and it comes up with what we think is 180 00:13:54,601 --> 00:14:00,801 the gold standard of verification. And what we see is when you apply this, our product is called 181 00:14:00,801 --> 00:14:06,241 SonarQube. So we talked to our customers, over a thousand customers recently, and we said, 182 00:14:08,681 --> 00:14:16,201 how likely are you or how often have you seen these issues from Agenda coding? And the statement 183 00:14:16,201 --> 00:14:21,161 to read here is SonarQube users are X percent more likely than non-users to have seen a very 184 00:14:21,161 --> 00:14:27,941 positive impact. So 81 percent more likely to have seen, to believe that AI coding has had a very 185 00:14:27,941 --> 00:14:34,001 positive impact on code security if you have that gold standard for verification. 186 00:14:34,561 --> 00:14:42,321 34% more likely to have seen an impact on time to market, 76% on outage frequency. 187 00:14:43,461 --> 00:14:48,781 So again, as I come back to how do we have confidence to ship, what we're seeing is 188 00:14:48,781 --> 00:14:54,701 that if you put these pieces together and you build verification incredibly strongly 189 00:14:54,701 --> 00:14:57,001 as a first-class citizen in the factory, 190 00:14:57,261 --> 00:14:59,141 you are actually building confidence 191 00:14:59,141 --> 00:14:59,801 and you're seeing 192 00:15:00,021 --> 00:15:09,681 results. Better security, more confidence in our time to market, improved or a reduction in number of outages, improved reliability. 193 00:15:10,541 --> 00:15:24,761 And of course, as I mentioned, you have to do this with evidence. Make sure you pass the quality gate. Make sure whatever, there's lots of acronyms up here, whichever ones you need to pass, whether that is your IT general controls inside of your 194 00:15:25,521 --> 00:15:29,161 internal controls and operations, whether that's MISRA if you're building cars, 195 00:15:29,421 --> 00:15:34,581 or any of the other ones that are on here, producing the evidence is what lets you sleep at night. 196 00:15:34,801 --> 00:15:37,441 It's what gives you confidence that you can move forward. 197 00:15:38,401 --> 00:15:43,541 So that, in essence, is how we think about the challenge of the future, that code 198 00:15:43,541 --> 00:15:47,861 is abundant, that confidence is scarce, 199 00:15:48,781 --> 00:15:52,441 and really the goal that we have and that we think 200 00:15:52,441 --> 00:15:58,501 that everyone should have as we think about the future of software development is how do we deliver 201 00:15:58,501 --> 00:16:04,501 confidence and the factories of the future need to be able to deliver confidence. So with that, 202 00:16:04,761 --> 00:16:08,561 I want to say thank you to everybody. We have a big booth. It's a big red booth 203 00:16:09,381 --> 00:16:14,981 on the floor right over that way and would love to talk to you more about all of this. 204 00:16:15,221 --> 00:16:16,401 Thank you very much. Have a great morning.