Machine transcription with reviewed terminology and names. Some recognition errors may remain. Timestamps refer to the original livestream.
0:35:00Morning, everybody.
0:35:02My name is Tariq Shaukat.
0:35:04I am the CEO of Sonar, as you just heard.
0:35:06And we're here to talk today about how we have been helping companies build software
0:35:13factories that actually work for enterprises.
0:35:15And I thought it'd be an interesting way to start looking at actual factories, because
0:35:21we know how to build actual factories.
0:35:23And for those of you who have a little bit of the gray hair and all of this sort of thing,
0:35:28you may remember the 1980s.
0:35:30And in the 1980s, you could find cars that were really good cars.
0:35:34But if you look at the picture here and if you remember those times,
0:35:37the cars didn't quite fit together perfectly.
0:35:40You'd have different pieces that would leave big gaps.
0:35:44It would be really good cars that just didn't kind of work the way they were supposed to.
0:35:49And then Toyota, and particularly Lexus, came out with a commercial, and it showed something very simple, which was a ball bearing running across the seams of this car.
0:36:00And it was kind of revolutionary at the time because it showed that you could actually build a car that fit together perfectly.
0:36:07And instead of building for volume and building and then fixing the old mindset, you could actually have a mindset focused on ship it right, on no rework, on customer happiness.
0:36:18and there was not this was not a single change what happened was that people really started
0:36:26thinking about the factories differently they started engineering building factories that
0:36:33were engineered to deliver quality and there's a number of different pieces that i think still
0:36:39resonate today the first is you prioritize quality instead of volume the second is that you fix the
0:36:46system, not just individual symptoms that you see. The third is that you design for something that
0:36:54can be built as opposed to designing the wheels and designing the washer bottles and whatever it
0:36:59is and then hoping it fits together. You solve problems as they occur as opposed to kicking the
0:37:04can down the road and fixing it at the end. And you empower everybody. Anyone could stop the line,
0:37:10not just the boss. Now, why am I talking about this? It's actually because modern software
0:37:15development actually embraced all of these principles. If you think about how we build
0:37:20software today, what is considered best practice, a lot of the terms and the practices came
0:37:27from that idea of a Toyota production system. Agile, Kanban, error budgets, if you're in
0:37:33the SRE world, continuous integration. These ideas all kind of sprung from this notion
0:37:39of how do we build factories that work. And it's worked really well for us in software
0:37:44development so far. But agents, as we all know, you wouldn't be sitting here if you
0:37:48didn't understand that agents are changing the game. And as agents are changing the game,
0:37:54weird things start to happen. First of all, this is research that was published last month
0:38:01of the effect of implementing asynchronous agents in your development lifecycle.
0:38:07And I'm sure many of you have seen this. You have an explosion of the number of lines of
0:38:12code written up 2,450%, you have an increase in the number of commits. So the volume code
0:38:19creation is now easy and it is now abundant. And by the way, more and more people, I'm
0:38:25sure many people in this room are saying, okay, if it's easy and abundant, how do I
0:38:29automate it? How do I build factories, agentic software factories that help automate and
0:38:35scale this? And today we just finished our what we call state of code developer survey.
0:38:40it'll be out next month but this is a little preview today 5% of developers
0:38:45that we spoke to say that they're running agentic software factories but
0:38:4934% said in a year they expect to and 39% in financial services so people are
0:38:55thinking about how do you actually build these next generation factories and
0:39:01there's a couple of gotchas in here security risk the agents can work really
0:39:05well to help you build stuff it can also work really well to help you hack stuff
0:39:10help you attack stuff, find vulnerabilities, find quality issues, etc.
0:39:14And so as I was writing this presentation kind of over the weekend,
0:39:19a little just in time, again from the Toyota production system,
0:39:23Satya Nadella posted this article on X.
0:39:25And if you haven't read it, I think it's a great article
0:39:27that I think everyone should internalize.
0:39:30But he was talking about insider risk of models inside of companies.
0:39:34And one of the things that he talked about
0:39:36was the need for an engineering approach to containment and governance
0:39:40built around strong deterministic system design,
0:39:44human controls, and reliable operating procedures.
0:39:47And this really resonated with me
0:39:50because what we are seeing is a paradigm shift
0:39:54in software factory design or software development,
0:39:58let's call it more broadly, that
0:40:00we need to engineer, we need to build systems that engineer confidence and quality into agentic software factories.
0:40:08So to start doing that, I think it's important to understand the limitations that we're facing. So what's the hard thing about this? And there's five that I'm going to call out. The first is that human in the loop doesn't scale.
0:40:22and this may be, I don't know, to this room
0:40:24it may not be all that controversial
0:40:26but virtually every company I talk to
0:40:29I say, what's your AI governance strategy?
0:40:30And they give me all this stuff and they say
0:40:32but don't worry, we have humans in the loop
0:40:34and the problem with humans in the loop
0:40:36as I mentioned earlier, 2,450% increase in lines of code
0:40:41240% increase in commits
0:40:43but the same study showed only a 30% increase in releases
0:40:48we haven't figured out how to scale
0:40:50from producing stuff to shipping stuff.
0:40:54There's automation needed, there's new tools needed,
0:40:56and really importantly, this is a concept that really,
0:41:01I see it every day now, and I'm sure many of you do as well,
0:41:05called cognitive surrender.
0:41:07And this study was done just a couple months ago,
0:41:09and it looked at how often do humans agree with AI.
0:41:14And it turns out they agree with AI when AI is right 93% of the time.
0:41:20And four out of five times they agree with it when the AI is wrong.
0:41:24So it's not that human in the loop is, we just need to make things faster.
0:41:29It's actually that it doesn't work in the agentic era.
0:41:34So that's problem one.
0:41:36Problem two is that agents struggle with large complex code bases.
0:41:40They are very, very smart, but you need to understand what happens in a code base in order to work on it.
0:41:47The left-hand side is a study that was done on code translation.
0:41:51And what it shows is that the agents were able to get it right 91% of the time
0:41:56when the code base was 10,000 lines of code.
0:42:01And by the time it got to even 50,000,
0:42:04and I'm sure everyone in this room operates off of much bigger code bases than this,
0:42:08but when it was trying to translate 50,000, the accuracy collapsed.
0:42:12Similarly, another study looked at how many tokens does it take on a large complex codebase
0:42:19to complete the same task.
0:42:21And the variance was 30x.
0:42:23Not 30%, but 30x.
0:42:25And the reason is because the agent spends a huge amount of their tokens trying to understand
0:42:31how to navigate the codebase, the structure of the codebase, things like this.
0:42:34And it's kind of like if I'm trying to go from L.A. to here, and if I pick a random starting point.
0:42:43If I happen to pick White Plains, I will get to New York pretty quickly.
0:42:46If I pick Albuquerque, it's going to take me a long time and a lot of cost.
0:42:50Same thing happens with agents.
0:42:53Errors compound.
0:42:55This is another important concept.
0:42:57This is a study out of Meter.
0:43:00And Meter looks at, and you can find this on their website,
0:43:03they look at how effective are agents at completing tasks that humans can do.
0:43:10And the way they look at this is can you complete a task that takes X number of hours.
0:43:15So you can see here the Claude Mythos preview, which is the latest version that they've published,
0:43:20was good at completing tasks that take a human 16 hours.
0:43:24So that's really good.
0:43:25But there's an important caveat in the fine print here, which is this is at 50% accuracy.
0:43:31When you dial up the accuracy to 80%, the curve flattens substantially.
0:43:38Because as you do long-running tasks, errors compound.
0:43:42If you have 95% accuracy on each task, by the time you're at 20 steps, you have 36% accuracy.
0:43:50And so that is another issue we have to deal with in software factories.
0:43:55probably the most aggressive headline I put in here is is this fourth point that LLMs are
0:44:01unreliable reward-seeking judges you know so the question that we always get is well can't you just
0:44:07ask an LLM and so people have been studying this as well and this is all again the first on the
0:44:13left-hand side is from PayPal AI's research lab they published a paper this month actually and
0:44:20and found that depending on the model and the situation,
0:44:23between 7% and 17% of verdicts change across what they call identical replications.
0:44:30Ask the same question, get a different answer between 7% and 17% of the time.
0:44:35And if you start challenging, this is a study out of Meta now,
0:44:39if you start challenging the LLM, are you sure?
0:44:42Very simple challenge.
0:44:43It'll actually change its mind 7% to 34%, again, depending on the model.
0:44:48So you ask the AI judge twice and you may get a different answer.
0:44:53And then lastly, and I won't dig into this one too much because I think it's probably self-evident to everyone,
0:44:59as a time to exploit near zero, the ability for us to respond is incredibly challenged.
0:45:06And so prevention is something that we believe the software factory has to bake in to the design, has to engineer into the factory.
0:45:15So as we've thought through this,
0:45:19we have written down what we call the code confidence formula,
0:45:24which is basically how do you build,
0:45:26how do you think about a software factory
0:45:28that delivers what we all want,
0:45:30which is outcomes that we can feel confident in.
0:45:34Not outcomes that we can trust,
0:45:36not blind trust or anything like that,
0:45:38but outcomes that we can feel comfortable with.
0:45:40We can sit and tell our executives, our customers,
0:45:43our shareholders, our CEOs, our boards, that yes, I believe this factory is working well.
0:45:48And we think there's four components to this. The first is being crystal clear about the intent.
0:45:57If you don't know what you're trying to build, you can't tell the agent what to build,
0:46:00and you will not get the outcome you want. The second is guide rails. And we very specifically
0:46:06talk about guide rails, not guard rails. Guard rails prevent something from going off the road.
0:46:12guide rails help keep you on it and I think what agents need are the small
0:46:16nudges to guide and govern those agents the third part is verification that needs
0:46:22to be independent thorough and issues fixed as opposed to just cans kicked
0:46:28down the road and then lastly evidence trust but verify trust but understand
0:46:34that it was done with proof that is consistent auditable and repeatable and
0:46:39And we multiply these together because we've seen a number of situations where one of these
0:46:44elements is missing.
0:46:45And if it's zero, the confidence goes to zero.
0:46:48If you do all of this but don't have evidence, you go to zero.
0:46:52You don't have confidence in the outcome.
0:46:55So we really think about factory design as requiring these four elements.
0:47:02And as we put together factories, as we help customers, I'm not going to walk through every
0:47:06box on this page but as we help our customers and partners build these software factories
0:47:12we're very deliberate about how do you get the intent bottom left hand side in purple
0:47:16how do you put the guardrails in place how do you do verification in a way that's thorough and
0:47:22independent and how do you produce the evidence and by doing this we see amazing results so these
0:47:29are real results now from our from our customers the first one is focusing on intent and guide
0:47:35rails and intent in guide rails we have a product called sonar vortex which gives intent guide
0:47:40rails and in the agentic loop starts to do verifications again nudging the agents and what
0:47:45we see when we provide agents in long running in large complex code bases with long running tasks
0:47:52with this information is a 36 percent reduction in token cost because you're putting it in white
0:48:00planes you actually you're putting it close to the answer you're helping it understand
0:48:03the map. You don't have to burn tokens on things we already know. And 92% reduction in issues in
0:48:10the commit. So it's cheaper and we can get a better outcome to go into that verification stage.
0:48:17This is again all because we're nudging the agent to keep on track.
0:48:22Then you get to verification. And we started life as a verification company. This is how we see the
0:48:29world is through a verification lens. And we have a very strong point of view that independent
0:48:33verification has to cover the full range of issues that is both precise and cost effective.
0:48:41Use CPUs when you can use CPUs, use GPUs when you need GPUs. So we look at deterministic mixture of
0:48:49methods, and then you add in agentic mixture of methods, and it comes up with what we think is
0:48:54the gold standard of verification. And what we see is when you apply this, our product is called
0:49:00SonarQube. So we talked to our customers, over a thousand customers recently, and we said,
0:49:08how likely are you or how often have you seen these issues from Agenda coding? And the statement
0:49:16to read here is SonarQube users are X percent more likely than non-users to have seen a very
0:49:21positive impact. So 81 percent more likely to have seen, to believe that AI coding has had a very
0:49:27positive impact on code security if you have that gold standard for verification.
0:49:3434% more likely to have seen an impact on time to market, 76% on outage frequency.
0:49:43So again, as I come back to how do we have confidence to ship, what we're seeing is
0:49:48that if you put these pieces together and you build verification incredibly strongly
0:49:54as a first-class citizen in the factory,
0:49:57you are actually building confidence
0:49:59and you're seeing
0:50:00results. Better security, more confidence in our time to market, improved or a reduction in number of outages, improved reliability.
0:50:10And of course, as I mentioned, you have to do this with evidence. Make sure you pass the quality gate. Make sure whatever, there's lots of acronyms up here, whichever ones you need to pass, whether that is your IT general controls inside of your
0:50:25internal controls and operations, whether that's MISRA if you're building cars,
0:50:29or any of the other ones that are on here, producing the evidence is what lets you sleep at night.
0:50:34It's what gives you confidence that you can move forward.
0:50:38So that, in essence, is how we think about the challenge of the future, that code
0:50:43is abundant, that confidence is scarce,
0:50:48and really the goal that we have and that we think
0:50:52that everyone should have as we think about the future of software development is how do we deliver
0:50:58confidence and the factories of the future need to be able to deliver confidence. So with that,
0:51:04I want to say thank you to everybody. We have a big booth. It's a big red booth
0:51:09on the floor right over that way and would love to talk to you more about all of this.
0:51:15Thank you very much. Have a great morning.
