AI Engineer World's Fair 2026
In the Land of AI Agents, the Verifiers Are King
Read the talk
In the Land of AI Agents, the Verifiers Are King
Coding agents can generate working code faster than organizations can trust it. Verification must shape generation, review and maintenance if that speed is to survive production.
From a talk by Tariq Shaukat
Before you start: Familiarity with pull requests, automated tests and continuous integration will help you follow the development workflow.
When plausible output reaches production
What happens when an AI-generated report looks credible enough to publish, but its sources are invented? Tariq Shaukat opens with reports he says KPMG and EY had to retract because of hallucinations, followed by law firms getting into trouble over fabricated citations and case law. These are failures at the point where plausible output becomes consequential work.
The same problem extends across professional services, legal work, marketing and finance: a model can produce something that sounds correct without giving its user a reliable way to establish that it is correct. For software development, the question becomes more specific. Code can run, tests can pass and an agent can finish its assignment—but is that enough to trust the result?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Task duration depends on the success threshold
METR’s task-completion time horizons offer one way to describe coding-agent progress: measure the difficulty of tasks by how long they take human experts, then estimate the task duration at which an agent achieves a specified probability of success. This is human task duration, not the agent’s running time. Shaukat describes rapid progress and cites preview benchmarking of Mythos conducted roughly a month before the talk.
His reading of the chart changes sharply when the required success rate rises:
| Required task success | Human task duration, as reported by Shaukat |
|---|---|
| 50% | Approximately 16–18 hours |
| 80% | Approximately 3.5 hours |
The exact estimates are Shaukat’s account of the chart; METR also warns that estimates above 16 hours are unreliable with its task suite. The useful comparison is that a longer task horizon does not imply dependable completion. The success threshold is part of the capability claim.
Even the higher threshold leaves an enterprise acceptance problem. A customer CTO’s response was that an employee delivering information that was only 80% accurate might end up under performance review. The analogy expresses the operational concern, although METR’s measure is successful task completion rather than the percentage of individual statements that are accurate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Working code still carries defects
Sonar, where Shaukat is CEO, approaches the next question through code verification. Its benchmark asks models to solve more than 4,000 problems, then evaluates both functional correctness and properties of the resulting code. The newly released Fable models discussed earlier at the conference had not yet been included in that testing.
Functional correctness asks whether the code produces the expected behavior on the evaluated cases. Complexity, bugs and security findings ask different questions about what will enter the repository. Shaukat describes strong functional performance across the tested models, but substantial variation in complexity; he singles out GPT-5.5 as doing particularly well on that dimension. Bugs and security issues remain. The production concern is therefore the quality of the code flowing into agentic workflows, even when generation succeeds at its immediate task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generation speed becomes maintenance work
Shaukat describes a study attributed to Carnegie Mellon in which an initial 3–5× increase in productivity or velocity dissipates over roughly three months toward the pre-agent baseline. That should be read as his characterization of a velocity result, not a general estimate of labor productivity. The underlying distinction matters: producing more code does not necessarily mean completing more durable, useful work.
As generated code accumulates, security problems, maintainability issues, reliability defects and complexity create additional work. The bottleneck moves from writing code to understanding, checking and repairing it. In Shaukat’s account, technical debt can arrive as quickly as the code itself—or faster.
This also limits what a local proof or passing test can establish about a large application. Individual code properties may be provable, while the surrounding software remains difficult to reason about because of dependencies, interactions and existing debt. Verification belongs inside development, where its findings can change what the agent generates, rather than appearing only at the end as a conventional code review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Surround generation with guide, verify and solve
Sonar calls this framework the Agent Centric Development Cycle, or ACDC. Code generation sits at the center, surrounded by three recurring disciplines:
- Guide: Give agents the information and boundaries needed to make appropriate changes.
- Verify: Check the outcomes with tools and techniques suited to different failure modes.
- Solve: Repair the problems those checks uncover.
The framework makes these disciplines part of the development life cycle. Their value comes from feeding one another, rather than operating as disconnected checks around a generator.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Guide with context and constraints
Shaukat introduces Sonar Vortex as a product launched around the time of the talk, then separates guidance into context and constraints. Context helps the agent understand the repository. At the scale of a million, a hundred million or a billion lines of code, useful guidance includes architectural awareness and semantic navigation maps: ways to find relevant definitions, relationships and components without rediscovering the whole system for each task.
Constraints describe what changes are acceptable. They include approved and prohibited dependencies, coding standards, guardrails and the intended architecture. Existing architecture tells an agent how the system is organized today; intended architecture tells it where a change should move the system. Without both, an agent can understand the current repository and still make a change that violates its direction.
Shaukat reports that Sonar’s testing found more than a 30% reduction in tokens used to solve a given problem when agents received context and constraints. Sonar’s launch material bounds the related result to discovery-heavy refactoring tasks, rather than every kind of coding task. The mechanism is straightforward: better navigation and clearer boundaries reduce the work spent searching and pursuing unsuitable approaches. Guidance becomes preemptive verification, leaving fewer problems to detect and fix later.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use different techniques for different failures
Zero-trust, multilayered verification starts from the fact that models have characteristic behavior and biases. Using different models and different verification techniques gives the system more than one way to challenge a generated result. The goal is to cover the intricacies of software with complementary checks.
| Verification layer | What it examines |
|---|---|
| Algorithmic | Data flows, control flows, known patterns, secrets |
| Agentic | Intent, business logic, anomalies, overlooked cases |
Algorithmic analysis can trace relationships and detect recognizable defects. Agentic review can ask whether a change makes sense for the requested behavior and the application’s business rules. Neither layer subsumes the other.
For example, consider this Java method for withdrawing an amount represented in cents:
java
static long withdraw(long balanceCents, long amountCents) {
if (amountCents <= 0) {
throw new IllegalArgumentException("Amount must be positive");
}
if (amountCents > balanceCents) {
throw new IllegalArgumentException("Insufficient funds");
}
return balanceCents - amountCents;
}
The branches make the rejection paths explicit, and tests can exercise them. But whether rejecting every overdraft is correct depends on the account’s business rules. An intent-aware review must compare that policy with the requirement. This illustrates why checking control flow and checking business intent are separate jobs.
Shaukat describes customers and partners using multilayered verification as experiencing 44% fewer AI-derived production outages. The published survey wording is narrower: developers using SonarQube were 44% less likely to report AI-code outages. That is an association in reported experience, not a measurement showing that incident frequency fell by the same amount or that verification alone caused the difference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Clean code reduces the next agent’s work
Generating technical debt is a reason to control it actively, not to stop generating code. Verified maintenance uses remediation agents and a continuing verification discipline to repair the repository as it changes. This raises a practical question: does clean code matter to agents as much as it matters to people?
Agents must understand a codebase before they can operate on it. Shaukat reports that running the same agentic tasks on a typical codebase and a cleaned version required fewer tokens and less reasoning effort on the cleaned version. He gives no numerical reduction. That comparison is a one-shot observation; the further expectation is that maintenance compounds because a cleaner result becomes the starting point for the next task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect three operational loops
Code review, static analysis and AI review tools may already exist in a development organization. ACDC asks that they operate as a system. In the agentic loop, the agent receives context and constraints while doing the work, gets verification feedback during generation, and fixes the problems that feedback identifies. Verification can therefore influence the change before it reaches a pull request.
The CI verification loop combines algorithmic and agentic review around pull requests. Review capacity must rise with generation capacity, or the queue simply moves downstream. Evals and quality gates determine whether changes pass. The code maintenance loop completes the system by addressing debt in the repository that future agent tasks will inherit. These loops share an objective, but act at different points: during a change, at its acceptance boundary and across the continuing life of the codebase.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The repository compounds in either direction
Neglect verification and maintenance during an AI coding rollout, and each task can leave a harder starting point for the next. More defects and complexity create more repair work; the early velocity gain begins to disappear. Shaukat returns to the Carnegie Mellon study as an illustration of that downward path.
Shaukat reports a 92% reduction in issues in a Sonar test with one large bank using advanced coding agents and the guide, verify, solve approach inside agentic loops. He explicitly describes this as a cumulative result while solving a problem over minutes and hours, not a 92% improvement independently delivered by every loop. The bank is unnamed, and the talk does not specify the issue-counting protocol or sample size.
The target setting is an enterprise with an existing codebase, potentially millions of lines long. In that environment, successful generation is only one part of the job. The system also has to preserve and improve the conditions under which subsequent changes will be made.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make verification part of the development system
The closing product slide maps SonarVortex and SonarSweep to the agentic loop, SonarQube and gitkr to CI verification, and SonarQube Remediation Agent to code maintenance. The product mapping makes the three operating locations concrete. Shaukat’s final recommendation is the development cycle itself: deliberate verification built into the system, so that guidance, checks and repairs improve both the current change and the repository that the next agent will encounter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Interactive task-horizon charts and methodology explaining human task duration, success thresholds, and benchmark limitations.
Sonar's original framework announcement explains guidance, generation, verification, and remediation.
Launch details for Vortex and the Remediation Agent, including the conditions behind reported token savings.
Further reading
Original study of Cursor adoption, temporary increases in code output, and persistent complexity and static-analysis warnings in open-source projects.
- Sonar's GPT-5.5 evaluationArticle
A 4,444-task Java evaluation separating functional test performance from complexity, bugs, and security findings.
Updates since the talk
Current guidance for adding context, verification, and remediation to coding-agent and CI workflows.
Read the complete timestamped transcript
- 0:00
[upbeat music] Please join me in welcoming the chief executive officer at Sonar, Tariq Shaukat. [upbeat music]
- 0:34
Morning, everyone. You enjoy that last talk? That was amazing. Um, I particularly loved the end, the being unreasonable part. I thought that was awesome. Um, I also wanna just...
- 0:46
I'm trying to calculate the odds of Tariq following Tariq as the first two sessions in the morning. Uh, [laughs] I think the odds are pretty low on this one, but, uh, thrilled to be here today.
- 0:56
Um, as, as was just mentioned, I am with Sonar. We are in the code verification space, and I'm here today to talk about verification. And I think we're all here, uh, in large part because we believe to some extent that AGI is here, it's coming.
- 1:12
The models, we just heard about Fable, it's really incredible what is going on in the, in the world today, and yet we work almost exclusively with enterprises around the world, and the conversation that we have more is the question mark version.
- 1:27
Is AGI here? And why are they asking these questions? It's because you can read the news every day, and I'm not trying to name and shame here, but if you look at KPMG putting out reports that they have to, uh, re- uh, retract because of hallucinations, uh, EY doing the same thing, law firms getting into lots and
- 1:50
lots of trouble because of made-up citations, made-up case law, things like this. I think we can really start to question, how do we get value out of AI? The models are amazing, as we just heard, but the hard part, as the other Tariq just said, is getting value out of it.
- 2:09
The struggle is that AI slop is everywhere. I'm sure you all see this inside of your organizations. I'm sure you see this in your everyday life, that AI is amazing.
- 2:21
The models are incredible at generating very plausible output. They're incredible at generating things that sound correct, but are they correct? And how do you know that they're correct is a big problem, and it's a big problem in professional services, as we saw.
- 2:36
It's a big problem in legal. But really, I think if we're honest, it's pr- it's a big problem in every sector, in every field, whether it's marketing or finance or you name it.
- 2:46
You have this question of, how do you actually know if it's true? How do you know if it's good or if it is slop? And the question that we ta-- uh, we deal in the coding space, in particular.
- 2:58
We deal with software development, and the question we get as we talk to, I'm sure, many of the people here in the room and a lot of our customers is, isn't software development different?
- 3:09
And we can look at the data on this and, uh, the Mythos models, um, this is data from, um, Meter. Uh, you may have seen this, M-E-T-R. Um, the coding agents are getting better, uh, very quickly.
- 3:23
They're getting a lot better very quickly, and you can see, uh, the progression, the exponential curve here. What this shows on this chart is how capable are the models at completing tasks that humans would take.
- 3:36
So can they complete a task that takes one hour, two hours, whatever it is? The latest Mythos model, at least per the benchmarking which was done a month or so ago in the preview mode, was you're getting to sixteen to eighteen hours.
- 3:49
So they're actually able-- the agents are able to complete long-running tasks, and it really is starting to transform how work is happening. But the critical caveat when you read the data is this is at a fifty percent success rate.
- 4:05
Okay? So y- it is, again, able to complete tasks, but is it able to complete tasks correctly is the question. So if you start looking at, all right, let's dial up
- 4:16
the accuracy, right? You dial it up to eighty percent and there's still progress, but it is much slower progress. Instead of eighteen hours, you're at about three and a half hours or something along these lines.
- 4:27
And by the way, this is still at eighty percent accuracy, and as I was presenting this to the CTO of one of my, uh, large customers, his response was, "But Tariq, I would still put someone who gave me eighty percent accurate information on a performance review probably," right?
- 4:43
This isn't necessarily enterprise grade. The problem is that the models themselves, in full disclosure, we have not yet, uh, done this benchmarking on the Fable models, obviously 'cause they are just being released.
- 4:59
But as you look at the models, the models are getting smarter, but they still produce a lot of problem- problematic code. This is benchmarking that we do. We give the models a series of over four thousand problems, and we basically ask it to generate the response to the problems, and then we analyze both the functional correctness, which
- 5:20
is critical, and they all do extremely well on this notion of functional correctness, right? Um, but then we look at how complex is the code, how buggy is the code, how secure is the code.
- 5:32
And what you see with even the state-of-the-art models is that complexity is still high. It's actually quite variable, as you can see here. Um, GPT FiveFive has done particularly well on the complexity side of things.
- 5:45
It still generates bugs. It doesn't generate massive amounts of bugs, but it still generates bugs, and it still generates security issues. So this is the output of the models that are going into the agentic workflows.
- 5:59
And again, this is not... You know, I'm at the AI engineer conference. This is not me saying AI is fake or, or, um, incorrect, but it is-
- 6:08
Um, trying to address this question of how do you really get value in a production setting out of AI? This is a study that was done in Carnegie Mellon, uh, University, and it looked at what is the actual productivity benefit that you see from the use of AI coding agents.
- 6:30
And what you see, I think, really resonates with a lot of what I see firsthand in the market, which is you have a initial just amazing boost of productivity, of velocity in particular.
- 6:44
What you see is a three to five X boost in productivity or in, in velocity. Um, that dissipates in three months. At the end of three months, it starts to come back to the, the normal before you were using the agents.
- 6:57
And if you ask why, it is because of the two pieces in red here, that you start to see there's an increase in velocity, but there's an increase in security issues, there's an increase in maintainability issues, there's an increase in reliability issues, and there's an increase in complexity.
- 7:15
So essentially, you're building the technical debt as quickly as you are generating the code, or maybe even more quickly, and that creates a different set of work. It creates a different bottleneck.
- 7:28
And so to us, this is now the critical question in AI, which is in a world in which code is provable, and there's sessions that, um, uh, I'm actually very much looking forward to attending about formal methods and proofs and things like this.
- 7:44
Code is provable. But when you start dealing with large code bases, software is not. It's still very complex. It is still very messy. There's lots of, um, dependencies. There's lots of, uh, technical debt already in most code bases.
- 7:59
And so this question of verification is actually key. And what I'm going to be arguing is that you can treat verification as an afterthought, or you can bake verification into the process.
- 8:12
And if you bake it into the process of generating code, of doing software development, you can actually start to get materially better outcomes from the coding agents than if you view it as an afterthought, if you view it as just the old school code review.
- 8:28
So as we've been thinking through this, we basically have constructed a framework, and there's lots of competing frameworks around this, but I'll just talk you through, uh, ours. We call it the Agent Centric Development Cycle.
- 8:39
For shorthand, we call it ACDC sometimes. And the idea here is how do you get verification powered agentic loops? At the center, there's a lot of focus on the code generation piece.
- 8:52
Like, how do you actually get the models and the agents to generate the code that you need to solve the problem? And what we argue is that y- you should surround this with the right disciplines, the right tools, the right processes to do three things, to guide the agents.
- 9:08
And Tariq was talking a lot about different aspects of this, actually. Guide the agents, verify the outcomes, and then solve the problems. And you have to make this part of the discipline, part of the process, part of the new software development life cycle if you wanna be successful in the AI world.
- 9:27
So if I double click on some of these pieces, what do we mean by guide? We've done a lot of experimenting around guide. We've just launched a product, um, yesterday, I think, called Sonar Vortex, that starts to get into this area.
- 9:40
What we find is critically important is to think about guide as context and constraints. And we separate out context and constraints very deliberately because context is you have your code repositories.
- 9:53
How do we make it easier for the agents to understand, for the models to understand what is in your code base? If you have a million lines of code, if you have a, a hundred million lines of code, you have a billion lines of code, the agents work better if they understand your code base.
- 10:08
So how do you give it architectural awareness? How do you provide, uh, semantic navigation, uh, maps, um, and, uh, and help them understand the territory, to borrow what Tariq was just talking about?
- 10:19
And we find it equally valuable, and I don't think this part is talked enough about, to provide the constraints as well. You have guidelines that you want your code to follow.
- 10:31
You have dependencies you are okay using. You have dependencies that you are not okay having. You have coding standards. You have guardrails. You have intended architecture. We spend a lot of time talking about existing architecture, but what about where you want to go?
- 10:46
And so this idea of context and constraints, uh, we've found in our testing generates a massive improvement in agent effectiveness and a massive, uh, improvement in token consumption o- over thirty percent reduction in tokens being used to solve a given problem.
- 11:06
And, and if you ask why, it's because you're actually making the life of the agent easier. You're helping it navigate better.
- 11:13
So then we get into the heart of this, and we really think of guide as preemptive verification. How do you make sure there's less to verify, less to fix, this sort of thing?
- 11:21
Then you get to the heart of verification. And what we believe quite strongly, and what we've seen work in practice, is this idea of zero trust multilayered verification. Zero trust, every model has biases, every model produces, has a character, has a personality.
- 11:39
So let's make sure we use different models and different techniques to make sure your code is safe, to make sure it's reliable, to make sure it's secure. And multilayered really speaks to the earlier point that software is complex.
- 11:53
Software is very messy. Software has lots of, of, of intricacies involved with it. And so what we believe, uh, and again have found to be quite, um, impactful here is that a combination of algorithmic verification, looking at things like data flows, control flows, known pattern secrets These areas combined with what is now possible with agentic
- 12:17
verification, looking at intent, business logic, the unknown unknowns, actually, again, to borrow from the last, uh, presentation. The fusion of these things, the, the deliberate multilayered fabric that you put in place can actually s- you can see the results of this in production.
- 12:36
So as we look at our partners and customers who use a multilayered verification approach, they are reporting AI-derived production outages being forty-four percent less frequent than the ones who do not.
- 12:51
So you can start seeing a material improvement in reliability, in security, and in maintainability.
- 12:58
And then the last point I mentioned is technical debt does explode, right? As you generate code, technical debt is also generated, and again, this is not stop doing it.
- 13:10
This is be aware and let's start controlling it. And so what we, um, have seen be super effective is to have an active process, to have an active discipline again around code maintenance and thinking about how you do verified code maintenance.
- 13:29
Um, I won't walk through every step of this, but a- the agents, whether that is a, uh, set of remediation agents, whether it's a strong discipline around verification, does keep your code base clean.
- 13:42
And a lot of people have asked me, "All right, but do agents care about clean code?" Hu- human developers care, care about clean code. Do agents care about clean code?
- 13:51
And what we find, again, is they absolutely do because the agents have to understand the code base if they're going to operate on it. So this is a one-shot view.
- 14:01
Um, we think this is something that compounds, but if you just do the exact same agentic tasks on a typical code base and then one that has been cleaned, you see a material reduction in the amount of tokens, reasoning energy, et cetera, needed for those cleaner, uh, code bases versus the typical code bases, right?
- 14:21
If you make the life of the, of the agent easier, if you maintain your code base, then you'll actually see compounding effects. Now, the important thing in our mind is to construct the system.
- 14:33
This is how I started, is saying, you know, I'm sure all of us do code reviews. You may use static analysis tools. You may use AI code, uh, review tools, a whole range of things, and we believe that you have to put this in a system.
- 14:46
And again, uh, we're happy to, in our booth downstairs, talk through what this looks like. But we really believe that the construction of the software development life cycle in an AI world, um, needs to embed this notion of guide, verify, and solve inside of it.
- 15:03
And you need to do it in three loops, and you need to think about these three loops. There's the agentic loop, which I think is the key buzzword of the conference, um, now, but how do you provide the agents as it's generating the code, as it's doing the work with the context and constraints,
- 15:20
with the in-loop verification so that the agent is getting verification as it's working, and how do you fix problems? That's, that's the blue loop here. What we, what we talk about is the inner loop verification piece.
- 15:32
There's a second, which is your continuous improvement process, and how do you really combine the power of algorithmic and agentic to generate your p- your pull request, review the code, and by the way, the velocity of this has to go up massively.
- 15:49
So to review the code using agents and to do this multilayered verification, and then you have your evals, and I think the opening speaker talked about how evals may be the buzzword of the, um, conference.
- 16:01
But you have your evals, and you have your quality gates to check are you actually passing. So you have your con- your code maintenance loop, agentic loop, CI verification loop, and deliberate design of these loops with verification at the center is a compounding system.
- 16:20
It's a system that reinforces itself, and it reinforces itself in the positive, and it reinforces itself in the negative. And we've seen customers who, uh, have kind of neglected, as they've rolled out AI coding tools, they've neglected verification.
- 16:35
They've neglected this idea of code quality, of code, um, maintenance, things like that, and you get into a downward spiral pretty quickly. This is what the Carnegie Mellon, uh, case study, uh, or, uh, study actually shows is that you actually have all the benefits start to dissipate.
- 16:53
Or you can get into this self-reinforcing loop, and one of the tests we did with one of the large banks who are using some of the cutting-edge, the folks who are all around here today, um, cutting-edge a- agentic coding tools, they can get a ninety-two percent reduction in issues if you actually take this guide, verify, solve approach
- 17:13
inside of those agentic loops. If, again, this compounds. It's not that each loop is ninety-two percent better. It's that as you go through solving the problem over minutes and hours, that you actually see a compounding benefit.
- 17:29
So that is, uh, essentially how we see the benefit here. The... How we see the controlled, um, value-creating use of AI in enterprise settings. And when I say enterprises, people with existing code bases, people with, with, you know, millions of lines of code already.
- 17:49
There's the agentic loop. There's the CI verification loop. There's the code maintenance loop. I am required by my marketing team to put up a version of this that has our products on here, so these are our products, and you can come and see us later.
- 18:02
But the most important thing is really to say
- 18:06
our recommendation is this agent, the ACDC, Agent Centric Development Cycle. The core part is deliberate verification built into the system. So if you'd like to learn more, um, we have a booth.
- 18:17
It's the big red booth downstairs. We'd love to talk more. We have some d- DoubleClick sessions coming up, so please do, uh, join those. And, uh, have a great conference.
- 18:26
Thank you all. [audience applauding] [upbeat music]