In the Land of AI Agents, the Verifiers Are King

What happens when an AI-generated report looks credible enough to publish, but its sources are invented? Tariq Shaukat opens with reports he says KPMG and EY had to retract because of hallucinations, followed by law firms getting into trouble over fabricated citations and case law. These are failures at the point where plausible output becomes consequential work.

The same problem extends across professional services, legal work, marketing and finance: a model can produce something that sounds correct without giving its user a reliable way to establish that it is correct. For software development, the question becomes more specific. Code can run, tests can pass and an agent can finish its assignment—but is that enough to trust the result?

METR’s task-completion time horizons offer one way to describe coding-agent progress: measure the difficulty of tasks by how long they take human experts, then estimate the task duration at which an agent achieves a specified probability of success. This is human task duration, not the agent’s running time. Shaukat describes rapid progress and cites preview benchmarking of Mythos conducted roughly a month before the talk.

His reading of the chart changes sharply when the required success rate rises:

The exact estimates are Shaukat’s account of the chart; METR also warns that estimates above 16 hours are unreliable with its task suite. The useful comparison is that a longer task horizon does not imply dependable completion. The success threshold is part of the capability claim.

Even the higher threshold leaves an enterprise acceptance problem. A customer CTO’s response was that an employee delivering information that was only 80% accurate might end up under performance review. The analogy expresses the operational concern, although METR’s measure is successful task completion rather than the percentage of individual statements that are accurate.

Sonar, where Shaukat is CEO, approaches the next question through code verification. Its benchmark asks models to solve more than 4,000 problems, then evaluates both functional correctness and properties of the resulting code. The newly released Fable models discussed earlier at the conference had not yet been included in that testing.

Functional correctness asks whether the code produces the expected behavior on the evaluated cases. Complexity, bugs and security findings ask different questions about what will enter the repository. Shaukat describes strong functional performance across the tested models, but substantial variation in complexity; he singles out GPT-5.5 as doing particularly well on that dimension. Bugs and security issues remain. The production concern is therefore the quality of the code flowing into agentic workflows, even when generation succeeds at its immediate task.

Shaukat describes a study attributed to Carnegie Mellon in which an initial 3–5× increase in productivity or velocity dissipates over roughly three months toward the pre-agent baseline. That should be read as his characterization of a velocity result, not a general estimate of labor productivity. The underlying distinction matters: producing more code does not necessarily mean completing more durable, useful work.

As generated code accumulates, security problems, maintainability issues, reliability defects and complexity create additional work. The bottleneck moves from writing code to understanding, checking and repairing it. In Shaukat’s account, technical debt can arrive as quickly as the code itself—or faster.

This also limits what a local proof or passing test can establish about a large application. Individual code properties may be provable, while the surrounding software remains difficult to reason about because of dependencies, interactions and existing debt. Verification belongs inside development, where its findings can change what the agent generates, rather than appearing only at the end as a conventional code review.

Sonar calls this framework the Agent Centric Development Cycle, or ACDC. Code generation sits at the center, surrounded by three recurring disciplines:

• Guide: Give agents the information and boundaries needed to make appropriate changes.
• Verify: Check the outcomes with tools and techniques suited to different failure modes.
• Solve: Repair the problems those checks uncover.

The framework makes these disciplines part of the development life cycle. Their value comes from feeding one another, rather than operating as disconnected checks around a generator.

Shaukat introduces Sonar Vortex as a product launched around the time of the talk, then separates guidance into context and constraints. Context helps the agent understand the repository. At the scale of a million, a hundred million or a billion lines of code, useful guidance includes architectural awareness and semantic navigation maps: ways to find relevant definitions, relationships and components without rediscovering the whole system for each task.

Constraints describe what changes are acceptable. They include approved and prohibited dependencies, coding standards, guardrails and the intended architecture. Existing architecture tells an agent how the system is organized today; intended architecture tells it where a change should move the system. Without both, an agent can understand the current repository and still make a change that violates its direction.

Shaukat reports that Sonar’s testing found more than a 30% reduction in tokens used to solve a given problem when agents received context and constraints. Sonar’s launch material bounds the related result to discovery-heavy refactoring tasks, rather than every kind of coding task. The mechanism is straightforward: better navigation and clearer boundaries reduce the work spent searching and pursuing unsuitable approaches. Guidance becomes preemptive verification, leaving fewer problems to detect and fix later.

Zero-trust, multilayered verification starts from the fact that models have characteristic behavior and biases. Using different models and different verification techniques gives the system more than one way to challenge a generated result. The goal is to cover the intricacies of software with complementary checks.

Algorithmic analysis can trace relationships and detect recognizable defects. Agentic review can ask whether a change makes sense for the requested behavior and the application’s business rules. Neither layer subsumes the other.

For example, consider this Java method for withdrawing an amount represented in cents:

static long withdraw(long balanceCents, long amountCents) {
    if (amountCents <= 0) {
        throw new IllegalArgumentException("Amount must be positive");
    }
    if (amountCents > balanceCents) {
        throw new IllegalArgumentException("Insufficient funds");
    }
    return balanceCents - amountCents;
}

The branches make the rejection paths explicit, and tests can exercise them. But whether rejecting every overdraft is correct depends on the account’s business rules. An intent-aware review must compare that policy with the requirement. This illustrates why checking control flow and checking business intent are separate jobs.

Shaukat describes customers and partners using multilayered verification as experiencing 44% fewer AI-derived production outages. The published survey wording is narrower: developers using SonarQube were 44% less likely to report AI-code outages. That is an association in reported experience, not a measurement showing that incident frequency fell by the same amount or that verification alone caused the difference.

Generating technical debt is a reason to control it actively, not to stop generating code. Verified maintenance uses remediation agents and a continuing verification discipline to repair the repository as it changes. This raises a practical question: does clean code matter to agents as much as it matters to people?

Agents must understand a codebase before they can operate on it. Shaukat reports that running the same agentic tasks on a typical codebase and a cleaned version required fewer tokens and less reasoning effort on the cleaned version. He gives no numerical reduction. That comparison is a one-shot observation; the further expectation is that maintenance compounds because a cleaner result becomes the starting point for the next task.

Code review, static analysis and AI review tools may already exist in a development organization. ACDC asks that they operate as a system. In the agentic loop, the agent receives context and constraints while doing the work, gets verification feedback during generation, and fixes the problems that feedback identifies. Verification can therefore influence the change before it reaches a pull request.

The CI verification loop combines algorithmic and agentic review around pull requests. Review capacity must rise with generation capacity, or the queue simply moves downstream. Evals and quality gates determine whether changes pass. The code maintenance loop completes the system by addressing debt in the repository that future agent tasks will inherit. These loops share an objective, but act at different points: during a change, at its acceptance boundary and across the continuing life of the codebase.

Neglect verification and maintenance during an AI coding rollout, and each task can leave a harder starting point for the next. More defects and complexity create more repair work; the early velocity gain begins to disappear. Shaukat returns to the Carnegie Mellon study as an illustration of that downward path.

Shaukat reports a 92% reduction in issues in a Sonar test with one large bank using advanced coding agents and the guide, verify, solve approach inside agentic loops. He explicitly describes this as a cumulative result while solving a problem over minutes and hours, not a 92% improvement independently delivered by every loop. The bank is unnamed, and the talk does not specify the issue-counting protocol or sample size.

The target setting is an enterprise with an existing codebase, potentially millions of lines long. In that environment, successful generation is only one part of the job. The system also has to preserve and improve the conditions under which subsequent changes will be made.

The closing product slide maps SonarVortex and SonarSweep to the agentic loop, SonarQube and gitkr to CI verification, and SonarQube Remediation Agent to code maintenance. The product mapping makes the three operating locations concrete. Shaukat’s final recommendation is the development cycle itself: deliberate verification built into the system, so that guidance, checks and repairs improve both the current change and the repository that the next agent will encounter.