Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

In 2025, customer growth was exposing three problems in Wisedocs’ AI pipeline: throughput could not meet demand, the system was too complicated to update, and developers avoided touching legacy code spread across more than ten repositories. Improving the underlying technology had become a business requirement.

The team committed to a six-month refactor. Was that the right decision, given how quickly coding agents were improving? The workload made the question consequential: Wisedocs processes complex medical claims in PDFs exceeding 10,000 pages, with some files larger than video files. A pipeline containing multiple ML models has several distinct scaling problems; making one component faster does not automatically make the whole system handle more work.

Bad code is not always a deliberate shortcut. Denys Linkov recalls trying to render a video-game character with Java’s System.out.println before he understood the difference between printing output and drawing an image. Experience changes the mistakes, but does not eliminate them. The useful question is what the resulting debt buys: a feature, a customer, or another business outcome—and whether that return exceeds the complexity that keeps accumulating afterward.

AI coding tools make that calculation more urgent. Linkov points to Anthropic case studies involving Spotify and Stripe as examples of faster shipping and refactoring. But faster implementation is only one part of product progress. He asks the audience whether technical products have improved over twenty years, then five years, then the past year. The shrinking window separates obvious long-term advances from a harder question: is the latest increase in shipping speed improving customers’ experience?

His anonymized uptime examples illustrate the concern: he describes services falling short of three- or four-nine reliability, without identifying the companies or supplying exact figures in the spoken explanation. Shipping faster does not establish that maintainability, reliability, or product quality improved. Those outcomes need their own attention.

Implementation began in April, after preparatory research. The team spent roughly two months evaluating five open-source orchestrators for its pipeline. Linkov places that work before the Google and OpenAI deep-research products were available. After gathering requirements, a three-person team built proofs of concept to check that the candidates could deliver the required behavior.

The original process combined manual research, some AI assistance, and a Confluence document comparing seventeen criteria. Linkov estimates that the research process could now be 90% faster; this is a retrospective estimate, not a measured rerun. His proposed workflow is:

1. Use deep research to investigate candidates.
2. Match the findings against the actual problem statements and criteria.
3. Assign subagents to examine individual products and criteria.
4. Build proofs of concept and evaluate the results.

The final step preserves the difference between a promising description and a working capability.

A polished research report can make a nonexistent feature seem credible. Accepting that report without testing the capability can send the project backward, regardless of how much research time the agent saved. The proof of concept is part of the selection process, not an optional confirmation after the decision.

After researching orchestration and model serving, Linkov began implementing Temporal activities and workflows that could reproduce behavior from the legacy system. He iterated on what he wanted, then gave the implementation task to o3 in Cursor. It worked faster than he could, but speed did not remove the need for supervision: the session required repeated guidance, manual edits, and code deletion.

Later reruns used Sonnet 4.6 and Opus 4.8. Linkov reports the following results for this particular task:

These are observations from a bounded refactor, not a general benchmark of every kind of repository change.

The interaction changed alongside the models. The earlier o3 run lacked substantial tool use in some categories; newer harnesses used subagents, planning calls, shell commands, and verification. Model execution became somewhat more expensive, while the work required less human intervention. Linkov estimates that rebuilding this particular task with the newer tools would take around one-fifth of the original time. The gain was therefore not just faster code generation: it was less manual effort to reach an acceptable result.

As models improve, the developer’s input shifts from small changes and explicitly supplied snippets toward a well-constructed specification. That makes missing requirements more expensive. Starting an agent before leaving for the evening feels productive until a flaw in the prompt or plan invalidates the unattended work. The quality of the handoff becomes part of the engineering task.

METR’s task-completion time horizons help describe this capability, but the time axis needs care: it measures task difficulty through human expert completion time, not how long an agent runs. A horizon is fitted at a specified probability of success. Linkov contrasts the commonly shared 50% horizon with an 80% or higher threshold. The improving trend remains, but an eighteen-plus-human-hour claim at one threshold cannot simply be carried over to the other. METR also warns that measurements above sixteen hours are unreliable with its current task suite.

For routine delegation, Linkov would prefer to reason about 90% or 99% success. Those are desired reliability thresholds, not results he demonstrates. A specification should support a reasonably dependable handoff; an hour-long attempt with a fifty-fifty chance of completion can consume elapsed time, compute, and the attention needed to discover that it failed.

In the Mythos Preview evaluation he presents, Linkov describes a notable decline in success around four hours of human task difficulty. He also points to inconsistent completion of some tasks around fifteen seconds and below fifteen minutes. His interpretation is that a short task is not automatically a reliable task: rising aggregate capability does not guarantee that a particular assignment will succeed.

Teams therefore need a shared understanding of what agents can accomplish and development practices that make completion checkable. Wisedocs’ verification processes matured during the refactor. Their purpose was practical: avoid spending time on repeated model attempts that never resolve the underlying problem.

The refactor consolidated roughly ten repositories into a monorepo, with the intention of adding new features afterward. The older repositories had accumulated more than six years of history. Linkov attributes their slower progress both to technical debt and to the absence of the AI coding tools available during the rebuild. The monorepo reached parity with the previous system within the first six months, according to Linkov.

Feature development continued beyond the parity milestone. Code volume and commit activity rose, although Linkov explicitly acknowledges that code volume is a weak productivity metric. The contribution graph uses a logarithmic scale and also reflects the gradual onboarding of more developers. Commit counts eased as work moved from reproducing existing behavior to adding product features, but delivery continued.

Eventually, almost every developer in the company contributed to the monorepo. Contributions extended beyond individual specialties: someone could change a schema, an API call, or another part of the stack as their work required. That broader participation is a distinct outcome from generating more code, even though the graphs do not isolate how much each change—repository structure, tooling, or team participation—caused the improvement.

Could a newer model perform the whole refactor from a broad goal? Linkov tried GPT-5.5 extra high, providing repository names and context about the underlying models and components. The agent reported completion after ten minutes and twenty-two seconds, having written about 2,000 lines of code. That small output prompted a closer inspection.

The result was scaffolding, not the required model implementations. A Ray Serve deployment and a bootstrap command were also missing. The agent’s completion report did not establish that the system had been rebuilt. This broader attempt exposed a gap in scope completion and self-validation that the successful bounded Temporal task did not resolve. Linkov’s expectation that substantial refactors would become consistently achievable within another six months was a forecast, not the outcome of this run.

The case for waiting is real. Models are improving, tool use is improving, and sandboxes and monitoring make agent behavior easier to support and inspect. Those advances make future cleanup less demanding. If the same work becomes easier next year, accepting some technical debt today can be a rational choice.

But AI-assisted development can recreate the same legacy conditions: a large volume of code, poor performance or quality, and too few people who understand how it works. When customer requirements change or failures appear, that lack of understanding makes adaptation harder. Guardrails remain necessary whether the team chooses a full refactor or only a partial one.

For Wisedocs, the earlier repository patterns had served customer demand and achieved business goals. Consolidation then enabled another stage of growth. Linkov reports shorter pipeline runtime, lower costs, and support for larger files after the refactor. He also reports that some features previously requiring multiple months could now ship in under a week.

The cleaner codebase also changed developers’ willingness to participate. People wanted to work in it, and patterns established there spread to other repositories. Those benefits extend beyond the immediate output of a coding agent: the system became easier for the organization to keep developing.

A monorepo rebuild is not the only route. An AI delivery system has multiple layers, and isolating particular components can avoid a full refactor. The timing decision depends on business value: what present constraints cost, what cleanup would unlock, and what might become cheaper later. Better future models do not remove the value of making the current system workable.

In the Q&A, Linkov distinguishes agent navigation from operating the complete system. Models have become better at exploring multiple repositories placed under a common parent directory. That does not make end-to-end testing, verification, and deployment equally simple across those repositories. A full-system sandbox also has to clone and configure them all, adding setup work before the agent can test anything. Consolidation addressed that operational friction as well as code discoverability.

Asked whether the requirements and guardrails were fixed before the project or discovered along the way, Linkov says the team revisited them. He reports that fifteen of the seventeen initial requirements were right. Verification practices evolved during implementation. In his account of the project’s starting point, plan mode was only emerging in Claude Code and was absent from Cursor; the team subsequently incorporated planning into its development lifecycle. Those tool limitations describe the historical workflow, not their present capabilities.

Another question assumes that pull-request reviews were already autonomous. Linkov corrects that assumption: all PR reviews during the refactor were human reviews. Local AI skills supplied additional code checks, but did not replace that review process. Although review was becoming more autonomous afterward, the original PRs served an additional purpose. A small group had built the new repository, and reviewing their changes helped other developers learn what the refactor contained. Review distributed knowledge as well as checking code.

Asked what would change most over the following six months, Linkov expects models to handle more complex assignments and companies to build more supporting infrastructure for refactors. The opportunity spans research, proofs of concept, code-quality validation, and checks for hidden assumptions. A library feature that appears to satisfy a requirement, for example, may exist only in beta. Catching that distinction early can matter as much as rewriting a file quickly. His forecast is for a faster path from understanding the problem to validating the system—not merely faster production of replacement code.