AI Agents for Performance: Ship Faster, Pay Less

What happens when generating code becomes easier than checking what it costs to run? Rajat Shah works on large-scale distributed systems for machine-learning model hosting in Netflix’s AI platform organization. His starting problem is performance-engineering throughput: how to keep infrastructure efficient as coding agents help engineers ship more code.

Shah frames the mismatch as code authoring becoming 10× faster, with a deliberately exaggerated suggestion that CPU bills are growing at the same pace. The underlying concern is concrete: agents optimized to produce working code quickly do not necessarily produce the fastest implementation.

Better models help, but they do not automatically know an organization’s internal platform, frameworks or established code patterns. An agent may import a convention from another codebase or invent a way to use an internal framework that its designers never anticipated. Performance depends on local knowledge as well as general coding ability.

A performance engineer typically works through a recurring sequence:

1. Trigger profiling on one production instance in a service fleet.
2. Download the profile, which may contain structured call-stack and CPU data in JSON.
3. Open a visualizer and search for expensive paths.
4. Find the corresponding packages and methods in the source repositories.
5. Diagnose a fix, submit it for code review and merge it.
6. Repeat the investigation for the next opportunity.

The visualizer makes the data readable, but interpreting it remains a learned skill. Finding a hot path is only the beginning; the engineer must connect it to code that can actually be improved.

That repetition makes proactive profiling difficult to sustain across many services. It is easy for profiling to become something people reach for during a 2 a.m. CPU incident. Shah estimates that identifying hot paths can take an engineer about 20 minutes. The experiment begins with a narrower question than autonomous optimization: can an LLM read the profile and shorten that search? The team tested the idea on live services.

The first assumption is that profilers expose comparable information across runtimes. A Java, Python or Go profiler can sample a production instance at high frequency and record its call stacks, including self CPU—work attributed to a method itself—and inclusive CPU, which includes work beneath it in the call path. The file formats need not be identical for an agent to reason over this common structure.

The second assumption is that coding agents already recognize common performance patterns from their training:

• Quadratic work: loops whose total work grows as O(N²).
• Loop invariants: computations repeated inside a loop even though their result could be calculated once.
• Repeated allocation: objects recreated on a frequently executed path when their construction could be moved elsewhere.
• Contention and batching: coordination or repeated small operations that could be reorganized to reduce overhead.

Shah makes this capability conditional on the quality of the code in training. The useful starting point is pattern recognition: combine an expensive runtime path with a familiar implementation smell to identify something worth investigating.

The first distinctive finding connects two rows in the profile. Rows one and three belong to the same call path: ImmutableMap.copyOf appears inside a tensor-merge method, shown on the slide as TensorSet.merge. Recognizing what the copy operation does lets the agent suspect repeated copying and quadratic accumulation. Shah reports that this initial inference came from the profile’s call stack, before the agent inspected the source. It gives the investigation a target; source inspection then establishes how that target is implemented.

To turn that suspicion into a proposed fix, the agent needs to connect runtime evidence to the correct version of the code:

1. Search for the method and identify its owning repository.
2. Resolve the production build to its exact Git commit and check out that revision.
3. Locate the relevant implementation, avoiding a detour through unrelated internal-library details.
4. Trace the full method call path with the source available.

The production commit is part of the evidence. Inspecting a different revision could mean explaining code that did not produce the profile. With this connection in place, the experiment on a couple of services could test whether the agent could go beyond detection and produce a real fix suitable for an engineer to approve.

In the tensor-merge example, Shah reports that the quadratic pattern accounted for 8.8% of CPU time during the profiling period. That is the cost attributed to the pattern, not the amount saved by removing it. With sufficient skill or prompt instructions and a capable coding agent, the workflow could proceed through repository checkout, implementation discovery and a proposed code change.

Shah reports reaching code review in under five minutes in the large-codebase experiment. He also reports observing production CPU and latency savings after optimization. The useful operational result is that the agent could carry an investigation through to a reviewable change, rather than merely name an expensive method.

The next opportunity was to reuse a discovery. A profile exposed Spectator metrics counter objects being created on every iteration of a hot path. Once the agent recognized that pattern, it could search other repositories for the same usage. Shah reports finding the counter-creation pattern in seven services, with potential CPU-cycle savings of 0.5–4.6% if it were fixed across those repositories. Those savings remain conditional; the broader mechanism is a production finding becoming a search pattern for additional services.

Once an agent can find and propose fixes, the next question is how to carry those discoveries into the development cycle. An individual invocation does not contain all the accumulated context of a performance engineer. Compact model memory is insufficient for remembering the organization’s patterns, previous investigations and framework-specific lessons.

Shah’s solution is a stateful catalog paired with a stateless LLM. Store patterns and anti-patterns centrally so agents working on different teams and products can reuse them. Some entries will be specific to an internal framework; others, such as quadratic accumulation, can be generalized across languages. The catalog becomes a durable description of what the organization has learned about efficient code.

The initial implementation can be Markdown files in a centralized Git repository. A vector database is unnecessary to get started. As production investigations uncover patterns, agents add entries; a later investigation in another service can consult those entries instead of rediscovering the same explanation. This is how one service’s profiling effort becomes reusable knowledge for the fleet.

An empty catalog is a valid starting point, but existing material can seed it. Shah suggests Jeff Dean’s writing on C++ optimization, TorchFix and internal performance playbooks. TorchFix is more precisely a Python static-analysis linter for PyTorch code with best-effort autofixes; its repository was archived in January 2026. Its documented checks still illustrate the kind of reusable patterns a catalog can capture. Internal playbooks contribute the organization-specific knowledge that a general coding agent lacks.

A useful entry supplies query hints, relevant symbols, services where the pattern has been confirmed and a confidence level that can be updated as evidence accumulates. That confidence should support a decision to request human review. It should not authorize a direct production push: an optimization can change business behavior that the agent does not understand or that the tests fail to cover.

Pair the anti-pattern with the preferred implementation. For the ImmutableMap accumulation example, the distinction is repeatedly copying a growing map versus accumulating entries and making the immutable copy once. A Java illustration makes that difference explicit:

import com.google.common.collect.ImmutableMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

final class MapAccumulation {
    static <K, V> ImmutableMap<K, V> copyEachStep(
            List<Map.Entry<K, V>> entries) {
        ImmutableMap<K, V> result = ImmutableMap.of();
        for (Map.Entry<K, V> entry : entries) {
            Map<K, V> next = new LinkedHashMap<>(result);
            next.put(entry.getKey(), entry.getValue());
            result = ImmutableMap.copyOf(next);
        }
        return result;
    }

    static <K, V> ImmutableMap<K, V> copyOnce(
            List<Map.Entry<K, V>> entries) {
        Map<K, V> result = new LinkedHashMap<>();
        for (Map.Entry<K, V> entry : entries) {
            result.put(entry.getKey(), entry.getValue());
        }
        return ImmutableMap.copyOf(result);
    }
}

For distinct, non-null keys and non-null values, the first method repeatedly copies an increasingly large map; the second accumulates into one mutable map before producing the immutable result. Both retain the last value for a repeated key. Recording the implementation alongside the pattern makes the catalog actionable, while preserving the need to check the actual application’s semantics.

A system that generates speculative optimization reviews can simply move the bottleneck onto reviewers. Before asking for attention, the agent should run unit, integration and basic functional tests. These checks establish whether the proposed change preserves the business behavior encoded in the test suite.

Incomplete coverage makes a second gate valuable: an automated canary comparison. Shah’s example uses two machines, one running the old code and one the proposed optimization. Send the same traffic to both for a period such as ten minutes, then compare their behavior. With suitable automation, this comparison can happen before the change requires a human reviewer’s attention.

The report gives the agent evidence for whether to open a review:

An increase in errors is a stop signal, even if CPU usage falls. Observability, canary execution and verification are standard engineering infrastructure that the agent must be able to use. The engineer remains the final decision-maker. Shah’s division of responsibility is direct: the profiler estimates the opportunity, the canary establishes the observed effect, and the engineer decides whether to accept the change.

Production profiling is a useful starting point, but changing code that already works in production requires care. As the catalog grows, its findings can move earlier in development. A reviewer agent can inspect a proposed change, look up the relevant pattern and leave an inline suggestion grounded in previous profiling evidence. The same catalog now supports both reactive diagnosis and preventative review.

The next step is to give the catalog to the authoring agent itself. Before writing a known inefficient pattern, the agent can retrieve the preferred approach and use it in the initial implementation. This consultation can slow code generation and consume additional tokens, so the catalog needs structure and indexing. Hierarchical navigation should lead the agent to the relevant entry without loading the whole collection into its context. Catching a pattern during authoring avoids both a later review correction and the possibility of deploying the known inefficiency.

Putting agents into more development stages only helps if the underlying checks are dependable. Business logic needs strong test coverage. Canary automation must be reliable, and the agent needs clear ways to invoke it and retrieve its numerical report. The Git catalog is another shared foundation: humans and agents can both read and author it. Engineers building a framework can record its good and bad usage patterns while they still have the design context in mind.

The integration chain should let the workflow trigger profiling on an instance in a CPU cluster, download the data, feed it to the model, retrieve the relevant source and validate a proposed fix through a production canary using shadow or real traffic. Weak connections between those steps create friction and can introduce production bugs. Begin with reactive investigation, make the foundations dependable and gradually move the accumulated knowledge into earlier development stages.

The first useful deployment does not need complete automation. In Shah’s manual baseline, engineers can spend hours finding problems. At level one, a person may still trigger the profiler, transfer the profile and run the canary, while the LLM identifies a potential fix. That already frees time previously spent inspecting flame graphs.

Level two connects the tools, integrations, harnesses and hooks into a predefined workflow. The sequence is fixed: trigger profiling, download the profile, analyze it, validate a candidate change with a canary and provide a suggested fix for review. The model does not need to invent the overall plan. Once the sequence works reliably, it can be scheduled—for example, weekly on each service—to find newly introduced problems closer to the time their code was written.

Level three allows the agent to plan, reason and act beyond a fixed workflow. That flexibility brings additional obligations: stronger evaluation, sandboxing and security guardrails against prompt injection and other attacks on agent infrastructure. Shah recommends starting with assistance and progressing to orchestration, where he sees the greatest practical benefit. Greater autonomy is a further investment to make when the workflow actually needs it.