From Signal to PR: Anatomy of a Self-Improving Agent

When a platform goes down late at night, how much work must happen before someone can even propose a fix? Arize’s experience building Alyx, its in-product assistant, exposed that problem from the inside. Jason Lopatecki describes the first version, roughly two years earlier, as poor. Signal grew out of the debugging and improvement tasks the team repeatedly performed while trying to make Alyx work well. The immediate problem is familiar: the platform is down, it is late, and restoring it still takes substantial human effort.

Traditional observability presents a person with dashboards, graphs, and logs to inspect. The next stage pairs that person with a coding agent and skills for systems such as Pyroscope or Google Cloud. Those skills make runtime evidence accessible during debugging. The repository describes the paths software could take; telemetry identifies the path it actually took. Without that evidence, an agent must reason across many possible executions.

Evaluations add another layer of evidence. Today, humans still make or review fixes; the longer-term goal is a continuous loop in which agents use traces, logs, and evaluations to improve the system that produced them. That changes observability’s consumer as well as its interface: evidence must be useful to an investigating agent, not only legible to a person clicking through a UI.

Generating code quickly does not automatically make improvement equally fast. Once coding agents can produce plausible changes, the bottleneck shifts toward confidence in the proposed fix: is this the change to push? The expensive work includes establishing what failed, why the proposed change addresses it, and whether the evidence is strong enough to proceed.

Signal inverts the usual sequence. Instead of waiting for a person to inspect an error and assign a debugging task, an agent examines the error first. It can open an issue even when it cannot produce a fix. The person then encounters an investigation with supporting evidence already assembled, rather than an unexplored ticket.

This does not reduce every human interaction to approval. Lopatecki describes people still steering the second and third steps of an investigation. Preparing the evidence removes the cold start; a person may still need to redirect the diagnosis or carry the change across the finish line.

The loop begins with a trigger and a way to gather context. A run can start periodically—for example, every five minutes—or when an event such as an error occurs. Observability skills let the agent decide which evidence it needs and retrieve it. This replaces a fixed sequence of dashboard clicks with an investigation that can choose its next query.

The key inputs serve different purposes:

Arize’s skills download production traces or logs into temporary files inside the repository. That puts the record of what happened beside the code that could explain it. The combination matters: useful data, a file format the harness can work with, and composable skills that retrieve more evidence as the investigation develops.

The starting point is the debugging workflow already running on a developer’s laptop. Moving it into a sandbox gives scheduled or event-driven runs an execution environment without requiring a person to start each session. The sequence is deliberately incremental:

1. Get the agent, repository, and evidence-gathering skills working locally.
2. Reproduce that working environment in a sandbox.
3. Let the observability platform launch it on a schedule or in response to errors.

The sandbox changes where and when the investigation runs; it should preserve the useful local debugging behavior.

Alyx supplies a concrete failure. Arize’s assistant uses to-do updates to manage its task list. In the example, it makes repeated update calls and encounters a stream-cancellation event. Signal runs in the background, gathering debugging information and raising issues around failures like this.

Lopatecki describes the proposed Alyx repair as a one- or two-line fix. The walkthrough does not expose the exact patch or establish the cancellation’s root cause. Larger changes are more likely to require a person to finish them, but an issue containing the relevant evidence still changes the starting point: the engineer can guide an existing investigation instead of assembling it from scratch. That is the practical shift from responder toward reviewer.

Traces and evaluations remain central after investigation becomes automated. Lopatecki predicts 10 times more tracing and 10 times more logging as agents become consumers of telemetry. This is a forecast about instrumentation, not a measured improvement or a prescribed collection target. His reasoning is that additional logs once became noise because people could not sift through them; agents could make much denser instrumentation useful, potentially covering nearly every part of some software paths.

The execution model is less mysterious than an opaque SRE-for-AI product might suggest. Take the local debugging experience in Claude Code or Cursor and run it periodically. Choose the harness, sandbox, and skills, with prebuilt options where useful. Signal’s role is to put that combination into a recurring system, rather than require a wholly different way to debug.

The product walkthrough uses a financial agent that answers questions about stock trading. Lopatecki explicitly discourages treating such agents as a reliable way to make money; the workload is useful here because it has many ways to fail. The interface is Arize AX, its SaaS platform, which can also be deployed in a VPC. He describes Phoenix as the open-source option and says Signal is available in AX at the time of the talk. The current Phoenix repository specifies Elastic License 2.0; subsequent AX packaging also distinguishes basic Signal access from the managed-agent and repository features needed for the complete workflow.

Signal periodically produces findings that can lead to several different artifacts:

• GitHub issue: Connect a repository and create an actionable record of the problem.
• Evaluator: Turn a discovered failure into a check intended to catch it again.
• Dataset example: Preserve the relevant case for subsequent evaluation.

Each finding has supporting evidence, including traces. Skills for Google Cloud and other logging systems let the investigation combine data from multiple places, and users can supply their own skills. The displayed examples include runs without guardrails, safety problems, and intent problems. The output need not be a code patch to improve the next debugging or evaluation cycle.

The configuration screen exposes the execution choices. The demonstrated harness is Claude Code, and Lopatecki names Claude managed agents and Arize sandboxes as options. Sandbox selection matters when an investigation must connect to production systems or a database: some customers do not want those connections exposed to Anthropic.

Lopatecki describes Arize installations inside customer VPCs, naming Uber and another customer he calls Bookings. The architectural point is the connection boundary: customers may be comfortable granting access within their own VPC while declining an external production connection. He also names Daytona among the sandbox choices, noting that existing relationships and comfort with a provider influence the decision.

The platform then tracks the agents that run: periodic Signal investigations as well as other named agents. A user can inspect a session, download its transcript, and resume a Claude session locally. The investigation therefore has a handoff path when automated work reaches a point that needs direct human guidance.

Prompts control the investigation’s emphasis—for example, asking it to be less aggressive or to look for security issues. Presets combine those instructions with prebuilt or custom skills and linked repositories. Together with evaluations, these controls connect the local debugging workflow to a recurring process that produces a fix or a prepared starting point. Observability extends beyond reporting a signal into supporting the work that follows it.

An audience question makes the architecture’s implication explicit: why not connect Claude Code to the data, let it read the traces, and have it push a PR? Lopatecki agrees that users should do this. The work that makes it effective is designing the skills around the data, not withholding access behind a special agent.

A useful skill first finds the right evidence—for example, a group of traces belonging to one session—and then places that evidence in a repository file. Coding harnesses work well with files, so the skill should produce a usable record of what happened rather than an undifferentiated data dump. Lopatecki reports that some repository evidence files are about 10 MB. That is an operational example, not a demonstrated context-size limit. Multiple skills must then compose so the agent can move from an initial symptom to the evidence needed for diagnosis.

His Pyroscope example shows why the access surface matters. A skill can investigate memory issues, use profiling facets, and group evidence by customer to determine whether one customer is driving the problem. These are meaningful investigative operations. Simply giving Claude access to all the data does not teach it which cohort to isolate or which profile to inspect; the skill must expose those operations in a form the agent can use effectively.

The final question asks where evaluations enter a workflow that begins with a production failure. Online evaluations run on production traces, and their results become data attached to those traces. Investigation skills can retrieve those results, visualize them, and obtain aggregate evaluation values across a group of traces.

Lopatecki describes first-generation LLM-as-a-judge evaluations as an AI assessment layer that runs periodically and adds preprocessed information. Signal uses those annotations alongside the raw trace data. The evaluation provides an assessment of behavior; the trace still supplies the execution evidence needed to investigate it.

Evaluators often start with failures already encountered: a prompt-injection attempt to catch, or a problematic response pattern. When an investigation discovers another recurring failure, the team can create an evaluator to detect it next time. This closes a concrete feedback loop: production behavior supplies evidence, investigation identifies a failure pattern, and a new evaluator makes that pattern visible in future runs.

The two layers can operate at different scales. Lopatecki reports customers applying LLM-as-a-judge evaluations across their full datasets, while Signal performs more periodic investigations over substantial amounts of data. He supplies no dataset sizes or throughput measurements for that comparison. Broad, recurring assessment and deeper, periodic investigation therefore serve complementary roles: the former keeps checking behavior, while the latter assembles the context needed to understand and address failures.