Self Driving Products: Product Signals to Pull Requests

What if the data your product collects could produce a pull request instead of another dashboard to investigate? That is the starting point for Joshua Snyder’s walkthrough of the pipeline being built at PostHog. Product analytics, session replay, web analytics, error tracking and experiments already supply different views of what users experience. The next step is to turn those observations into work the system can perform.

The existing workflow has several waiting points. A product signal changes a dashboard metric. Someone notices, investigates and perhaps creates a Linear issue rather than fixing the problem immediately. Later, an engineer writes a patch, opens a PR, reviews it and ships. Snyder describes this manual signal-to-shipping process as taking a few hours to a few days. That is his characterization of the workflow, not a measured benchmark.

The proposed alternative starts a background agent when the signal arrives. The agent investigates and creates a GitHub PR, moving the developer’s attention from raw dashboards, errors and logs to a candidate fix. Human review remains one endpoint; for changes judged low risk, Snyder also proposes shipping immediately behind a feature flag. The intended handoff is a prepared change, not merely an alert.

Snyder reports that PostHog ingests trillions of events per month. This describes the platform’s incoming event volume, not the throughput of the agent pipeline. Much of that input is noise, and multiple observations may describe one failure: an error-tracking issue and a session recording can point to the same underlying problem. Sending each observation independently to a coding agent would miss that relationship.

The pipeline therefore separates five operations:

1. Ingest signals from different sources.
2. Group related signals into a report about a problem.
3. Research the cause and identify the relevant repository.
4. Assess whether the report is actionable.
5. Execute the task, create a PR and iterate until its checks are green.

Research and assessment sit between detection and implementation because an observed symptom is not yet a sufficiently defined coding task.

Some signal sources are public, which creates a prompt-injection path. An attacker visiting a website could deliberately trigger an error whose text instructs the downstream agent to publish the customer’s PostHog data. The pipeline begins with an LLM classifier that checks for malicious instructions and drops signals it judges unsafe. This is the team’s stated safeguard, not a guarantee that accepted content is safe.

Accepted signals then enter a common structure. An error may contain a stack trace, a log may contain JSON or plain text, and an experiment may contain results represented in a chart. Normalization gives each signal a source product, a type, content and an importance weight. A TypeScript representation of that described record is:

type Signal = {
  sourceProduct: string;
  type: string;
  content: string;
  weight: number;
};

These fields preserve where the evidence came from while giving later stages a consistent record to process. The pipeline then embeds the signal’s content.

Consider a null-pointer exception and a customer’s Slack message saying checkout is broken. Their formats differ, but they may be evidence of the same failure. Grouping brings those signals into a report. As signals join the report, their weights contribute to its importance; when the report’s weight exceeds a threshold, the pipeline promotes it and starts a research agent.

The team initially tried embedding the signals directly and clustering related vectors. In their experience, an off-the-shelf embedding model responded too strongly to structure. A checkout error and an onboarding error clustered together because both looked like errors, while a Slack message about onboarding remained with other Slack messages. Session replays formed another separate group. The grouping captured source format when the pipeline needed shared product meaning.

The replacement asks an LLM what each signal concerns and generates several search queries from it. The pipeline then matches those queries in embedding space, rather than matching the raw signals themselves.

For the onboarding example, the desired connection is between the onboarding error and the onboarding Slack message. Normalizing a record’s fields does not necessarily normalize what its embedding emphasizes. Snyder reports that query generation made grouping substantially better, without supplying a numerical evaluation. Once a grouped report is important enough, it proceeds to research.

The research agent runs the Claude Agent SDK inside a Modal sandbox. A promoted report supplies its starting evidence, but the agent can investigate beyond that initial bundle. Through PostHog’s MCP server, it can retrieve additional product data—for example, pulling logs while investigating a report that already contains an error and a session replay. Snyder credits that additional evidence with making the research more accurate.

The agent also receives codebase context and access to external MCP servers. Linear and Notion were especially useful for grounding the investigation in organizational context. Its output is a problem summary and a priority, plus a proposed reviewer identified with git blame. Research therefore prepares both the technical explanation and the human handoff for a possible PR.

A researched problem does not automatically become a coding task. The actionability step routes it according to what is still missing:

Returning a report to the pool preserves the opportunity to act later. The inbox handles a different constraint: a product decision may require a person’s judgment even when the available evidence is clear. Only the immediately actionable route proceeds directly to a fix.

The source affects how often a report reaches that last route. Specific error reports, including Sentry-style data, tend to give coding agents a well-defined problem. Slack messages and session replays often describe broader difficulties with several plausible solutions. Detecting that a user struggled is useful evidence, but it may not determine which product change should follow.

For an actionable report, the execution stage clones the user’s repository into a sandbox and runs the Claude Agent SDK to build a fix. The agent pushes a PR, but that first submission is not the endpoint. A failing CI check or a new PR comment triggers another sandbox run.

The pipeline saves a sandbox snapshot and rehydrates it when work needs to continue. Feedback can come from a reviewing agent as well as other PR comments. Snyder does not specify which snapshot mechanism the system uses; the mechanism described here is save, restore and continue, without assuming that live process memory survives.

The loop continues until the PR is green. Its intended overnight result is a set of PRs ready for review, rather than CI failures and comments that developers must pull into a local environment to address. Green checks are the target of this loop; they do not by themselves establish that a change is semantically correct or automatically merged.

The first lesson came from evaluation. Early experiments used the team’s own data locally, with informal checks of whether the results looked reasonable. That did not represent the variety of customer data the pipeline would encounter. Representative evaluations and visibility into production behavior are necessary to tell whether an iteration improves the system rather than merely looking better on familiar inputs.

The second lesson is to inspect what embeddings actually match. With mixed input formats, structural similarity can compete with semantic similarity. The failed clustering approach was a concrete instance of that problem: improving the representation required thinking about the content supplied to the embedding model, not just selecting a clustering method.

The third lesson is that an agent’s willingness to make a change is not evidence that the task is well defined. Give a coding agent a generic report that onboarding is broken and it will try to fix something. Without sufficient specificity, that behavior produces noisy PRs with little meaningful value. The actionability gate must be allowed to withhold a task even when the coding agent could produce a patch.

The fourth lesson concerns when to optimize. Snyder’s deliberately provocative phrase, “tokens are free,” comes with an immediate correction: they are not. Early concern about the enormous stream of signals led the team to avoid agents where possible or postpone their use until late in the pipeline. During experimentation, that constrained what they could discover.

Repeatedly running an agent on the same problem exposes both clever solutions and recurring patterns. Snyder uses 100 runs as an illustration of that exploration, not a required sample size. The exploratory pipeline was initially too expensive per PR, but observing repeated behavior revealed where an expensive agent step could become a one-shot LLM call or a faster trained model. The sequence is to discover a useful procedure first, then replace its predictable parts with cheaper operations.

At recording time, Snyder described the built system as an alpha pipeline turning product signals into grouped reports and merge-ready PRs, with rollout planned over the following months. The larger ambition extends beyond that PR workflow: developers would spend more time building features while the system handles routine bugs and experiments on pricing or onboarding.

That future loop would ship experiments and measure their impact. For straightforward changes, an agent could approve the PR and deploy behind a feature flag. If results were poor, the system could roll back the flag and remove the change from the codebase later. These are proposed extensions of the pipeline, rather than capabilities established by the alpha description.

The next improvement is learning from what happens after a PR is generated. A rejected PR, a deployment problem and an error resolved in production each provide different feedback for the next proposed change. Snyder identifies using those outcomes as an area for further iteration. The desired loop therefore reaches beyond generating code: it connects a product observation to a change, then uses the observed result to improve subsequent work.

For a product already producing substantial user-behavior data, Snyder’s closing recommendation is to experiment with agents on that evidence and inspect what they discover. The opportunity starts with data the product already has—and with finding out which useful investigations an agent can perform on it.