# Why We Killed Our Multi-Agent Pipeline — Subbiah Sethuraman and Abhilash Asokan, ZS Associates

Subbiah Sethuraman · Abhilash Asokan

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/u6jJcIFDLE4-why-we-killed-our-multi-agent-pipeline) · [Video](https://www.youtube.com/watch?v=u6jJcIFDLE4)

## Why We Killed Our Multi-Agent Pipeline

A pharmaceutical analytics pipeline found the right cause but recommended the wrong action. The redesign separated statistical detection from investigation and used a domain graph to constrain the search.

Familiarity with language-model agents, database queries and basic graph concepts will help; the pharmaceutical workflow is introduced in the article.

### Why are prescriptions falling?

[0:00](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=0s)

Why are doctors writing fewer prescriptions for a drug, and what should the company do about it? That is a commercial analytics question: after drug discovery and clinical trials, the work shifts to getting treatments to patients. At ZS, Subbiah Sethuraman, who introduces himself as head of AI Engineering, and Abhilash Asokan, Director of AI Engineering, work in a setting that includes brand performance across markets, sales-rep engagement, patient adoption and therapy switches.

An analyst moves through four connected steps:

1. **Signal:** Detect a change, such as a drop in prescriptions.
2. **Why:** Investigate whether a competitor entered the market, payer coverage declined or sales reps failed to communicate the drug’s benefits.
3. **Action:** Choose an intervention that addresses the cause—for example, increase rep coverage if a region is underserved.
4. **Outlook:** Estimate how the intervention could affect brand or sales performance.

The steps form a causal chain. The action depends on the explanation, and the outlook depends on the action.

![Slide titled “How an analyst works” shows four numbered boxes linked by arrows: Signal, Why, Action, and Outlook.](https://ai.engineer/api/talk-essay-media/e1d0de24400c4862210332c74ca274337b95147eab78c051b0e3b50c97bed58e)

The analyst workflow: Signal → Why → Action → Outlook.

### Turning the workflow into agents

[2:21](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=141s)

ZS initially translated this human workflow into specialized agents. An orchestrator connected the stages, while the explanation step split into two distinct jobs: locating the decline and identifying its driver.

| Agent | Responsibility |
| --- | --- |
| Signal detection | Identify changes worth investigating |
| Source localization | Locate a national decline within a region or payer |
| Driver attribution | Identify why performance declined |
| Synthesis | Propose an action and its outlook |
| Orchestrator | Connect the agents |

This division mirrored the analyst’s sequence, but it also divided responsibility for understanding the result.

![An Orchestrator box connects to four boxes labeled Signal Detection, Source Localization, Driver Attribution, and Synthesis. Source localization and driver attribution share the heading “Why it’s happening.”](https://ai.engineer/api/talk-essay-media/8bb646c731e92aa72a5cf2b6229a2013acf650b7c7fd907a4876205b050ea42d)

An orchestrator coordinates signal detection, source localization, driver attribution, and synthesis.

### The right cause led to the wrong action

[3:18](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=198s)

The pipeline produced an information packet with a signal, a reason, an action and an outlook. In Sethuraman’s illustrative packet, brand prescriptions fall 18% in a territory over roughly four weeks. The explanation identifies a payer coverage change that makes the drug more expensive for patients. Yet the recommended action is to send more sales reps to doctors, followed by a forecast of improved sales.

The packet looks complete until its parts are read together. More rep visits do not address the identified affordability problem. Because the action fails to address the cause, the recovery outlook loses its foundation too. **Locally plausible outputs did not produce a coherent recommendation.** No single agent owned the complete connection between evidence, cause, intervention and expected result.

Sethuraman attributes the failure to how the work was divided, rather than to the language model itself. Three architectural problems emerged in sequence:

- **Unnecessary model judgment:** A sales drop can be detected with statistical methods; a language model need not decide whether it exists.
- **Context lost at handoffs:** Driver attribution found the coverage problem, but synthesis failed to preserve its significance for patient affordability.
- **Missing shared domain knowledge:** The agents lacked a common understanding of business metrics, their relationships and why they rise or fall. Even a metric such as `TRx`, total prescriptions, needs that context to be useful in an investigation.

### Observe the investigation before redesigning it

[6:01](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=361s)

The first instinct was to redesign the topology, skills, tools, handoffs or schemas between agents. Instead, Asokan’s team opened an empty directory, ran [Claude Code](https://code.claude.com/docs/en/sub-agents) with Bash and database access, supplied an actual signal they had already identified, and watched how it investigated. The experiment began with the work itself rather than a new arrangement of agent roles.

The shell entry point for that kind of observation is small. Database access must already be available in the environment; the prompt gives the agent an investigation rather than asking it to discover a signal:

```bash
mkdir pharma-investigation
cd pharma-investigation
claude 'Investigate the supplied prescription-decline signal using the available database. Examine where the decline is concentrated and which evidence supports or contradicts possible causes.'
```

The important output of this exercise is the sequence of operations the agent chooses, which can reveal where a dedicated tool or a deterministic process belongs.

Signal detection was the first responsibility to move out of the agent. The team had observed inconsistent behavior: sometimes the agent applied statistical methods, sometimes it barely inspected the data, and sometimes it confused noise with a signal. They replaced that judgment with an automated statistical workflow.

The new pipeline scans each KPI for anomalies and broken trends, applying guardrails, thresholds and prioritization before any investigator starts. It places selected signals on a queue; arrival of a signal wakes the agent. **The agent investigates a detected change; it does not decide whether the change qualifies as a signal.** This boundary gives the investigation a defined starting point.

### Keep parallel work, centralize judgment

[7:59](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=479s)

To address incoherent recommendations, the team consolidated the investigation around one main agent. Observing Claude Code also exposed repeated operations—writing a function and querying the database—that they packaged into a tool. The redesign reduced the number of reasoning handoffs without requiring all work to run serially.

Claude Code sometimes launched subagents dynamically for focused tasks, and ZS retained that pattern. A main investigator could delegate a question such as whether rep activity had changed in a particular region, then use the returned findings in the larger investigation. Subagents may reason within their tasks; the architectural rule is that **the main agent retains final judgment** about how their findings fit together.

That produced a lighter architecture, but consolidation alone did not supply business knowledge. A single agent could still misunderstand the relationships among entities and metrics. The remaining problem required an explicit domain model.

### Make domain relationships explicit

[9:42](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=582s)

The agent had been inspecting database tables and inferring relationships from them. Asokan describes that approach as difficult to scale and prone to producing relationships that did not exist in the data. ZS brought in its commercial analytics domain experts to map the domain into a **knowledge graph**.

The graph connects geographic entities to payers and accounts, connects those entities to brands, and connects brands and payers to KPIs. It also represents relationships among primary, secondary and tertiary KPIs, including which metrics can drive others. This supplies more than entity definitions: it gives the investigator a structured account of which relationships deserve examination.

![A graph links National, Region, Territory X, Account, and HCP with Brand A, an NRx decline signal, rep reach, abandoned fills, patient cost, coverage tier, and Payer Y.](https://ai.engineer/api/talk-essay-media/12fcd3a0fa6e3705a184747f268e1891a035dae60c15060b90811fc0d3789d11)

The business-domain graph structures the investigation around geography, entities, drivers, and a signal.

### Use the graph to bound the investigation

[10:51](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=651s)

The graph’s operational role is a **control plane**: it determines what the agent can examine, which paths it can take and which hypotheses it can evaluate. It is not simply another source from which to retrieve facts.

Consider a national decline in `TRx`. The first question is where the decline is concentrated. It might be in a region, a territory or a combination involving a territory, payer or account. Exploring every combination creates a large search space. The graph guides source localization through the relevant dimensions.

Once the agent narrows down the location, it investigates why the decline occurred. KPI relationships provide the next candidate explanations: which metric could be driving the observed change? In Asokan’s design, each edge is a hypothesis the agent can evaluate, and the investigation stays within that graph-defined surface. An edge supplies a candidate explanation, not evidence that the explanation is true.

The resulting loop ties graph navigation to the underlying data:

1. Start with an entity and inspect its graph neighborhood.
2. Treat the neighboring edges as candidate hypotheses.
3. Query the original data and examine the actual numbers.
4. Reason about whether the evidence supports or contradicts each hypothesis.
5. Follow a supported relationship and repeat the process.
6. Stop when the available hypotheses are exhausted or a root cause is found.

The graph bounds the search, while the data determines whether a proposed path deserves further investigation.

### An architecture derived from the work

[13:28](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=808s)

Asokan reports that investigations taking 50-plus turns and substantial tokens could produce analysis in roughly 20–30 minutes that might take an analyst three or four weeks. This is an approximate comparison from their pharmaceutical commercial analytics work; the talk supplies no evaluation dataset, quality criterion or cost breakdown. The shorter turnaround still involved a long investigative loop.

The closing design principle is to let the observed work shape the architecture instead of imposing the human workflow’s boundaries on it. In this system, deterministic operations establish the signal, and agentic work investigates it. One agent owns the reasoning end to end while invoking subagents, tools or skills for particular tasks. The graph then governs navigation and the next investigative decision. That places responsibility for the complete explanation in one agent while keeping its search grounded in explicit domain relationships.

![Key Takeaways slide lists four principles: architecture should be derived, not designed; separate the deterministic from the agentic; give one agent end-to-end ownership of reasoning; treat the graph as a control surface, not just a lookup layer.](https://ai.engineer/api/talk-essay-media/05c4a148169e1507a41c7870c72c89f49905b393730ecb2af8ad81edb8051aba)

Four takeaways: derive the architecture, separate deterministic work, give one agent reasoning ownership, and use the graph as a control surface.

## Resources

- [Claude Code subagents](https://code.claude.com/docs/en/sub-agents) — Current guidance for delegating focused tasks to Claude Code subagents with separate contexts, tool access, and permissions.
- [Understanding formulary tiers](https://www.medicare.gov/health-drug-plans/part-d/what-drug-plans-cover/how-drug-plans-work) — Medicare’s explanation of drug formularies, tier levels, and their relationship to patient copayments.

## Slides

### Why We Killed Our Multi-Agent Pipeline

Subbiah Sethuraman and Abhilash Asokan share mistakes and lessons from building multi-agent pipelines for pharma commercial teams.

[0:21](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=21s) [Title slide introducing lessons from building AI for pharma commercial teams.](https://assets.ai.engineer/sites/aiecode2025/sha256/72d92dd79c6be9ea5cde367bb071fe9d1203dc27f6ff49d0e2550007f6dc6287.webp)

Why We Killed Our
Multi-Agent Pipeline

Lessons from building AI for pharma commercial teams

Subbiah Sethuraman · Abhilash Asokan
ZS Associates
AI Engineer World's Fair 2026

### How an analyst works

A drop in prescriptions can prompt an analyst to check the signal, investigate its cause, decide on an action, and assess the outlook for sales.

[1:23](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=83s) [Four sequential analyst questions: signal, why, action, and outlook.](https://assets.ai.engineer/sites/aiecode2025/sha256/55a336ddaa185f74bdd6f1fe3d71f276b0ce900527fd50ff6e8be47b238dc43d.webp)

How an analyst works
Four questions, answered in order.

1
Signal
Is the change real?

2
Why
What's causing it?

3
Action
What should we do?

4
Outlook
What happens next?

### The agentic system

An orchestrator connects four agents: one detects signals, two locate their source and attribute the driver, and synthesis determines action and outlook.

[2:21](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=141s) [An orchestrator directs signal detection, source localization, driver attribution, and synthesis agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/026b66e1288a66809b46b310f35cb8220ff248b23c7986541aef9d4ea7de2115.webp)

We built an agentic system to mimic that
One agent per step, an orchestrator running the team.

Orchestrator
runs the team

SIGNAL
Signal Detection

WHY IT'S HAPPENING
Source Localization
Driver Attribution

two agents to answer one question

ACTION + OUTLOOK
Synthesis

### The packet it generated

Brand A’s new prescriptions fell 18% in Territory X over four weeks; almost all the drop was tied to Payer Y’s worse coverage tier. The packet recommends more sales visits and calls the outlook healthy heading into next quarter.

[3:21](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=201s) [An example packet reports a prescription decline, an insurance coverage cause, a sales-rep action, and a healthy outlook.](https://assets.ai.engineer/sites/aiecode2025/sha256/9dc6e81a30f9a009bb4425d7bce7416d63c68f43cf74e606327d847f2bcf80de.webp)

The packet it generated

SIGNAL
Brand A's new prescriptions dropped 18% in Territory X over four weeks.

WHY
Almost all of the drop is in one insurer, Payer Y. They moved Brand A to a worse coverage tier, so patients now pay more — and many walk away at the pharmacy counter.

ACTION
Send sales reps to visit more doctors in Territory X to push prescriptions back up.

OUTLOOK
The overall outlook for the brand is healthy heading into next quarter.

### It looks correct. Until you read it closely.

The pipeline identified an insurance affordability problem, then recommended more doctor visits and called the situation recoverable if rep activity improves. No single agent saw the whole picture.

[4:05](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=245s) [The identified insurance cause is marked correct, while the proposed action and outlook are marked incorrect because the action does not address the cause.](https://assets.ai.engineer/sites/aiecode2025/sha256/089ea3cf50d717c3071c9f90cd5cf1b99e067bebb7ac69d37e1e7361cfd2ad35.webp)

It looks correct. Until you read it closely.

✓ THE CAUSE
Patients can't afford the drug after an insurance change

✕ THE ACTION
Reps should visit more doctors — which says nothing about insurance

✕ THE OUTLOOK
“Recoverable if rep activity improves”

And here's the unsettling part
The action doesn't match the cause.
Nothing was made up. Every fact is real — but no single agent ever saw the whole picture.

### Why it failed

The pipeline asked an LLM to judge a statistical signal, lost the reasoning behind the cause at handoffs, and gave agents no shared understanding of what drives prescriptions.

[4:40](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=280s) [Three problems: a language model decided the signal, context was lost at handoffs, and agents lacked shared business understanding.](https://assets.ai.engineer/sites/aiecode2025/sha256/57a3ec6fd191ca4c5a9d7e62f6c9ac1ef714d3a41e58fe82881116a2b8fa5626.webp)

Why it failed

Not the model. The way we split the work.

1 A language model was deciding the signal
We asked an LLM whether an 18% move was a real signal or normal noise — a question statistics should answer, not a model.

2 Context was lost at every handoff
Each agent passed on its conclusion, not its reasoning. The clue that explained the drop never reached the agent writing the advice.

3 No shared understanding of the business
Each agent inferred how pharma metrics relate from raw tables. Nothing told them what actually drives prescriptions, or in which direction.

### Deriving the architecture

The team gave Claude Code Bash, a database and a real KPI decline, then watched its traces to derive what the agent touches. They decided where the math lives up front.

[5:59](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=359s) [A bare agent uses Bash and a database to investigate a KPI decline while its behavior is observed.](https://assets.ai.engineer/sites/aiecode2025/sha256/f9fc38952d17a9290085a6f0bf029928e4e10498f9a3932c744a1b187cfba91a.webp)

So we stopped designing the architecture — and started deriving it.

A bare agent.
Bash and the database
One real KPI decline.
Then we watched.

What the agent touches, we derived from traces. Where the math lives, we decided up front.

### Statistics finds the signal

A deterministic workflow tests and prioritizes signals before the agent starts. The agent then investigates the signal, reasons about its cause, and writes a recommendation.

[6:54](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=414s) [Two columns contrast deterministic signal detection with the agent's investigation and explanation.](https://assets.ai.engineer/sites/aiecode2025/sha256/cd191fd470af310d23180d60cffb29df701fb80a83878ce6e8e23decfa374973.webp)

FIX 1 · Language model determined the signal

Statistics finds the signal. The agent explains it.

STATISTICS
Find the signal.

Is an 18% drop real, or just noise? Deterministic tests and guardrails decide — before the agent starts.

THE AGENT
Explains it.

Once the signal is real, the agent investigates, reasons about cause, and writes the recommendation.

A language model is a reasoning engine, not a statistics engine.

### What replaced the pipeline

One analyst agent owns the investigation and judgment, delegating focused tasks to sub-agents. A knowledge graph supplies pharma context.

[7:57](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=477s) [Signal detection and a knowledge graph feed an analyst agent, which uses sub-agents for isolated investigations.](https://assets.ai.engineer/sites/aiecode2025/sha256/15f05f6bc7fab835648e11f6609052995ab7970602b80fb0ccb86a4b9a5cfdd2.webp)

What replaced the pipeline.

FIX 2 · context was lost at every handoff

Knowledge graph
Signal Detection
Analyst agent
Sub-agent
Sub-agent

launched for one isolated task — e.g. verify a driver branch: did rep activity really fall?

One agent owns the whole investigation. Sub-agents investigate. Knowledge Graph provides pharma context

### The business domain as a graph

The graph localizes an 18% NRx decline to Territory X, then traces possible drivers through coverage, patient cost and abandoned fills. It also connects rep reach to the investigation.

[9:27](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=567s) [A graph connects geographic entities, a prescription decline and potential drivers including coverage, patient cost and abandoned fills.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc7d140ce6d013bbb7ab38418320f77c1b6b183325499c65067f3e7ca42f372c.webp)

FIX 3 · No shared understanding of the business

We gave the agent the business domain — as a graph.

WHERE · localize first
National
CONTAINS
Region
Territory X
Account
HCP

Brand A
OF
NRx ↓ 18%
measured at
Rep reach ↓ small
+ small

WHY · only after WHERE
Payer Y
SETS
Coverage tier ↓
+ raises
Patient cost ↑
+ raises
Abandoned fills ↑
− suppresses

Legend: geography · entity · driver · signal

The graph didn't just add context. It structured the investigation.

### The graph as the agent's control surface

The agent ranks children to locate concentration, then follows driver KPIs and checks evidence for each edge. Graph edges are hypotheses, not proof of cause.

[10:56](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=656s) [A graph contract and where and why protocols guide investigation; each edge remains a hypothesis requiring evidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/f6fe02dcb5e66408d8eb80555bd0e7159676aea0b5afb62c39e6be7323e726f4.webp)

The graph is the agent's control surface.

FIX 3, THE MECHANICS · the contract the agent must use

GRAPH CONTRACT THE AGENT MUST USE
(:Entity)-[:CONTAINS]->(:Entity)
(:KPI)-[:EXISTS_AT]->(:EntityLevel)
(:KPI)-[:HAS_DRIVER_CATEGORY]->(:DriverCat)
(:DriverCat)-[:HAS_DRIVER_KPI
    {effect, diagnostic_q}]->(:KPI)

WHERE PROTOCOL
rank children by contribution
return top_n = 5
stop if systemic / not concentrated

WHY PROTOCOL
walk driver KPIs, max_depth = 6
cycle guard
classify SUPPORTS / CONTRADICTS
    from deterministic evidence checks — read, not improvised

Every edge is a hypothesis the data still has to support — not proof of cause.

### The agent's investigation loop

The agent checks graph edges against real numbers, keeping or rejecting each hypothesis before repeating. The illustrated surviving path runs from lower tier through higher cost and abandonment to lower NRx.

[12:32](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=752s) [The agent traverses the graph, queries numbers and reasons about each edge, repeating before concluding with a surviving path.](https://assets.ai.engineer/sites/aiecode2025/sha256/9c1343d6ac43ae96d128c1ddb2bbb3ed5fb884f81a40fe76aa4120589e6b3e62.webp)

FIX 3 · the contract, running

The agent runs this loop.

repeat — next candidate

TRAVERSE
the graph proposes the next move

QUERY
pull the real numbers

REASON
keep or kill the edge

CONCLUDE · the surviving path: tier ↓ → cost ↑ → abandonment ↑ → NRx ↓

Traverse. Query. Reason. Repeat.

### Key Takeaways

Derive the architecture, separate deterministic work, and give one agent end-to-end ownership of reasoning. The agent uses the graph to navigate and choose its next move.

[13:43](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=823s) [Four takeaways: derive architecture, separate deterministic and agentic work, give one agent ownership of reasoning, and use the graph as a control surface.](https://assets.ai.engineer/sites/aiecode2025/sha256/89c098efd5ef080009ccadd5d2635fd968faa9d6d2ac26755e172bca6e8624db.webp)

Key Takeaways

1 Architecture should be derived, not designed
2 Separate the deterministic from the agentic
3 Give one agent end-to-end ownership of the reasoning
4 Treat the graph as a control surface, and not just a lookup layer

Subbiah Sethuraman · Abhilash Asokan · ZS Associates

### AI Engineer World's Fair

[14:55](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=895s) [White AI Engineer World's Fair logo and website on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf3347691de1d801e32b1f91dd35e73bd7fbbf657f6ff3f344c9b84abb4b94fb.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=0s) [upbeat music] Uh, okay, so I'm Subbiah. I head AI Engineering at ZS.

[0:17](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=17s) Uh, I'm Abhilash. I'm Director of AI Engineering at ZS.

[0:20](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=20s) So ZS, we are a tech firm. We work with many of the top companies in the world, including a lot of the t- top pharmas, actually. Um, so today's talk, I think we wanted to, as I think already introduced, right?

[0:29](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=29s) We wanted to talk from our experience, right? Building multi-agent pipelines. Uh, what are the mistakes we did, right? And what did we learn and how did we fix them?

[0:38](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=38s) So, um, so I'm going to orient it more on pharma commercial domain and for people in the room probably who are not aware, quickly, uh, pharma has two main functions.

[0:46](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=46s) One is R&D, right, the drug discovery, right, uh, and the clinical trials part. And then there's the commercial. How do you take a drug to a patient, basically. And, and within commercial, there are different functions.

[0:58](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=58s) Like, once you create a drug, right, um, what is the performance of a brand? How is the drug performing in different markets, right? Uh, then there are things around your field force, your reps.

[1:07](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=67s) How effectively are they engaging, uh, right, with everyone? There are things around patient journey. How are patients are adopting a drug, right? I think if there is any therapy switch which is happening.

[1:17](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=77s) So as you can think about, there is a lot of analytics which really happens in a commercial domain. And h-how do typically analysts work?

[1:25](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=85s) So there are four steps, right, what analysts do, right? So first, there is always something called a signal detection, right? So signal can be something like, okay, the prescriptions what a doctor is writing.

[1:35](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=95s) Maybe is there, is there a drop in the prescription? So that's a signal. So once you've got a signal, the second thing what an analyst does is why is this signal really firing?

[1:45](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=105s) What is the reason for it? So is it, like, there is a competitor drug which has come in? Because of that, is it, is it reducing? Is it, is it because maybe a payer coverage for the drug has reduced, or maybe the, the reps on the ground, there's no proper, uh, they're actually not taking the benefits to

[1:58](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=118s) the doctors. And once you arrive at the reason, the next step becomes, okay, what is the action do you take? So if reps-- suppose if reps, the coverage is not good in a particular region, should-- do we have to increase that?

[2:12](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=132s) And once you do that, what is the, what is the outlook, right? Is my brand-- is my sales performance, is, is it going to improve? Right? So these are the four things which happens.

[2:21](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=141s) Now, for some of the top pharmas, what we have done is how do we in an agentic way, right? I think how do we actually mimic this, this analytics work?

[2:29](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=149s) So what we did, we built agents for every step, right? Signal detection. We said, "Okay, we'll have an agent for signal detection." It'll identify, identify the signals for me.

[2:39](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=159s) Second, what are the root cause, right, for the signals, right? So in this case we have two agents. One we call it a source localization. So for example, if my sales is dropping at a national level, is it because it's dropping at, say, a particular region, or is it dropping for a payer?

[2:53](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=173s) So we need to understand that. We are, we are trying to identify the source of it. And then once we understand what is the real reason, right, I think the, the sales performance has gone down.

[3:03](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=183s) That's another agent, the driver attribution agent. And then the last step is your synthesis, right? So depending on the cause, now what is the action you have to take and what is the outlook?

[3:11](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=191s) Like, typically how an analyst used to do. And all of this, we used to have an orchestrator agent which connects all these agents together.

[3:18](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=198s) So now what happened once we had this system, what did it generate, right? It generates something, an information packet like this, right? It clearly tells you the signal, right?

[3:27](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=207s) So first it says maybe my brand's prescriptions have dropped eighteen percent in some territory, right, in some time frame, maybe four weeks. It then tells you the reason. Why did it, why did it drop, right?

[3:38](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=218s) The reason it says because a payer actually, right, the coverage for this drug has actually-- they moved it to a lower tier. So for patients it's expensive actually to, to buy this drug.

[3:49](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=229s) The action it says, okay, because doctors are writing less prescriptions, maybe send more sales reps to talk to doctors, right, and increase the number of, uh, prescriptions which you're writing.

[4:00](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=240s) And then if you take this action, maybe your outlook, your sales performance is, is going to increase.

[4:05](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=245s) So all of this looks good high level, but then if you, if you look at, at it closely, it's not very coherent, right? The cause is right. It identified the right cause, right?

[4:14](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=254s) Because patients can't afford the dug- drug, right? But the action it said, it didn't really focus on the payer part, the insurance part of it, right? It just said reps, right?

[4:24](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=264s) Uh, focus more reps actually, right? And then the outlook. Because the action is wrong, the outlook is not going to match. So at each level if you see, it is actually derived the right fact, but then there is no single agent which is owning-- which understands the end-to-end picture, basically.

[4:40](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=280s) So why did this happen? Like, why did it fail, right? So obviously it's not the LLM which failed, right? It's the way how we split the work, right? Because we tried mimicking the analyst behavior and we did it.

[4:53](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=293s) The first, first key issue is, like, a language model is actually det-determining your signals. So signals like things like your sales drop is a simple information which you can use statistical methods to actually go and fetch this information.

[5:05](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=305s) You don't need a language model actually, right, to fetch this information. Second is as your multi-agents, there is a lot of context handoff which is happening and context is actually getting lost at each of these handoffs.

[5:17](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=317s) So for example, the driver attribution agent is actually determining the right cause, but then the next agent, the synthesis agent, is actually is not able to understand why is that, right?

[5:27](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=327s) The, the, the patients are finding the drug to be expensive. It's not understanding the weightage of an insurance coverage going, going down. So that is a key information, right, which is getting lost.

[5:37](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=337s) And the last big piece is there is no shared understanding of the business domain knowledge for all these agents. All these agents don't understand metrics, right? So things like TRX, the number of transaction-- uh, number of prescriptions which a doctor writes, right?

[5:50](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=350s) What is the relationship between them? Why does it go up or down? So those are the, the reasons. So then what did we do? Uh, so I called Abhilash to come and solve for this.

[6:01](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=361s) Uh, thank you, Subbiah. So we had, like, three problems. Uh, so we are thinking, how are we going to solve this? Uh, our first instinct was we'll go back to the drawing board.

[6:12](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=372s) Uh, we'll start designing it again. So maybe the topology was wrong, the skills were wrong, the tools were wrong, or maybe the-- maybe we have to de-de-define a better handoff-- design a better handoff or a better schema between agents.

[6:27](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=387s) But we took a step back. We didn't do any of that. We, like all of us, we went back to Claude Code. So we opened a very pla-plain, empty directory and ran Claude Code, then give it just bash and the database.

[6:43](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=403s) Then give it an actual signal which we identified, then started observing what it is doing.

[6:49](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=409s) So while we look at what Claude does, we are able to figure out fix for all the three issues we discussed.

[6:55](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=415s) The first, the first part. So what was happening was the agent was

[7:01](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=421s) looking at data and deciding on a signal.

[7:05](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=425s) Sometimes it applied some statistical methods, sometimes it barely look at the data and say, "This is a signal." Sometimes it's actually a signal, sometimes it's a noise. This is something we don't want an agent to do.

[7:16](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=436s) This is a completely a deterministic workflow. So we separated it out from the agentic system.

[7:22](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=442s) So we built a pure deterministic workflow with different statistical methods. We put guardrails, we put thresholds, we put prioritization. Everything happened before the agent even kick-starts. So we ran an automated pipeline which scans through the data, identify signals for each of the KPI.

[7:41](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=461s) Is anything happening with that? Any anomalies which is happening? Any trend which is very, uh, any trend which is breaking? Based on that, we identify the signal, we put it on a queue.

[7:50](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=470s) The moment the signal comes to the queue, the agent wakes up. So the agent's job is to investigate, not to identify.

[7:59](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=479s) The second part. So mainly the issue which we previously, what Subbiah was mentioning, there is no coherence in the output which the agents produced.

[8:10](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=490s) So we started consolidating. We look at Cl- how Claude Code operates. It's able to do a lot of operations, so we started designing around that. So it's repeatedly writing a function and querying database, so we give it a tool for that.

[8:25](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=505s) So this consolidated the entire process into a single agent. That doesn't mean that we didn't do parallelism. We still do parallelism. What we removed is, do we need distributed reasoning?

[8:36](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=516s) We didn't want the judgment to be distributed between agents. That, we wanted to consolidate to a single agent. So that was exactly what we did. Then occasionally we observed Claude Code.

[8:46](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=526s) It's launching sub-agents dynamically for a very particular focused task. So we did the same thing. Because if-- suppose you want to understand if rep activity in a particular region, that is an investigation which you need to run.

[9:00](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=540s) That, you can still delegate to a sub-agent. You can get back the, uh, results back, not the reasoning or the judgment. That is still controlled by the, by the main agent.

[9:09](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=549s) But the investigation part of it, we de- we're delegating it to a sub-agent. So these are some of the things we kept based on observing what Claude was doing-- Claude Code was doing.

[9:18](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=558s) So that give us a more lighter architecture than what was initially there. But still it doesn't solve the problem. That's why you see a knowledge graph in the diagram.

[9:28](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=568s) So it still doesn't have the business context. It still in-- doesn't understand all the entities, the domain, the KPIs, how do they relate to each other.

[9:42](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=582s) So that's something we wanted to solve for because the agent was looking at data, looking at tables, then trying to infer the relationship that-- which was not scalable. And it often produced relationship which i-- which is not actually exists in the data.

[9:57](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=597s) So what we did. So we've been in this-- working in this field for a lot, lo-whole, whole lot of years, so we've been doing this for our clients. So we have a lot of domain experts who understand the pharma domain very well.

[10:09](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=609s) We are doing commercial analytics for the clients. So we start, sat with them, start building a knowledge graph. We sta-start trying to map out the domain.

[10:19](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=619s) So we are able to build a knowledge graph. We are able to identify different entities, their relationship between that. So if you see, there are, like, geographic entities. There are payers.

[10:29](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=629s) How, how geographic entities connect to a payer or an account? How does this connect to a brand? Then from ba-brand or payer, how does it go to a KPI?

[10:38](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=638s) How does a KPI relate to like a, uh, secondary KPI, tertiary KPI? How does one KPI drives another KPI? So we started mapping out all of this information and then creating our, our knowledge graph.

[10:51](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=651s) Once you have the knowledge graph, then we want to let the agent to navigate the knowledge graph. So the knowledge graph is not just something the agent looks up for data.

[11:01](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=661s) It is a control plane for the agent.

[11:04](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=664s) So what do you mean by that? So the knowledge graphs dictates what the agent can look into, what path it can take, what investigation hypothesis, uh, it can evaluate.

[11:17](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=677s) So for example, if you-- when you do this analysis, the root of the problem, say something like your TRX is declining at a national level. This could be when you do source localization, so that's our, uh, contract for how to find the where.

[11:31](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=691s) It might happen within a region. It could be concentrated in a territory. It would be concentrated in a combination of a territory or a payer or an account. So there are a whole lot of dimensions which the agent needs to evaluate.

[11:42](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=702s) Now, it's a lot of permission-- permutation combination. So the knowledge graphs guides the agent on how to find where this is concentrated. Then the why, why part of it.

[11:53](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=713s) Why it is happening? When you figure out that, okay, something is concentrated at, say, the decline is concentrated at a particular region, now you need to figure out the why This is where, where the, uh, KPIs and the relations come, comes in.

[12:05](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=725s) Once the agent is able to narrow down the where, then it's able to go and figure out the, um, figure out the why part of it, which KPIs is driving what.

[12:16](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=736s) Graph acts as the control surface. So agent, every edge is a hypothesis, so the agent can go and evaluate that hyposi- hypothesis. Uh, it doesn't go outside of this.

[12:28](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=748s) So that gives a more bounded surface for the agent to investigate.

[12:34](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=754s) So like everyone was mentioning about loop, so we also build a loop. So what the agent does. So the agent first start with an entity. It goes to the graph,

[12:46](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=766s) it looks at the neighborhood of the graph, then it, it figures out the edges. So it's got some hypothesis. It'll go, it'll go back to the original data, evaluate that hypothesis, look at the actual numbers, then reason over it.

[13:02](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=782s) Then it'll, it'll either find it, find it contradicting or it fi- finding supporting the, the evidence supporting the hypothesis. If it's supporting it, it started traversing through the graph.

[13:13](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=793s) So this repeats this until it run out of all the hypothesis or it's able to figure out the root cause. So this, this concludes this run. This runs and it'll be able to pr- fi-figure out the, uh, root cause of the problem.

[13:28](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=808s) So once we built this, so maybe like after like fifty plus turns, a whole lot of tokens, it's able to produce something an analyst, uh, was able to produce maybe in three or four weeks in like maybe twenty, thirty minutes.

[13:41](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=821s) Thank you.

[13:44](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=824s) So just to wrap it up, um, key takeaways. First thing is, I think we should not be introducing human constraints or design constraints into architecture. I think let the architecture be derived actually, number one.

[13:57](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=837s) Second, I think any complex workflows will have deterministic parts and agentic parts. Don't let agents actually run the deterministic part, right? So I think we need to break, break that off.

[14:08](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=848s) The third, you need to have one agent, right, which owns the reasoning end-to-end, right? This agent can actually take the call to use sub-agents, tools, skills to actually spawn off other, other tasks basically, right?

[14:20](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=860s) But you need one agent to own the, the reasoning. And the last, probably the most important, I think graph cannot be treated just as a lookup layer. I think graph has to be treated as a control plane which the agent uses to navigate and takes the next decisions basically.

[14:36](https://www.youtube.com/watch?v=u6jJcIFDLE4&t=876s) Okay. With that, I think thank you. Thanks. Thanks for attending this. [audience clapping] [outro jingle]
