AI Engineer World's Fair 2026
Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase
Read the talk
Learned Execution Graphs for API Anomalies and Drift
Representing each request as an execution graph helps separate a slow service, a changed processing path and a shifting request mix—then choose a response without delaying payments.
From a talk by Ritvik Pandya
Before you start: Familiarity with API request flows, distributed tracing and basic latency monitoring will help; the article introduces execution graphs and drift categories as they arise.
What changed in this request?
When an API request deviates from normal behavior, how do you find what changed—and decide whether to fix it? Ritvik Pandya, who introduces himself as leading a payments team at JP Morgan, approaches this through short-lived execution graphs. Unlike persistent property graphs queried through systems such as Neo4j, these graphs describe how an individual request proceeds through a system. Their purpose is to make deviations visible and reduce manual investigation.
Consider a payment request. It enters through an edge layer, passes through gateways and ingress, and reaches authentication and authorization. An orchestrator then invokes downstream services, some in parallel, before the system notifies the client. In the illustrated graph, those parallel branches are Fraud Score, Ledger Write and FX Rate; they converge at Notify. Representing the journey as a directed acyclic graph, or DAG, preserves both the sequential dependencies and the parallel work.
The graph tells you which services execute in which order, but it also identifies the context available at each node and what passes to the next one. Retries and loops need explicit representation: Pandya treats their occurrences as separate entities so they remain trackable. An execution graph therefore records the work that happened, rather than collapsing every visit to a service into one indistinguishable node.
A compact TypeScript representation of the illustrated dependencies makes the parallel branches explicit. Each array names the steps that must precede that node; it does not impose an order among Fraud Score, Ledger Write and FX Rate.
typescript
type Step =
| "Edge GW"
| "AuthN/Z"
| "Orchestrator"
| "Fraud Score"
| "Ledger Write"
| "FX Rate"
| "Notify";
const predecessors: Record<Step, readonly Step[]> = {
"Edge GW": [],
"AuthN/Z": ["Edge GW"],
"Orchestrator": ["AuthN/Z"],
"Fraud Score": ["Orchestrator"],
"Ledger Write": ["Orchestrator"],
"FX Rate": ["Orchestrator"],
"Notify": ["Fraud Score", "Ledger Write", "FX Rate"],
};
function readySteps(completed: ReadonlySet<Step>): Step[] {
return (Object.keys(predecessors) as Step[]).filter(
step => !completed.has(step) &&
predecessors[step].every(parent => completed.has(parent)),
);
}
const completed = new Set<Step>([
"Edge GW", "AuthN/Z", "Orchestrator",
]);
const ready = readySteps(completed);
// ["Fraud Score", "Ledger Write", "FX Rate"]
Timing and request context can then be attached to these execution nodes, while repeated attempts receive distinct identities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend analysis effort where the baseline breaks
A detector should not spend the same resources on every request. Pandya compares the first tier to an airport boarding-pass check: if the request matches its normal end-to-end baseline, let it pass without escalating to a deeper inspection. A delay prompts the next question: did the structure change? An unexpected node or a missing processing step can explain why execution no longer matches the reference graph.
Further analysis can use KL divergence or an exponential moving average, but the comparison must reflect the request's context. Client A may be local, while client B's request arrives from outside the country and requires additional checks. Their normal latencies need not match. A threshold built around the appropriate client baseline helps avoid treating expected differences as incidents.
The decision sequence is straightforward:
- Represent the full request as a DAG.
- Establish its baseline and detect a deviation.
- Localize the deviation to the relevant part of the graph.
- Compare it with the threshold appropriate to that system and request.
- Alert or act only when the deviation exceeds that threshold.
Detection is not automatically a mandate for remediation. A localized difference that remains within the accepted threshold requires neither an alert nor an automated intervention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Localize the slow FX service
Return to the payment graph. The request is progressing through the system, but the foreign-exchange rate service is taking longer than usual. The graph gives the investigation a specific target: FX Rate, rather than the entire payment pipeline. Once that node is identified, previous cases of FX service failure become relevant evidence for deciding what to investigate or repair.
Pandya describes a pre-live benchmark example using OpenTelemetry and DeathStarBench, with millions of traces over seven days, followed by injected anomalies to train the system. This establishes the intended preparation sequence: observe ordinary execution, introduce faults and develop the detector before putting it live. The talk does not provide the selected workload, exact trace count, injection procedure or evaluation results for that example.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An incident, a new step or a new request mix?
An anomaly and drift pose different operational questions. Suppose the trip from home to the office normally takes an hour. One day it takes longer because of traffic or an accident: that is an isolated incident, and the response depends on how critical the disruption is. Now imagine noticing during a late-afternoon coffee break that the same commute routinely takes twenty minutes longer than it did a year ago. More cars and heavier Bay Area traffic have changed the pattern. The old baseline may no longer describe normal conditions.
Different kinds of drift call for different remedies:
- Structural change: A coffee shop starts asking whether you have a membership before applying discounted or regular pricing. That added question is a new processing step. Likewise, adding or removing a service node means the API's baselines and alerts must account for a changed path.
- Capacity-related degradation: Request volume grows until a service takes longer or cannot serve the expected load. Depending on the use case, the response may be to scale the service, add instances or make a call asynchronous.
Updating a baseline for an intentional extra step and addressing insufficient capacity are distinct operations, even if both first appear as increased latency.
A third case is covariate drift: the population of requests changes. Pandya's example begins with 60% local requests and 40% international requests. International payments may require currency conversion, so requests in US dollars and other currencies have different normal processing times. As the product becomes more popular abroad, the international share grows. The system can be working correctly while its aggregate latency changes.
Two responses are possible: compare local and international requests against separate execution graphs, or revise the aggregate request-time baseline to reflect the new mix. The distinction matters because a changing mix does not necessarily require a service repair. By contrast, if the same request now produces different behavior, the relevant response may be to reconsider or roll back a change. Classify the deviation before choosing the action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn diagnosis into a controlled rollout
The detection methods here form a statistical module within a larger system involving neural algorithms; the talk does not specify that larger system's learning procedure. The operational flow starts with OpenTelemetry continuously supplying data. Those observations support root-cause analysis and drift classification, which in turn inform a candidate remedy.
Before automating that remedy, assess its risk. Pandya illustrates a staged rollout: apply the remedy to 5% or 10% of machines, monitor and verify it, then expand to 100% of nodes. This separates selecting an action from authorizing its broad execution. A plausible diagnosis still needs evidence that the proposed change behaves as intended.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep analysis off the payment path
Pandya uses an illustrative normal end-to-end request time of 700 milliseconds to compare two latency patterns. One node becoming slow directs attention to that service. Delays spread across the graph instead suggest investigating something shared by the affected nodes. The graph helps distinguish these patterns even when both appear to the client as a slow payment.
The detection system must not become another source of payment latency. Telemetry is fed asynchronously into OpenTelemetry, with Kafka as a possible transport and stream assessment downstream. That processing can split into two paths:
| Path | Purpose | Tradeoff |
|---|---|---|
| Hot path | Fast decisions and possible automation | Prioritizes response speed |
| Reconciliation path | Slower, deeper assessment | May provide greater accuracy |
The architecture slide places the OTel SDK and Collector before Kafka/MSK and a Stream Assembler, then branches into TIER-0 fast gate and TIER-2 async processing. The separation lets immediate assessment coexist with analysis that needs more time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A missing event is not necessarily a missing service
Asynchronous observation introduces an ambiguity. Suppose seven nodes participate in a request, but one node's telemetry arrives late. The detector temporarily sees six nodes. Has the execution structure changed, or is the observed graph simply incomplete? Treating every absent event as a removed node creates false alarms. Structural-change detection therefore needs explicit timing criteria for when the available evidence is sufficient.
Pandya recommends a tail-based approach for this example because the analysis needs each node's start and end. More precisely, span instrumentation records those timings; tail sampling decides which traces to retain using accumulated span information. It does not guarantee completeness. The current Collector processor documentation explicitly accounts for late-arriving spans and decisions made before they arrive, so waiting and retention policies remain part of the design. A new endpoint presents a separate cold-start problem: it needs its own baseline, rather than inheriting an overly generic one.
For detection, Pandya names maximum mean discrepancy, or MMD, and KL divergence, followed by confirmation with ADWIN. Classification then identifies the kind of problem and informs automation where feasible. These are stages in a proposed detection flow; the precise integration and configuration are not supplied.
Pandya reports a substantial reduction in mean time to discovery, attributing faster issue resolution to using a single observation window instead of waiting for multiple windows. He supplies neither a numerical improvement nor the window duration. That single-window account describes his system; ADWIN itself adapts its window length as the observed data changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the baseline specific and the decision explainable
Labels help the system learn, but they require careful tuning. The baseline also needs to correspond to a meaningful operation. One baseline for all POST requests lumps together work with different behavior: real-time payments and wire payments should be considered separately. Define the observation window for structural changes, and use per-client baselines where client differences would otherwise generate noise. The relevant unit of comparison is the operation and its context, not merely its HTTP method.
An alert needs an explanation, not just a score. Pandya compares an unexplained anomaly score to a doctor reporting a health score of twenty-two: the number alone does not tell you what is wrong or what to do. Supporting observations let an operator make an informed decision. Finally, the system must know when a new deployment occurred. That context connects changed behavior to a possible cause and helps determine whether rollback is an appropriate response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source microservices workloads for experimenting with end-to-end service performance.
Further reading
Research introducing DeathStarBench and analyzing performance characteristics of cloud and IoT microservices.
Explains head versus tail sampling and demonstrates Collector policies for retaining traces.
Author-hosted manuscript explaining adaptive windows for detecting changes in streaming data.
Foundational treatment of maximum mean discrepancy for testing whether samples come from different distributions.
Pandya's separate presentation on deterministic routing, data locality and graph-based observability, with an organizer-hosted transcript.
Updates since the talk
Current configuration guidance covering sampling policies, buffering, decision timing and late spans.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi. Uh, thanks, thanks, Na.
- 0:17
And, uh, hope everyone is out of, uh, the lunch coma and [laughs] will survive this talk. So, uh, yeah, myself Ritvik, I, uh, lead the payments team in, uh, JP Morgan.
- 0:30
And, uh, today I'll be talking about learned execution graphs, how these graphs can help to, uh, detect any anomaly and, uh, drifts. Also, how we can automate a few things around that and, uh, you know, um,
- 0:50
uh, at the same time, if we can reduce the manual, you know, detection work and, uh, going on that side. Uh, so whenever we hear about graph, uh, there are persistence graph and property graphs, uh, which Neo4j and, you know, other products, uh, we
- 1:15
use for them. We query those, uh, graphs and get the answers out of it. What I'm talking about today is execution graph. It's short-lived graph. Uh, and idea here is holistically try to identify how the request processing happens, and if there is any deviation on that, and how to detect that, and how to fix that.
- 1:41
So here is a simple example. Uh, say we have set of applications. Uh, you have one edge layer, uh, the first layer where, you know, request comes in. And then, uh, you have some gateways.
- 1:56
Uh, if gate is there, you have ingress layer on top of it. Then authentication, authorization happens. After that, there is some orchestration layer and a few other systems which could be called in parallel.
- 2:10
Uh, once everything is done, you are notifying your client that what's the update on that request, right? So here, the idea is representing this, uh, overall request processing as DAG.
- 2:25
And using DAG simplifies most of the things here.
- 2:30
One, now you know that, uh, in what order service execution will be happening, right? So that's one of the thing. The other thing is, uh, you know the context that in-- at what node, what context will be there and what will be passed to the next node.
- 2:49
Uh, in that way, it will be very, um, you know, ordered and simplified, um, uh, simply can be represented. Uh, there are a few other, uh, use cases could be there, uh, in terms of retries and, uh, the loops, et cetera.
- 3:05
Uh, the idea here is, uh, every loop, uh, to put in the graph as a separate entity itself. So, uh, in that way, it could be tracked, uh, easily.
- 3:23
How we can make this system, uh, more, uh, reliable at the same time not using, uh, most of the resources, right? So in the tier one check or ACR first check, it's like, uh, going to airport and you, you, you know, it's just boarding passes.
- 3:42
Some-someone is looking at the boarding pass and let you go. So, uh, now if you know the baseline of your request execution end to end, if, uh, everything looks good, you don't need to go to the tier two, uh, or next tier of check, right?
- 3:57
Once if you find that there is some delay, so now you need to check that what changed here. One of the, uh, the drift here could be because of the structural change.
- 4:11
So if any new node or new step added, which you are not aware of, that could be one of the thing or one of the step which is removed, that could be the another reason, right?
- 4:22
Uh, once you know about that, then further, uh,
- 4:29
further analysis could be done, uh, in terms of KL deviations or divergence and, uh, exponential MA. Uh, so in simpler terms, if you know that client A's request is taking this much time normally, and, uh, client B's request could take-- might take more time than the client A, uh, because of, say, one client is
- 4:54
local to you and one client is, uh, you know, the request is coming from outside, and, uh, there are a few more checks needs to be done. So in, in that case, the baseline will change client to client.
- 5:06
And now you know that, uh, what your threshold it and, uh, how you can, uh, reduce the noise of such alerts.
- 5:16
Uh, so here the idea is very simple.
- 5:22
First, you, uh, represent the entire request processing as DAG. You come up with the baseline, you find out the deviation, and then you try to find out where exactly the issue is.
- 5:38
Once you localize that, then you compare that based on your system that whether it is, uh, within your threshold or not. If it is within the threshold, yeah, you don't need to, uh, you know, do the alerts or automate anything.
- 5:53
But if it is out of the threshold, then certain action needs to be taken.
- 6:00
Uh, coming back to our example here, say overall, uh, request processing From all the different nodes within our system, uh, is happening, but somehow the foreign transaction, uh, rate service is taking more time than the usual.
- 6:19
Now, if you represented this whole, uh, request processing in multiple nodes, you know where the problem is or where the issue is. And correspondingly, you will, uh, you know, now you can, uh, exactly know where the problem is, so you can solve it, uh, that what-- how to, uh, how FX system or what all different, uh, cases
- 6:42
were there in the past where FX, FX rate system was failing.
- 6:50
Here is one of the example, uh, for benchmark, uh, OpenTelemetry and that Starbench were used. And say for seven days of the time, millions of, uh, traces were, um,
- 7:06
you know, injected or, uh, in the system. Then you ca-inject the problem or, uh, anomaly there. And based on that, you train your system before a-anything goes on live.
- 7:21
So, uh, again, uh, basic thing here is what is anomaly and what is drift, right? So say you are driving from your home to office every day, and, uh, one of-- uh, a-and usually it takes one hour.
- 7:39
But, uh, one fine day, it took you more time than one hour. Uh,
- 7:44
the reason might be some traffic or, you know, um, car accident or anything. But, uh, he-- this is one of the incidents and, uh, based on your system and criticality of your system, uh, you can decide how to address that.
- 8:01
The other part is one fine day you are, uh, taking sip of coffee around four PM and realize that a year back, it used to take one hour for you from your home to office.
- 8:13
But nowadays, it, it is taking twenty more minutes, right? So what happened? In Bay Area, your number of car increases or traffic increase. So this is over the time, what you are seeing is pattern changed, and that's where you might need to come up with, uh, the new baseline itself.
- 8:36
So, uh, that's, that's the drift that over the time you start seeing some delays or, you know, some, uh, performance deviation. Then once you know that there is a drift, uh, you can further categorize it.
- 8:51
First category is structural. Uh, so say you-- somehow in the system, a new node is added or one of the node is removed, as I mentioned earlier. For example, you like, uh, again, you know, there is a shop where you like, uh, drinking coffee and, uh, one fine day, they are ask-- start asking you about membership.
- 9:13
So they added one more, uh, step in it. Now, every day, they might ask you for, "Hey, do you have membership with us? If you have, then there is, there are a special discount for you.
- 9:22
If you don't have membership, then, uh, the regular prices will be there." So in that way, uh, you know, same way in our service processing or a request processing, if new node is added, that means, uh, now you need to consider that step also in your all baselines and new alerts.
- 9:43
The other one is, uh, say, because of the volume of request, uh, one of your service is taking more time or it's not, you know, uh, cannot serve the request or, or the volume which you are expecting now over the time.
- 10:04
So yeah, such kind of drifts you might need to treat differently because now you ne-- either you need to scale up those services or instances of those services and/or you need to either make it asynchronous call or based on, based on the use case, you know, what-whatever, uh, works there.
- 10:22
Uh, covariate is different, uh, one of the category. Say when you started the business, uh, you were seeing
- 10:32
around sixty percent of local requests, but, uh, and forty percent, uh, requests from, you know, uh, out of the country. And that's where you might need to change the currency or, you know, one extra step is there.
- 10:46
So you-- now you know that what is the baseline for your, uh, request in US dollar, but, uh, what is the baseline for, uh, any of the other currency.
- 10:57
Over the time, what happens is your product is so popular that you started getting more, uh, requests from the outside. So nothing changed. Your system is working fine, right?
- 11:09
But now you need to come up with the criteria and reassess your baselines again, sorry, here, where, uh,
- 11:19
either you need to come up with two different, uh, you know, uh, graphs to compare. That one is for local and one is for, uh, outside-- uh, request from outside.
- 11:31
Or what you can do is you can increase your, uh, average request time baseline. So once you know the pattern, uh, you know the solution. So, uh, that's where you need to ca-categorize, uh, these drifts.
- 11:48
One category could be for the same request, uh, now you are seeing the different behavior itself. Then probably if and when it's needed, you might need to roll back such, uh, changes or either you need to reconsider that.
- 12:06
So where I'm going with this is In that way, you need to reevaluate and reassess your system before, uh, identifying that what action needs to be taken.
- 12:18
This whole talk is mostly about statistical, uh, uh, you know, part of, uh, uh, the solution. Uh, it's, it's, uh, part of bigger neural, uh, specific, uh, um, algorithms and, uh, system in a way, but this is just one of the module which, uh, I'm talking about here now.
- 12:41
So once you know the drift or deviation, and here is the simple, uh, DAG for how, uh, this whole system would work.
- 12:58
OpenTelemetry, uh, will keep feeding the data. Once you have that data, uh, root cause analysis could be taken based on, uh, once your system knows about all the data points.
- 13:11
Now you know which type of drift it is and what solution could be there, then
- 13:19
you identify what action needs to be taken. Once you know what action needs to be taken, further you need to also, uh, find out that what is the risk if we go with this approach or if we automate this, uh, solution, right?
- 13:33
So once you know the risk, either you can go with, uh, roll out that sys-- uh, um, solution for say, five percent or ten percent of your s- uh, machines, monitor it, verify everything looks good, and then you roll out for your hundred percent of the nodes.
- 13:53
Here are a couple of example. Uh, again, uh, say generally overall request processing takes seven hundred milliseconds. But, uh, on the left side what you are seeing is, uh, in the graph approach itself how it could help you is, uh, now you know that, uh, which specific node is taking more time, suddenly you get alert on that.
- 14:15
Or otherwise, if you are seeing that, uh, the delay is all across, then something which is common which you need to fix here.
- 14:26
All these things, because in the payments and the real-time, uh, payment processing, we want to keep it very faster, right? So
- 14:36
we don't want delay the actual request processing. The solution which we generally use, uh, everyone in the industry is asynchronously feeding the data to, uh, OpenTelemetry. From there, uh, some Kafka could be used and, uh, stream assessment, uh, could be used on top of it.
- 14:56
There could be two different paths. One is, say, hot path where you can take a decision very faster and, uh, um, work on the solution or automate that solution.
- 15:07
The other one is more recon kind of solution where it might take some time, but more, more accurate could-- it could be.
- 15:21
Few of, uh, uh, other challenges which, uh, we need to fine-tune here. So all I talked about is, "Hey, you have seven nodes in your system and every node is feeding the data to your, uh, telemetry."
- 15:35
What if one of the system is delaying the event?
- 15:41
Should we consider it as a structural change? Because now what you have data in your system is for six nodes and the seven, uh, no-- uh, the data from the seven node is delayed already.
- 15:53
So we need to fine-tune that, uh, that, uh, those numbers also that when to consider that, uh, there is a structural change or not. So basically here we are, uh, trying to reduce, uh, any f-false alarm.
- 16:08
Based on the use cases and here in this use case, uh, we should go with, uh, tail-by base system because what we are trying to track here is, uh, on, uh, or the, in this specific example is when the service request started and when it ended, right?
- 16:26
So, uh, for each and every node. The other part is the cold start. Uh, if there is a new endpoint, consider the new baseline. Don't make it very generic.
- 16:39
So in that way, on the detect side, any MMD or KL, uh, could be used. Uh, and once you confirm, uh, this with admin and then classify, uh, the problem, uh, that will give you where or what exact solution needs to be done and next step whatever, uh, if we can automate it, we'll automate
- 17:04
it. So what we see, uh, in general here is, uh, mean time to, uh, discovery reduced a lot.
- 17:14
To make it very real time instead of comparing or waiting for multiple windows of, uh, the time duration, it was, uh, a single window which we, uh, uh, you know, uh, which we identified, uh, helped a lot, uh, to fix the issues fast.
- 17:33
Uh, few of other things which we might need to make sure. One is, uh, the labels. Uh, when it comes to, uh, learnings, the labels helps a lot. But, uh, at the same time, uh, we need to make sure that, uh, the system is very fine-tuned in terms of that.
- 17:52
Uh, the other part is instead of saying that all the post requests should have, you know, this is a baseline for all the post requests, uh, try to come up with very, uh, a number which works for you, post for payments for real-time payment or post for wire payments or, uh, based on, you know, uh, your use
- 18:14
cases. So that, that would help a lot. Um,
- 18:18
again, uh, if anything is, uh, uh, you are considering a structural change or something, uh, the window, uh, should be well-defined.
- 18:28
For each and every client in my previous ex-- uh, example which I talked about, uh, if you can come up with a new baseline would really help, uh, to reduce the noise.
- 18:38
Uh, explainability, uh, all the data should be well explained. If you go to the doctor and doctor says your health score is twenty-two, it doesn't make much sense to you.
- 18:50
So, uh, yeah, the actual, uh, more data, uh, can explain, you know, more things to you and, uh, can, we can take the informative decision on top of it.
- 19:02
Uh, whole system should be aware of the new deployment. Uh, so, uh,
- 19:07
uh, based on that you can take either a rollback decision or not. So yeah, that's, that's about it. Uh, thanks everyone. Uh, I would like to connect with you all, uh, here is my LinkedIn. [audience clapping]
- 19:20
I can answer your questions. [outro music]