# Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

Ritvik Pandya, JP Morgan Chase

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/u1yaOeEX4e8-learned-execution-graphs-anomaly-detection-drift) · [Video](https://www.youtube.com/watch?v=u1yaOeEX4e8)

## Learned Execution Graphs for API Anomalies and Drift

Representing each request as an execution graph helps separate a slow service, a changed processing path and a shifting request mix—then choose a response without delaying payments.

Familiarity with API request flows, distributed tracing and basic latency monitoring will help; the article introduces execution graphs and drift categories as they arise.

### What changed in this request?

[0:17](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=17s)

When an API request deviates from normal behavior, how do you find what changed—and decide whether to fix it? Ritvik Pandya, who introduces himself as leading a payments team at JP Morgan, approaches this through **short-lived execution graphs**. Unlike persistent property graphs queried through systems such as Neo4j, these graphs describe how an individual request proceeds through a system. Their purpose is to make deviations visible and reduce manual investigation.

Consider a payment request. It enters through an edge layer, passes through gateways and ingress, and reaches authentication and authorization. An orchestrator then invokes downstream services, some in parallel, before the system notifies the client. In the illustrated graph, those parallel branches are Fraud Score, Ledger Write and FX Rate; they converge at Notify. Representing the journey as a **directed acyclic graph**, or DAG, preserves both the sequential dependencies and the parallel work.

![Diagram connects Edge GW to AuthN/Z and Orchestrator, branches to Fraud Score, Ledger Write, and FX Rate, then rejoins at Notify.](https://ai.engineer/api/talk-essay-media/dac08773e37dee843ab8cae6cf2578179fb43af2a06dc2afd54c46173f349716)

From span tree to execution DAG: parallel services converge on Notify.

The graph tells you which services execute in which order, but it also identifies the context available at each node and what passes to the next one. Retries and loops need explicit representation: Pandya treats their occurrences as separate entities so they remain trackable. An execution graph therefore records the work that happened, rather than collapsing every visit to a service into one indistinguishable node.

A compact TypeScript representation of the illustrated dependencies makes the parallel branches explicit. Each array names the steps that must precede that node; it does not impose an order among Fraud Score, Ledger Write and FX Rate.

```typescript
type Step =
  | "Edge GW"
  | "AuthN/Z"
  | "Orchestrator"
  | "Fraud Score"
  | "Ledger Write"
  | "FX Rate"
  | "Notify";

const predecessors: Record<Step, readonly Step[]> = {
  "Edge GW": [],
  "AuthN/Z": ["Edge GW"],
  "Orchestrator": ["AuthN/Z"],
  "Fraud Score": ["Orchestrator"],
  "Ledger Write": ["Orchestrator"],
  "FX Rate": ["Orchestrator"],
  "Notify": ["Fraud Score", "Ledger Write", "FX Rate"],
};

function readySteps(completed: ReadonlySet<Step>): Step[] {
  return (Object.keys(predecessors) as Step[]).filter(
    step => !completed.has(step) &&
      predecessors[step].every(parent => completed.has(parent)),
  );
}

const completed = new Set<Step>([
  "Edge GW", "AuthN/Z", "Orchestrator",
]);

const ready = readySteps(completed);
// ["Fraud Score", "Ledger Write", "FX Rate"]
```

Timing and request context can then be attached to these execution nodes, while repeated attempts receive distinct identities.

### Spend analysis effort where the baseline breaks

[3:23](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=203s)

A detector should not spend the same resources on every request. Pandya compares the first tier to an airport boarding-pass check: if the request matches its normal end-to-end baseline, let it pass without escalating to a deeper inspection. A delay prompts the next question: did the structure change? An unexpected node or a missing processing step can explain why execution no longer matches the reference graph.

Further analysis can use **KL divergence** or an **exponential moving average**, but the comparison must reflect the request's context. Client A may be local, while client B's request arrives from outside the country and requires additional checks. Their normal latencies need not match. A threshold built around the appropriate client baseline helps avoid treating expected differences as incidents.

The decision sequence is straightforward:

1. Represent the full request as a DAG.
2. Establish its baseline and detect a deviation.
3. Localize the deviation to the relevant part of the graph.
4. Compare it with the threshold appropriate to that system and request.
5. Alert or act only when the deviation exceeds that threshold.

**Detection is not automatically a mandate for remediation.** A localized difference that remains within the accepted threshold requires neither an alert nor an automated intervention.

### Localize the slow FX service

[6:00](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=360s)

Return to the payment graph. The request is progressing through the system, but the foreign-exchange rate service is taking longer than usual. The graph gives the investigation a specific target: **FX Rate**, rather than the entire payment pipeline. Once that node is identified, previous cases of FX service failure become relevant evidence for deciding what to investigate or repair.

![Payment execution diagram with FX Rate and its connecting edges highlighted red, above a per-node deviation panel.](https://ai.engineer/api/talk-essay-media/c0ae8e03adeb8944fb85fa76755fa6bebd46f9b4770afbb7e58f4cce65944402)

The execution graph highlights FX Rate as the anomalous span.

Pandya describes a pre-live benchmark example using OpenTelemetry and [DeathStarBench](https://github.com/delimitrou/DeathStarBench), with millions of traces over seven days, followed by injected anomalies to train the system. This establishes the intended preparation sequence: observe ordinary execution, introduce faults and develop the detector before putting it live. The talk does not provide the selected workload, exact trace count, injection procedure or evaluation results for that example.

### An incident, a new step or a new request mix?

[7:21](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=441s)

An **anomaly** and **drift** pose different operational questions. Suppose the trip from home to the office normally takes an hour. One day it takes longer because of traffic or an accident: that is an isolated incident, and the response depends on how critical the disruption is. Now imagine noticing during a late-afternoon coffee break that the same commute routinely takes twenty minutes longer than it did a year ago. More cars and heavier Bay Area traffic have changed the pattern. The old baseline may no longer describe normal conditions.

Different kinds of drift call for different remedies:

- **Structural change:** A coffee shop starts asking whether you have a membership before applying discounted or regular pricing. That added question is a new processing step. Likewise, adding or removing a service node means the API's baselines and alerts must account for a changed path.
- **Capacity-related degradation:** Request volume grows until a service takes longer or cannot serve the expected load. Depending on the use case, the response may be to scale the service, add instances or make a call asynchronous.

Updating a baseline for an intentional extra step and addressing insufficient capacity are distinct operations, even if both first appear as increased latency.

A third case is **covariate drift**: the population of requests changes. Pandya's example begins with 60% local requests and 40% international requests. International payments may require currency conversion, so requests in US dollars and other currencies have different normal processing times. As the product becomes more popular abroad, the international share grows. The system can be working correctly while its aggregate latency changes.

Two responses are possible: compare local and international requests against separate execution graphs, or revise the aggregate request-time baseline to reflect the new mix. The distinction matters because a changing mix does not necessarily require a service repair. By contrast, if the *same request* now produces different behavior, the relevant response may be to reconsider or roll back a change. Classify the deviation before choosing the action.

### Turn diagnosis into a controlled rollout

[12:18](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=738s)

The detection methods here form a statistical module within a larger system involving neural algorithms; the talk does not specify that larger system's learning procedure. The operational flow starts with OpenTelemetry continuously supplying data. Those observations support root-cause analysis and drift classification, which in turn inform a candidate remedy.

Before automating that remedy, assess its risk. Pandya illustrates a staged rollout: apply the remedy to 5% or 10% of machines, monitor and verify it, then expand to 100% of nodes. This separates selecting an action from authorizing its broad execution. A plausible diagnosis still needs evidence that the proposed change behaves as intended.

### Keep analysis off the payment path

[13:53](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=833s)

Pandya uses an illustrative normal end-to-end request time of 700 milliseconds to compare two latency patterns. One node becoming slow directs attention to that service. Delays spread across the graph instead suggest investigating something shared by the affected nodes. The graph helps distinguish these patterns even when both appear to the client as a slow payment.

The detection system must not become another source of payment latency. Telemetry is fed asynchronously into OpenTelemetry, with Kafka as a possible transport and stream assessment downstream. That processing can split into two paths:

| Path | Purpose | Tradeoff |
| --- | --- | --- |
| Hot path | Fast decisions and possible automation | Prioritizes response speed |
| Reconciliation path | Slower, deeper assessment | May provide greater accuracy |

The architecture slide places the OTel SDK and Collector before Kafka/MSK and a Stream Assembler, then branches into TIER-0 fast gate and TIER-2 async processing. The separation lets immediate assessment coexist with analysis that needs more time.

![Architecture diagram flows from OTel SDK and Collector through Kafka/MSK and Stream Assembler to TIER-0 fast gate and TIER-2 async branches.](https://ai.engineer/api/talk-essay-media/c0c62c77d942d5261dc3383114352d0321c87f768376a360b3c73f736f77abeb)

Trace processing branches into a fast gate and asynchronous analysis.

### A missing event is not necessarily a missing service

[15:21](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=921s)

Asynchronous observation introduces an ambiguity. Suppose seven nodes participate in a request, but one node's telemetry arrives late. The detector temporarily sees six nodes. Has the execution structure changed, or is the observed graph simply incomplete? Treating every absent event as a removed node creates false alarms. Structural-change detection therefore needs explicit timing criteria for when the available evidence is sufficient.

Pandya recommends a tail-based approach for this example because the analysis needs each node's start and end. More precisely, span instrumentation records those timings; [tail sampling](https://opentelemetry.io/blog/2022/tail-sampling/) decides which traces to retain using accumulated span information. It does not guarantee completeness. The [current Collector processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md) explicitly accounts for late-arriving spans and decisions made before they arrive, so waiting and retention policies remain part of the design. A new endpoint presents a separate cold-start problem: it needs its own baseline, rather than inheriting an overly generic one.

For detection, Pandya names **maximum mean discrepancy**, or [MMD](https://jmlr.org/papers/v13/gretton12a.html), and KL divergence, followed by confirmation with [ADWIN](https://www.cs.upc.edu/~gavalda/papers/adwin06.pdf). Classification then identifies the kind of problem and informs automation where feasible. These are stages in a proposed detection flow; the precise integration and configuration are not supplied.

Pandya reports a substantial reduction in mean time to discovery, attributing faster issue resolution to using a single observation window instead of waiting for multiple windows. He supplies neither a numerical improvement nor the window duration. That single-window account describes his system; ADWIN itself adapts its window length as the observed data changes.

### Make the baseline specific and the decision explainable

[17:33](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1053s)

Labels help the system learn, but they require careful tuning. The baseline also needs to correspond to a meaningful operation. One baseline for all `POST` requests lumps together work with different behavior: real-time payments and wire payments should be considered separately. Define the observation window for structural changes, and use per-client baselines where client differences would otherwise generate noise. The relevant unit of comparison is the operation and its context, not merely its HTTP method.

**An alert needs an explanation, not just a score.** Pandya compares an unexplained anomaly score to a doctor reporting a health score of twenty-two: the number alone does not tell you what is wrong or what to do. Supporting observations let an operator make an informed decision. Finally, the system must know when a new deployment occurred. That context connects changed behavior to a possible cause and helps determine whether rollback is an appropriate response.

![Six tradeoff cards cover label scarcity, cardinality blow-up, trace incompleteness, per-client state, explainability, and deploy ambiguity.](https://ai.engineer/api/talk-essay-media/202382bd46b62d35cf4bd061af94bbffd3b9067232bee79179dbabeb2812bc03)

Execution-graph tradeoffs include label scarcity and per-client state.

## Resources

- [DeathStarBench](https://github.com/delimitrou/DeathStarBench) — Open-source microservices workloads for experimenting with end-to-end service performance.
- [DeathStarBench research paper](https://arxiv.org/abs/1905.11055) — Research introducing DeathStarBench and analyzing performance characteristics of cloud and IoT microservices.
- [Tail sampling with OpenTelemetry](https://opentelemetry.io/blog/2022/tail-sampling/) — Explains head versus tail sampling and demonstrates Collector policies for retaining traces.
- [Tail sampling processor configuration](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md) — Current configuration guidance covering sampling policies, buffering, decision timing and late spans.
- [Adaptive windowing and ADWIN](https://www.cs.upc.edu/~gavalda/papers/adwin06.pdf) — Author-hosted manuscript explaining adaptive windows for detecting changes in streaming data.
- [A Kernel Two-Sample Test](https://jmlr.org/papers/v13/gretton12a.html) — Foundational treatment of maximum mean discrepancy for testing whether samples come from different distributions.
- [Database DevOps at Scale](https://www.conf42.com/Database_DevOps_2026_Ritvik_Pandya_monitoring_telemetry_reliability) — Pandya's separate presentation on deterministic routing, data locality and graph-based observability, with an organizer-hosted transcript.

## Slides

### Learned Execution Graphs for Real-Time Anomaly Detection & Drift…

Ritvik Pandya introduces a way to turn request traces into execution DAGs, learn normal graph patterns, and score anomalies and classify drift near real time at production throughput.

[0:20](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=20s) [Title slide introducing learned execution graphs for detecting anomalies and classifying drift in APIs.](https://assets.ai.engineer/sites/aiecode2025/sha256/e85d40d6386c1d7e3478189790d141e08495a8f936976297198fd4bc063e48c8.webp)

AI ENGINEER WORLD’S FAIR 2026 · GRAPHS TRACK

Learned Execution Graphs for Real-Time Anomaly Detection & Drift Classification in APIs

Turn every request’s trace into an attributed execution DAG. Learn the distribution of normal graphs. Score anomalies and classify drift — in near real time, at production throughput.

Ritvik Pandya
Engineering Lead · J.P. Morgan Chase — CIB Payments

Views are my own. Examples & figures generalized.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
01 / 12

### A different kind of graph

A persistent graph stores facts for later queries. An execution graph captures how one request ran: its causal edges can help locate deviations and distinguish causes from symptoms before the short-lived signal disappears.

[0:59](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=59s) [Comparison of persistent property graphs with short-lived execution graphs built from request traces.](https://assets.ai.engineer/sites/aiecode2025/sha256/39fa9282f7aee1ad1af7c442c25385e82fb3864f4854be36d508863db5c312ab.webp)

01 · FRAMING

A different kind of graph

WHAT THE TRACK MEANS BY “GRAPH”
Persistent / property graph
• Knowledge graphs, GraphRAG, entity links
• Lives in a database (Neo4j, ...)
• Queried for what is related to what
• Edges are facts; the graph is the data

THIS TALK
Execution graph (a trace)
• One execution DAG per request, built from its spans
• Lives in the request path for milliseconds
• Describes how this call actually ran
• Edges are causality; the graph is behavior

Same word, opposite lifecycle. One is a store you query. The other is a signal you have ~40 ms to read before it’s gone.

Why the graph, not just metrics? For “is it slow?” you don’t need it — but where, cause vs. symptom, and what kind of change are structural, and a number can’t answer them.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
02 / 12

### From span tree to execution DAG

A request passes through the gateway, authentication and orchestrator, then fans out to three services. A constructed join brings their paths together at Notify.

[1:39](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=99s) [A request flows through gateway, authentication and orchestration, fans out to three services, then joins at notification.](https://assets.ai.engineer/sites/aiecode2025/sha256/9027d50825f251ad7a7e5af01dec693d97eed7611708683b1dc9ea486b17a429.webp)

02 · DEFINITION
From span tree to execution DAG

OpenTelemetry gives you a span tree — one parent per span. We enrich it into an execution DAG: add span links, async producer→consumer edges, shared-resource and join causality. (“Notify” re-converges via a constructed join edge.)

fan-out
join edge

Edge GW
ingress

AuthN/Z
token-svc

Orchestrator
payments-core

Fraud Score
ml-svc

Ledger Write
cockroachdb

FX Rate
ext-api

Notify
response

G = (V, E)   V = spans (operations)   E = parent-child + links + async + join   node feats: svc, op, status, duration   edge feats: sync/async, lag

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
03 / 12

### Acyclicity is load-bearing, not cosmetic

A DAG orders spans so each node can gather upstream context in one pass and separate root causes from downstream symptoms. Repeated calls can be unrolled into distinct nodes to restore the DAG.

[2:28](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=148s) [Three reasons for using a DAG: topological order, causal direction and ordered propagation, with a caveat about retries and loops.](https://assets.ai.engineer/sites/aiecode2025/sha256/4113c664da39996dc3a7dbde3f2b6e542e9e2a54bd107ee1b38e992cc7c504a8.webp)

03 · WHY A DAG

Acyclicity is load-bearing, not cosmetic

1
Topological order
A DAG has a valid linearization → you can compute the critical path and attribute end-to-end latency to the exact span that owns it.

2
Causal direction
Edges point parent→child. That lets the detector separate a root cause (upstream) from its symptoms (everything downstream of it).

3
Ordered propagation
Process spans in topological order — each node aggregates its full upstream context in one causal pass, instead of iterating rounds and over-smoothing the whole graph.

THE HONEST CAVEAT — RETRIES & LOOPS
Retries, polling, and sagas can introduce cycles. Restore the DAG before modeling: unroll repeats into distinct nodes (call#1, call#2) or type the edge as retry/async. A retry storm then shows up as a structural feature — exactly the signal you want.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
04 / 12

### “Learned” = model the distribution, not rules

A graph-hash and timing gate checks requests first. Node-level comparisons investigate delays and structural changes; client profiles track behavioral drift.

[3:20](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=200s) [Table compares graph-hash baselines, node deviation and structural checks, and per-client drift analysis.](https://assets.ai.engineer/sites/aiecode2025/sha256/40e1cb937b0e2826d8b5625f0f513c4a740bf2d3dd838189e5938b02a3867495.webp)

04 · THE MODEL
“Learned” = model the distribution, not rules

Hand-written thresholds break on every deploy. Instead, learn per-node baselines and the graph’s normal structure from telemetry — no labels, nothing to train. Three statistical layers, cheapest first:

TIER | METHOD | LEARNS | CATCHES | GRANULAR | COST | ROLE
Tier 0 | graph-hash + timing baseline | topology + latency normal | rare / off-timing shape | ✓ by signature | ~µs | gate
Tier 1 | deviation ratios + structural compare | per-node baselines + topology | slow node + structural change | ✓ per node | <1ms | attribute
Tier 2 | per-client EMA + KL divergence | client graph profiles | behavioral drift + cause | ✓ per client | ~ms | classify

All three run in microseconds to milliseconds — no GPU, no inference, no black box. (A learned graph autoencoder is a natural extension; for payments, statistics wins on latency and interpretability.)

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
05 / 12

### From graph to localized cause

The system learns node baselines from normal traffic, scores deviations to locate the worst node, and checks for structural changes. A false-positives-per-day budget sets the alert threshold.

[5:20](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=320s) [Five-step workflow represents a trace as a DAG, establishes baselines, measures deviation, localizes a cause and applies a threshold.](https://assets.ai.engineer/sites/aiecode2025/sha256/91a9c8d9a0458f1952804da5fae5eac7d491f98bcbe6ec0b2838270947e5dfef.webp)

05 · THE APPROACH
From graph to localized cause

No model to train — learn per-node baselines from normal traffic. A request’s deviation from them is the score; the worst node is the cause.

1 Represent
trace → DAG
2 Baseline
per-node stats
3 Deviate
obs / baseline
4 Localize
argmax ratio
5 Threshold
FP/day budget

THE BASELINES — learned, not trained

Per-node statistical baselines — latency (μ, σ, p99), dependency set, execution frequency. Mined from normal traffic. No labels, no neural net.

baseline(n) = {latency μ/σ, deps, freq}
deviation(n) = observed / baseline → score
cause = argmax(deviation) → localize

TWO CHECKS ON ONE GRAPH

deviation ratio → bottleneck node  attribute
structure check → missing / reordered / new  structural

One finds a node that’s slow; the other finds a node that shouldn’t be there — or a mandatory one that’s gone.

Threshold = a false-positives-per-day budget, not a latency guess. Baselines recalibrate as normal re-learns — gated by the drift classifier and the deploy log, so an expected change doesn’t page.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
06 / 12

### Score the graph, then localize the span

In the payment graph, FX Rate reaches 38× its baseline latency while the other nodes stay near 1×. The deviation localizes the anomaly to FX Rate even though end-to-end p99 barely moved.

[5:58](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=358s) [Payment execution graph highlights the FX rate node at 38 times its baseline latency.](https://assets.ai.engineer/sites/aiecode2025/sha256/f2221b7c7b6b7dd212a038272639e7559f483c0ee4b719d7570ad20fc13896a3.webp)

06 · ANOMALY DETECTION

Score the graph, then localize the span

The same payment DAG — now the external FX dependency degrades. Each node’s observed latency over its baseline: the offending span stands an order of magnitude above normal while the rest sit at ~1×.

Edge GW
ingress

AuthN/Z
token-svc

Orchestrator
payments-core

Fraud Score
ml-svc

Ledger Write
cockroachdb

FX Rate
ext-api

Notify
response

PER-NODE DEVIATION (observed / baseline)

edge-gw 1.0×   auth 1.1×   orchestrator 1.0×   fraud 1.2×   ledger 1.1×   notify 1.0×   fx-rate 38×

fx-rate z-score ≫ threshold → ANOMALY, localized to fx-rate · end-to-end p99 barely moved

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
07 / 12

### One real experiment beats ten illustrations

The proposed benchmark trains on ~1.9M normal traces over 7 days, then injects failures. Illustrative results show detection of 86% of injected incidents and top-1 localization in 83% of detected cases.

[6:47](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=407s) [Illustrative benchmark setup and example detection, localization, latency, and false-alert results.](https://assets.ai.engineer/sites/aiecode2025/sha256/acb599e484347b9a6bb5f1e532ca97db9e40aeea37d92d927f69a3891ef921bb.webp)

07 · EVIDENCE

One real experiment beats ten illustrations

ILLUSTRATIVE result shape — reproduce on the public benchmark, then replace with your measured run. No production data needed.

SETUP
Benchmark  OpenTelemetry + DeathStarBench
Train on  ~1.9M normal traces · 7-day window
Inject  FX latency · missing span · retry storm
post-deploy topology · traffic shift · CPU sat.

RESULTS (report these)
Detection  86% of injected incidents
Deviation scoring  0.4 ms median
Localization  top-1 83% of detected
False alerts  4 / service / day
Structural + KL check  < 2 ms (no GPU)

Illustrative: “On ~1.9M synthetic traces across 6 injected failure classes, per-node deviation flagged 86% of incidents at 0.4 ms median latency and localized the responsible node in 83% of detected cases — no GPU, no inference step.”

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
08 / 12

### The pivot

An anomaly is one graph outside the normal cloud; drift shifts the cloud over time. Retraining an anomaly detector on drift can teach it the shifted pattern as normal, so the shift itself needs detection.

[7:20](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=440s) [A point outside a normal cloud illustrates an anomaly; shifted distributions illustrate drift over time.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb069a5496a4a4f53cea794a144af4b50cbbfcc0e7431fff1a440b3689b73dcb.webp)

08 · THE PIVOT

ANOMALY — a graph off the normal cloud
instantaneous · a single request is wrong

DRIFT — the normal cloud itself shifts
reference
current
over time · “normal” has changed

An anomaly detector retrained on drift learns the drift as normal. You must detect the shift itself — and say what kind it is.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
09 / 12

### Two DAGs: one observed, one decided

An execution-graph deviation triggers a decision path through root cause analysis, risk scoring, action and verification. Failed verification leads to rollback; an incident KB retrieves a matching runbook.

[12:16](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=736s) [An execution graph triggers a remediation flow from telemetry through root cause analysis, action, and verification, with rollback on failed verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/569f0541e9f412f8e98a7ebd13b4671d50e35432c4693782d27a4b0e6d0888b5.webp)

10 · CLOSING THE LOOP
Two DAGs: one observed, one decided

OBSERVED
Execution DAG

deviation + drift
class (KL)

DECIDED
Remediation decision DAG

Telemetry → RCA → Candidate Actions → Risk Scoring → Action → Verify
rollback on failed verification

INCIDENT KB matches the drift fingerprint against historical incidents (RAG, similarity-ranked) → retrieves the runbook before a human reads the alert.

The execution-graph signal is the trigger; an interpretable decision DAG is the response.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
11 / 12

### Scalar metrics tell you “when,” never “where”

Both requests take 940 ms, but their graphs point to different problems: an FX dependency stalls in Request A, while Request B slows across all components.

[13:49](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=829s) [Two requests have the same end-to-end latency; one has a slow FX dependency and the other shows a uniform slowdown.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3e194647dd143d1e4c138e10a023c315d6bb9a7aee3618b367a825522ccf965.webp)

A1 · MOTIVATION

Scalar metrics tell you “when,” never “where”

Two requests. Identical end-to-end latency. Completely different critical paths.

Request A — 940 ms end-to-end
Components: GW, auth, core, frd, db, fx
one dependency stalls — FX subtree

Request B — 940 ms end-to-end
Components: GW, auth, core, frd, db, fx
uniform slowdown — shared saturation

Same number → opposite cause → opposite fix. A scalar collapses the graph; you can’t recover structure from one number.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A1 / 6

### Scoring every request in near real time

OpenTelemetry spans pass through Kafka and a stream assembler, then enter a sub-ms gate and asynchronous per-client drift analysis. The slide reports 100% scored, with deep analysis sampled.

[14:23](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=863s) [Pipeline from telemetry collection through stream assembly to fast and asynchronous scoring.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f8e30c8fbbfc3d409c3b265a8877939c9f6f311d4a94abdbd60bd4e0d045989.webp)

Scoring every request in near real time
A2 · ARCHITECTURE

OTel SDK + Collector
spans emitted in-band

Kafka / MSK
span stream
partition by trace-id

Stream Assembler
Flink: windowed
trace → DAG

TIER 0 — fast gate
graph-hash + timing · scores on assembly · sub-ms

TIER 2 — async
per-client EMA + KL · off-path · classifies drift

100k+
requests / sec scored

< 1 ms
span-emit hot-path overhead

100%
scored · deep-analyze sampled

Two-tier by design: spans emit in-band with negligible overhead; scoring runs in near real time on assembled traces — a sub-ms deviation gate first, heavier per-client and structural analysis off-path. The hot path never waits on it.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A2 / 6

### When is a trace done?

Closing a trace too early can turn a delayed span into a false anomaly. The slide also recommends tail-based sampling and service-level priors for endpoints without a baseline.

[15:21](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=921s) [Trace assembly timeline and challenges involving late spans, sampling bias, and cold starts.](https://assets.ai.engineer/sites/aiecode2025/sha256/10c39b46c370eec6b438150d9b4f771e3d8ecab027595a4afacce7dfc2e6401e.webp)

APPENDIX A3 · TRACE ASSEMBLY
“When is a trace done?” — the assembly problem

spans arrive out of order, across a window
Timeline: GW · auth · core · frd · db · fx (late) · notify · watermark / close

Late spans ≠ missing spans
Close the window too early and a slow leaf looks like a structural hole — a false anomaly. Watermark on span timing, not wall-clock.

Sampling bias
Head-based sampling skews the graph distribution you learn from. Prefer tail-based / always-sample-on-error so anomalies survive.

Cold start
A new endpoint has no baseline. Back off to service-level priors until the per-signature distribution fills in.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A3 / 6

### From drift happened to which drift

Compare embedding distributions, confirm the change point, then classify drift. Structural and timing shifts suggest different causes; deploy logs help distinguish expected changes from incidents.

[16:37](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=997s) [Detect, confirm, and classify drift, distinguishing structural and temporal changes.](https://assets.ai.engineer/sites/aiecode2025/sha256/7be9fad80a2b32ae4f7df9565d1407b1756cc54a856b0c9cb834b968bfee505f.webp)

APPENDIX A4 · CLASSIFYING DRIFT
From “drift happened” to “which drift”

1 DETECT
Sliding reference vs detection window. Distance on graph-embedding distributions:
MMD · Wasserstein · KL

2 CONFIRM
Change-point test on the score / embedding stream to reject noise and find onset:
ADWIN · Page-Hinkley

3 CLASSIFY
Light classifier on drift features → the quadrant:
Δstructure vs Δtiming, deploy-corr, gradual/sudden

THE DISCRIMINATIVE FEATURE
Decompose the embedding shift into a structural component and a temporal component. Structure moved → deploy/dependency. Only timing moved → capacity. Correlate against the deploy log to split expected from incident.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A4 / 6

### What good looks like — the scorecard

The illustrative scorecard targets a 60–80% reduction in MTTD versus scalar alerting and drift classification in less than one window. It also tracks top-1 localization and false alerts per service per day.

[17:00](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1020s) [Illustrative targets for detection, localization, drift lag, and false alerts, with reporting recommendations.](https://assets.ai.engineer/sites/aiecode2025/sha256/5cc9bac7406a76311cea395d8fa116bf20d41a3885c5b3da4d0ae8d64fbb063f.webp)

What good looks like — the scorecard
APPENDIX A5 · EVALUATION

↓ 60–80%
MTTD
time-to-detect vs scalar alerting

0.8–0.9
Top-1 localize
right span, automatically

< 1 window
Drift lag
onset → classified

< 10
False alerts
per service / day

MEDIAN TIME-TO-DETECT (illustrative)
Graph detection: 3
Scalar alerting: 14

REPORT IT HONESTLY
• Detection latency, not just accuracy
• Localization top-1, not graph-level AUC alone
• FP/day — the real alert-fatigue cost
• Inference p99 + graphs/sec at peak

Figures are illustrative targets — substitute your own measured numbers under whatever you can disclose.
LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A5 / 6

### Where this bites — know before you build

High-cardinality attributes expand the signature space, while per-client tracking costs memory. Dropped spans can mimic anomalies; clean deployments can look anomalous until rebaselining, so include deployment logs.

[17:31](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1051s) [Six implementation tradeoffs covering labels, cardinality, incomplete traces, client state, explainability, and deployment ambiguity.](https://assets.ai.engineer/sites/aiecode2025/sha256/70693c284f70477a1907232e4a07fe49163fe2981f3e4424b06f742bfdf27e42.webp)

TRADEOFFS
Where this bites — know before you build

Label scarcity
Anomalies are rare and drift is unlabeled. Stay self-supervised; treat any labels as evaluation, not training.

Cardinality blow-up
High-cardinality attributes explode the signature space. Bucket / drop them from the graph key or Tier 0 never stabilizes.

Trace incompleteness
Dropped spans mimic structural anomalies. Gate on trace completeness before you trust a structural verdict.

Per-client state
Tracking EMA + KL per client is memory at scale. Bound it — cap tracked clients, decay idle ones, shard the rest.

Explainability
Ops won’t action a bare score. Per-node error + the drift class ARE the explanation — ship them, not a number.

Deploy ambiguity
A clean deploy looks anomalous until rebaseline. Wire the deploy log in as a feature, not an afterthought.

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK
A6 / 6

### Thank you

Ritvik Pandya invites questions and connections on LinkedIn.

[19:12](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1152s) [Closing slide inviting questions and professional contact, with a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc5c89917c36f3733f26b7668241dbd707df626fca81b9c33de0b53260abe768.webp)

Thank you

Let’s keep the conversation going — questions now, or connect anytime.

Ritvik Pandya
Engineering Lead · J.P. Morgan Chase — CIB Payments
linkedin.com/in/ritvik-pandya

Scan to connect on LinkedIn

LEARNED EXECUTION GRAPHS · AIE WORLD’S FAIR ’26 · GRAPHS TRACK

### AI Engineer World's Fair

[19:22](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1162s) [White AI Engineer World's Fair branding on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/a091086c13450c7b7ea24868da60113cda3a62e2bedbcdd70f450c0decce5fd8.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:00](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=0s) [upbeat music] Hi. Uh, thanks, thanks, Na.

[0:17](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=17s) And, uh, hope everyone is out of, uh, the lunch coma and [laughs] will survive this talk. So, uh, yeah, myself Ritvik, I, uh, lead the payments team in, uh, JP Morgan.

[0:30](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=30s) And, uh, today I'll be talking about learned execution graphs, how these graphs can help to, uh, detect any anomaly and, uh, drifts. Also, how we can automate a few things around that and, uh, you know, um,

[0:50](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=50s) uh, at the same time, if we can reduce the manual, you know, detection work and, uh, going on that side. Uh, so whenever we hear about graph, uh, there are persistence graph and property graphs, uh, which Neo4j and, you know, other products, uh, we

[1:15](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=75s) use for them. We query those, uh, graphs and get the answers out of it. What I'm talking about today is execution graph. It's short-lived graph. Uh, and idea here is holistically try to identify how the request processing happens, and if there is any deviation on that, and how to detect that, and how to fix that.

[1:41](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=101s) So here is a simple example. Uh, say we have set of applications. Uh, you have one edge layer, uh, the first layer where, you know, request comes in. And then, uh, you have some gateways.

[1:56](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=116s) Uh, if gateway is there, you have ingress layer on top of it. Then authentication, authorization happens. After that, there is some orchestration layer and a few other systems which could be called in parallel.

[2:10](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=130s) Uh, once everything is done, you are notifying your client that what's the update on that request, right? So here, the idea is representing this, uh, overall request processing as DAG.

[2:25](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=145s) And using DAG simplifies most of the things here.

[2:30](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=150s) One, now you know that, uh, in what order service execution will be happening, right? So that's one of the thing. The other thing is, uh, you know the context that in-- at what node, what context will be there and what will be passed to the next node.

[2:49](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=169s) Uh, in that way, it will be very, um, you know, ordered and simplified, um, uh, simply can be represented. Uh, there are a few other, uh, use cases could be there, uh, in terms of retries and, uh, the loops, et cetera.

[3:05](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=185s) Uh, the idea here is, uh, every loop, uh, to put in the graph as a separate entity itself. So, uh, in that way, it could be tracked, uh, easily.

[3:23](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=203s) How we can make this system, uh, more, uh, reliable at the same time not using, uh, most of the resources, right? So in the tier one check or ACR first check, it's like, uh, going to airport and you, you, you know, it's just boarding passes.

[3:42](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=222s) Some-someone is looking at the boarding pass and let you go. So, uh, now if you know the baseline of your request execution end to end, if, uh, everything looks good, you don't need to go to the tier two, uh, or next tier of check, right?

[3:57](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=237s) Once if you find that there is some delay, so now you need to check that what changed here. One of the, uh, the drift here could be because of the structural change.

[4:11](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=251s) So if any new node or new step added, which you are not aware of, that could be one of the thing or one of the step which is removed, that could be the another reason, right?

[4:22](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=262s) Uh, once you know about that, then further, uh,

[4:29](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=269s) further analysis could be done, uh, in terms of KL deviations or divergence and, uh, exponential MA. Uh, so in simpler terms, if you know that client A's request is taking this much time normally, and, uh, client B's request could take-- might take more time than the client A, uh, because of, say, one client is

[4:54](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=294s) local to you and one client is, uh, you know, the request is coming from outside, and, uh, there are a few more checks needs to be done. So in, in that case, the baseline will change client to client.

[5:06](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=306s) And now you know that, uh, what your threshold it and, uh, how you can, uh, reduce the noise of such alerts.

[5:16](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=316s) Uh, so here the idea is very simple.

[5:22](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=322s) First, you, uh, represent the entire request processing as DAG. You come up with the baseline, you find out the deviation, and then you try to find out where exactly the issue is.

[5:38](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=338s) Once you localize that, then you compare that based on your system that whether it is, uh, within your threshold or not. If it is within the threshold, yeah, you don't need to, uh, you know, do the alerts or automate anything.

[5:53](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=353s) But if it is out of the threshold, then certain action needs to be taken.

[6:00](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=360s) Uh, coming back to our example here, say overall, uh, request processing From all the different nodes within our system, uh, is happening, but somehow the foreign exchange, uh, rate service is taking more time than the usual.

[6:19](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=379s) Now, if you represented this whole, uh, request processing in multiple nodes, you know where the problem is or where the issue is. And correspondingly, you will, uh, you know, now you can, uh, exactly know where the problem is, so you can solve it, uh, that what-- how to, uh, how FX system or what all different, uh, cases

[6:42](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=402s) were there in the past where FX, FX rate system was failing.

[6:50](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=410s) Here is one of the example, uh, for benchmark, uh, OpenTelemetry and DeathStarBench were used. And say for seven days of the time, millions of, uh, traces were, um,

[7:06](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=426s) you know, injected or, uh, in the system. Then you ca-inject the problem or, uh, anomaly there. And based on that, you train your system before a-anything goes on live.

[7:21](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=441s) So, uh, again, uh, basic thing here is what is anomaly and what is drift, right? So say you are driving from your home to office every day, and, uh, one of-- uh, a-and usually it takes one hour.

[7:39](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=459s) But, uh, one fine day, it took you more time than one hour. Uh,

[7:44](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=464s) the reason might be some traffic or, you know, um, car accident or anything. But, uh, he-- this is one of the incidents and, uh, based on your system and criticality of your system, uh, you can decide how to address that.

[8:01](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=481s) The other part is one fine day you are, uh, taking sip of coffee around four PM and realize that a year back, it used to take one hour for you from your home to office.

[8:13](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=493s) But nowadays, it, it is taking twenty more minutes, right? So what happened? In Bay Area, your number of car increases or traffic increase. So this is over the time, what you are seeing is pattern changed, and that's where you might need to come up with, uh, the new baseline itself.

[8:36](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=516s) So, uh, that's, that's the drift that over the time you start seeing some delays or, you know, some, uh, performance deviation. Then once you know that there is a drift, uh, you can further categorize it.

[8:51](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=531s) First category is structural. Uh, so say you-- somehow in the system, a new node is added or one of the node is removed, as I mentioned earlier. For example, you like, uh, again, you know, there is a shop where you like, uh, drinking coffee and, uh, one fine day, they are ask-- start asking you about membership.

[9:13](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=553s) So they added one more, uh, step in it. Now, every day, they might ask you for, "Hey, do you have membership with us? If you have, then there is, there are a special discount for you.

[9:22](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=562s) If you don't have membership, then, uh, the regular prices will be there." So in that way, uh, you know, same way in our service processing or a request processing, if new node is added, that means, uh, now you need to consider that step also in your all baselines and new alerts.

[9:43](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=583s) The other one is, uh, say, because of the volume of request, uh, one of your service is taking more time or it's not, you know, uh, cannot serve the request or, or the volume which you are expecting now over the time.

[10:04](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=604s) So yeah, such kind of drifts you might need to treat differently because now you ne-- either you need to scale up those services or instances of those services and/or you need to either make it asynchronous call or based on, based on the use case, you know, what-whatever, uh, works there.

[10:22](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=622s) Uh, covariate is different, uh, one of the category. Say when you started the business, uh, you were seeing

[10:32](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=632s) around sixty percent of local requests, but, uh, and forty percent, uh, requests from, you know, uh, out of the country. And that's where you might need to change the currency or, you know, one extra step is there.

[10:46](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=646s) So you-- now you know that what is the baseline for your, uh, request in US dollar, but, uh, what is the baseline for, uh, any of the other currency.

[10:57](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=657s) Over the time, what happens is your product is so popular that you started getting more, uh, requests from the outside. So nothing changed. Your system is working fine, right?

[11:09](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=669s) But now you need to come up with the criteria and reassess your baselines again, sorry, here, where, uh,

[11:19](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=679s) either you need to come up with two different, uh, you know, uh, graphs to compare. That one is for local and one is for, uh, outside-- uh, request from outside.

[11:31](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=691s) Or what you can do is you can increase your, uh, average request time baseline. So once you know the pattern, uh, you know the solution. So, uh, that's where you need to ca-categorize, uh, these drifts.

[11:48](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=708s) One category could be for the same request, uh, now you are seeing the different behavior itself. Then probably if and when it's needed, you might need to roll back such, uh, changes or either you need to reconsider that.

[12:06](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=726s) So where I'm going with this is In that way, you need to reevaluate and reassess your system before, uh, identifying that what action needs to be taken.

[12:18](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=738s) This whole talk is mostly about statistical, uh, uh, you know, part of, uh, uh, the solution. Uh, it's, it's, uh, part of bigger neural, uh, specific, uh, um, algorithms and, uh, system in a way, but this is just one of the module which, uh, I'm talking about here now.

[12:41](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=761s) So once you know the drift or deviation, and here is the simple, uh, DAG for how, uh, this whole system would work.

[12:58](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=778s) OpenTelemetry, uh, will keep feeding the data. Once you have that data, uh, root cause analysis could be taken based on, uh, once your system knows about all the data points.

[13:11](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=791s) Now you know which type of drift it is and what solution could be there, then

[13:19](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=799s) you identify what action needs to be taken. Once you know what action needs to be taken, further you need to also, uh, find out that what is the risk if we go with this approach or if we automate this, uh, solution, right?

[13:33](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=813s) So once you know the risk, either you can go with, uh, roll out that sys-- uh, um, solution for say, five percent or ten percent of your s- uh, machines, monitor it, verify everything looks good, and then you roll out for your hundred percent of the nodes.

[13:53](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=833s) Here are a couple of example. Uh, again, uh, say generally overall request processing takes seven hundred milliseconds. But, uh, on the left side what you are seeing is, uh, in the graph approach itself how it could help you is, uh, now you know that, uh, which specific node is taking more time, suddenly you get alert on that.

[14:15](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=855s) Or otherwise, if you are seeing that, uh, the delay is all across, then something which is common which you need to fix here.

[14:26](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=866s) All these things, because in the payments and the real-time, uh, payment processing, we want to keep it very faster, right? So

[14:36](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=876s) we don't want delay the actual request processing. The solution which we generally use, uh, everyone in the industry is asynchronously feeding the data to, uh, OpenTelemetry. From there, uh, some Kafka could be used and, uh, stream assessment, uh, could be used on top of it.

[14:56](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=896s) There could be two different paths. One is, say, hot path where you can take a decision very faster and, uh, um, work on the solution or automate that solution.

[15:07](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=907s) The other one is more recon kind of solution where it might take some time, but more, more accurate could-- it could be.

[15:21](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=921s) Few of, uh, uh, other challenges which, uh, we need to fine-tune here. So all I talked about is, "Hey, you have seven nodes in your system and every node is feeding the data to your, uh, telemetry."

[15:35](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=935s) What if one of the system is delaying the event?

[15:41](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=941s) Should we consider it as a structural change? Because now what you have data in your system is for six nodes and the seven, uh, no-- uh, the data from the seventh node is delayed already.

[15:53](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=953s) So we need to fine-tune that, uh, that, uh, those numbers also that when to consider that, uh, there is a structural change or not. So basically here we are, uh, trying to reduce, uh, any f-false alarm.

[16:08](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=968s) Based on the use cases and here in this use case, uh, we should go with, uh, tail-based system because what we are trying to track here is, uh, on, uh, or the, in this specific example is when the service request started and when it ended, right?

[16:26](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=986s) So, uh, for each and every node. The other part is the cold start. Uh, if there is a new endpoint, consider the new baseline. Don't make it very generic.

[16:39](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=999s) So in that way, on the detect side, any MMD or KL, uh, could be used. Uh, and once you confirm, uh, this with ADWIN and then classify, uh, the problem, uh, that will give you where or what exact solution needs to be done and next step whatever, uh, if we can automate it, we'll automate

[17:04](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1024s) it. So what we see, uh, in general here is, uh, mean time to, uh, discovery reduced a lot.

[17:14](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1034s) To make it very real time instead of comparing or waiting for multiple windows of, uh, the time duration, it was, uh, a single window which we, uh, uh, you know, uh, which we identified, uh, helped a lot, uh, to fix the issues fast.

[17:33](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1053s) Uh, few of other things which we might need to make sure. One is, uh, the labels. Uh, when it comes to, uh, learnings, the labels helps a lot. But, uh, at the same time, uh, we need to make sure that, uh, the system is very fine-tuned in terms of that.

[17:52](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1072s) Uh, the other part is instead of saying that all the POST requests should have, you know, this is a baseline for all the POST requests, uh, try to come up with very, uh, a number which works for you, POST for payments for real-time payment or POST for wire payments or, uh, based on, you know, uh, your use

[18:14](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1094s) cases. So that, that would help a lot. Um,

[18:18](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1098s) again, uh, if anything is, uh, uh, you are considering a structural change or something, uh, the window, uh, should be well-defined.

[18:28](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1108s) For each and every client in my previous ex-- uh, example which I talked about, uh, if you can come up with a new baseline would really help, uh, to reduce the noise.

[18:38](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1118s) Uh, explainability, uh, all the data should be well explained. If you go to the doctor and doctor says your health score is twenty-two, it doesn't make much sense to you.

[18:50](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1130s) So, uh, yeah, the actual, uh, more data, uh, can explain, you know, more things to you and, uh, can, we can take the informative decision on top of it.

[19:02](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1142s) Uh, whole system should be aware of the new deployment. Uh, so, uh,

[19:07](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1147s) uh, based on that you can take either a rollback decision or not. So yeah, that's, that's about it. Uh, thanks everyone. Uh, I would like to connect with you all, uh, here is my LinkedIn. [audience clapping]

[19:20](https://www.youtube.com/watch?v=u1yaOeEX4e8&t=1160s) I can answer your questions. [outro music]
