Learned Execution Graphs for Anomaly Detection & Drift in APIs — Ritvik Pandya, JP Morgan Chase

Talk transcript

1. 0:00

   [upbeat music] Hi. Uh, thanks, thanks, Na.

2. 0:17

   And, uh, hope everyone is out of, uh, the lunch coma and [laughs] will survive this talk. So, uh, yeah, myself Ritvik, I, uh, lead the payments team in, uh, JP Morgan.

3. 0:30

   And, uh, today I'll be talking about learned execution graphs, how these graphs can help to, uh, detect any anomaly and, uh, drifts. Also, how we can automate a few things around that and, uh, you know, um,

4. 0:50

   uh, at the same time, if we can reduce the manual, you know, detection work and, uh, going on that side. Uh, so whenever we hear about graph, uh, there are persistence graph and property graphs, uh, which Neo4j and, you know, other products, uh, we

5. 1:15

   use for them. We query those, uh, graphs and get the answers out of it. What I'm talking about today is execution graph. It's short-lived graph. Uh, and idea here is holistically try to identify how the request processing happens, and if there is any deviation on that, and how to detect that, and how to fix that.

6. 1:41

   So here is a simple example. Uh, say we have set of applications. Uh, you have one edge layer, uh, the first layer where, you know, request comes in. And then, uh, you have some gateways.

7. 1:56

   Uh, if gate is there, you have ingress layer on top of it. Then authentication, authorization happens. After that, there is some orchestration layer and a few other systems which could be called in parallel.

8. 2:10

   Uh, once everything is done, you are notifying your client that what's the update on that request, right? So here, the idea is representing this, uh, overall request processing as DAG.

9. 2:25

   And using DAG simplifies most of the things here.

10. 2:30

    One, now you know that, uh, in what order service execution will be happening, right? So that's one of the thing. The other thing is, uh, you know the context that in-- at what node, what context will be there and what will be passed to the next node.

11. 2:49

    Uh, in that way, it will be very, um, you know, ordered and simplified, um, uh, simply can be represented. Uh, there are a few other, uh, use cases could be there, uh, in terms of retries and, uh, the loops, et cetera.

12. 3:05

    Uh, the idea here is, uh, every loop, uh, to put in the graph as a separate entity itself. So, uh, in that way, it could be tracked, uh, easily.

13. 3:23

    How we can make this system, uh, more, uh, reliable at the same time not using, uh, most of the resources, right? So in the tier one check or ACR first check, it's like, uh, going to airport and you, you, you know, it's just boarding passes.

14. 3:42

    Some-someone is looking at the boarding pass and let you go. So, uh, now if you know the baseline of your request execution end to end, if, uh, everything looks good, you don't need to go to the tier two, uh, or next tier of check, right?

15. 3:57

    Once if you find that there is some delay, so now you need to check that what changed here. One of the, uh, the drift here could be because of the structural change.

16. 4:11

    So if any new node or new step added, which you are not aware of, that could be one of the thing or one of the step which is removed, that could be the another reason, right?

17. 4:22

    Uh, once you know about that, then further, uh,

18. 4:29

    further analysis could be done, uh, in terms of KL deviations or divergence and, uh, exponential MA. Uh, so in simpler terms, if you know that client A's request is taking this much time normally, and, uh, client B's request could take-- might take more time than the client A, uh, because of, say, one client is

19. 4:54

    local to you and one client is, uh, you know, the request is coming from outside, and, uh, there are a few more checks needs to be done. So in, in that case, the baseline will change client to client.

20. 5:06

    And now you know that, uh, what your threshold it and, uh, how you can, uh, reduce the noise of such alerts.

21. 5:16

    Uh, so here the idea is very simple.

22. 5:22

    First, you, uh, represent the entire request processing as DAG. You come up with the baseline, you find out the deviation, and then you try to find out where exactly the issue is.

23. 5:38

    Once you localize that, then you compare that based on your system that whether it is, uh, within your threshold or not. If it is within the threshold, yeah, you don't need to, uh, you know, do the alerts or automate anything.

24. 5:53

    But if it is out of the threshold, then certain action needs to be taken.

25. 6:00

    Uh, coming back to our example here, say overall, uh, request processing From all the different nodes within our system, uh, is happening, but somehow the foreign transaction, uh, rate service is taking more time than the usual.

26. 6:19

    Now, if you represented this whole, uh, request processing in multiple nodes, you know where the problem is or where the issue is. And correspondingly, you will, uh, you know, now you can, uh, exactly know where the problem is, so you can solve it, uh, that what-- how to, uh, how FX system or what all different, uh, cases

27. 6:42

    were there in the past where FX, FX rate system was failing.

28. 6:50

    Here is one of the example, uh, for benchmark, uh, OpenTelemetry and that Starbench were used. And say for seven days of the time, millions of, uh, traces were, um,

29. 7:06

    you know, injected or, uh, in the system. Then you ca-inject the problem or, uh, anomaly there. And based on that, you train your system before a-anything goes on live.

30. 7:21

    So, uh, again, uh, basic thing here is what is anomaly and what is drift, right? So say you are driving from your home to office every day, and, uh, one of-- uh, a-and usually it takes one hour.

31. 7:39

    But, uh, one fine day, it took you more time than one hour. Uh,

32. 7:44

    the reason might be some traffic or, you know, um, car accident or anything. But, uh, he-- this is one of the incidents and, uh, based on your system and criticality of your system, uh, you can decide how to address that.

33. 8:01

    The other part is one fine day you are, uh, taking sip of coffee around four PM and realize that a year back, it used to take one hour for you from your home to office.

34. 8:13

    But nowadays, it, it is taking twenty more minutes, right? So what happened? In Bay Area, your number of car increases or traffic increase. So this is over the time, what you are seeing is pattern changed, and that's where you might need to come up with, uh, the new baseline itself.

35. 8:36

    So, uh, that's, that's the drift that over the time you start seeing some delays or, you know, some, uh, performance deviation. Then once you know that there is a drift, uh, you can further categorize it.

36. 8:51

    First category is structural. Uh, so say you-- somehow in the system, a new node is added or one of the node is removed, as I mentioned earlier. For example, you like, uh, again, you know, there is a shop where you like, uh, drinking coffee and, uh, one fine day, they are ask-- start asking you about membership.

37. 9:13

    So they added one more, uh, step in it. Now, every day, they might ask you for, "Hey, do you have membership with us? If you have, then there is, there are a special discount for you.

38. 9:22

    If you don't have membership, then, uh, the regular prices will be there." So in that way, uh, you know, same way in our service processing or a request processing, if new node is added, that means, uh, now you need to consider that step also in your all baselines and new alerts.

39. 9:43

    The other one is, uh, say, because of the volume of request, uh, one of your service is taking more time or it's not, you know, uh, cannot serve the request or, or the volume which you are expecting now over the time.

40. 10:04

    So yeah, such kind of drifts you might need to treat differently because now you ne-- either you need to scale up those services or instances of those services and/or you need to either make it asynchronous call or based on, based on the use case, you know, what-whatever, uh, works there.

41. 10:22

    Uh, covariate is different, uh, one of the category. Say when you started the business, uh, you were seeing

42. 10:32

    around sixty percent of local requests, but, uh, and forty percent, uh, requests from, you know, uh, out of the country. And that's where you might need to change the currency or, you know, one extra step is there.

43. 10:46

    So you-- now you know that what is the baseline for your, uh, request in US dollar, but, uh, what is the baseline for, uh, any of the other currency.

44. 10:57

    Over the time, what happens is your product is so popular that you started getting more, uh, requests from the outside. So nothing changed. Your system is working fine, right?

45. 11:09

    But now you need to come up with the criteria and reassess your baselines again, sorry, here, where, uh,

46. 11:19

    either you need to come up with two different, uh, you know, uh, graphs to compare. That one is for local and one is for, uh, outside-- uh, request from outside.

47. 11:31

    Or what you can do is you can increase your, uh, average request time baseline. So once you know the pattern, uh, you know the solution. So, uh, that's where you need to ca-categorize, uh, these drifts.

48. 11:48

    One category could be for the same request, uh, now you are seeing the different behavior itself. Then probably if and when it's needed, you might need to roll back such, uh, changes or either you need to reconsider that.

49. 12:06

    So where I'm going with this is In that way, you need to reevaluate and reassess your system before, uh, identifying that what action needs to be taken.

50. 12:18

    This whole talk is mostly about statistical, uh, uh, you know, part of, uh, uh, the solution. Uh, it's, it's, uh, part of bigger neural, uh, specific, uh, um, algorithms and, uh, system in a way, but this is just one of the module which, uh, I'm talking about here now.

51. 12:41

    So once you know the drift or deviation, and here is the simple, uh, DAG for how, uh, this whole system would work.

52. 12:58

    OpenTelemetry, uh, will keep feeding the data. Once you have that data, uh, root cause analysis could be taken based on, uh, once your system knows about all the data points.

53. 13:11

    Now you know which type of drift it is and what solution could be there, then

54. 13:19

    you identify what action needs to be taken. Once you know what action needs to be taken, further you need to also, uh, find out that what is the risk if we go with this approach or if we automate this, uh, solution, right?

55. 13:33

    So once you know the risk, either you can go with, uh, roll out that sys-- uh, um, solution for say, five percent or ten percent of your s- uh, machines, monitor it, verify everything looks good, and then you roll out for your hundred percent of the nodes.

56. 13:53

    Here are a couple of example. Uh, again, uh, say generally overall request processing takes seven hundred milliseconds. But, uh, on the left side what you are seeing is, uh, in the graph approach itself how it could help you is, uh, now you know that, uh, which specific node is taking more time, suddenly you get alert on that.

57. 14:15

    Or otherwise, if you are seeing that, uh, the delay is all across, then something which is common which you need to fix here.

58. 14:26

    All these things, because in the payments and the real-time, uh, payment processing, we want to keep it very faster, right? So

59. 14:36

    we don't want delay the actual request processing. The solution which we generally use, uh, everyone in the industry is asynchronously feeding the data to, uh, OpenTelemetry. From there, uh, some Kafka could be used and, uh, stream assessment, uh, could be used on top of it.

60. 14:56

    There could be two different paths. One is, say, hot path where you can take a decision very faster and, uh, um, work on the solution or automate that solution.

61. 15:07

    The other one is more recon kind of solution where it might take some time, but more, more accurate could-- it could be.

62. 15:21

    Few of, uh, uh, other challenges which, uh, we need to fine-tune here. So all I talked about is, "Hey, you have seven nodes in your system and every node is feeding the data to your, uh, telemetry."

63. 15:35

    What if one of the system is delaying the event?

64. 15:41

    Should we consider it as a structural change? Because now what you have data in your system is for six nodes and the seven, uh, no-- uh, the data from the seven node is delayed already.

65. 15:53

    So we need to fine-tune that, uh, that, uh, those numbers also that when to consider that, uh, there is a structural change or not. So basically here we are, uh, trying to reduce, uh, any f-false alarm.

66. 16:08

    Based on the use cases and here in this use case, uh, we should go with, uh, tail-by base system because what we are trying to track here is, uh, on, uh, or the, in this specific example is when the service request started and when it ended, right?

67. 16:26

    So, uh, for each and every node. The other part is the cold start. Uh, if there is a new endpoint, consider the new baseline. Don't make it very generic.

68. 16:39

    So in that way, on the detect side, any MMD or KL, uh, could be used. Uh, and once you confirm, uh, this with admin and then classify, uh, the problem, uh, that will give you where or what exact solution needs to be done and next step whatever, uh, if we can automate it, we'll automate

69. 17:04

    it. So what we see, uh, in general here is, uh, mean time to, uh, discovery reduced a lot.

70. 17:14

    To make it very real time instead of comparing or waiting for multiple windows of, uh, the time duration, it was, uh, a single window which we, uh, uh, you know, uh, which we identified, uh, helped a lot, uh, to fix the issues fast.

71. 17:33

    Uh, few of other things which we might need to make sure. One is, uh, the labels. Uh, when it comes to, uh, learnings, the labels helps a lot. But, uh, at the same time, uh, we need to make sure that, uh, the system is very fine-tuned in terms of that.

72. 17:52

    Uh, the other part is instead of saying that all the post requests should have, you know, this is a baseline for all the post requests, uh, try to come up with very, uh, a number which works for you, post for payments for real-time payment or post for wire payments or, uh, based on, you know, uh, your use

73. 18:14

    cases. So that, that would help a lot. Um,

74. 18:18

    again, uh, if anything is, uh, uh, you are considering a structural change or something, uh, the window, uh, should be well-defined.

75. 18:28

    For each and every client in my previous ex-- uh, example which I talked about, uh, if you can come up with a new baseline would really help, uh, to reduce the noise.

76. 18:38

    Uh, explainability, uh, all the data should be well explained. If you go to the doctor and doctor says your health score is twenty-two, it doesn't make much sense to you.

77. 18:50

    So, uh, yeah, the actual, uh, more data, uh, can explain, you know, more things to you and, uh, can, we can take the informative decision on top of it.

78. 19:02

    Uh, whole system should be aware of the new deployment. Uh, so, uh,

79. 19:07

    uh, based on that you can take either a rollback decision or not. So yeah, that's, that's about it. Uh, thanks everyone. Uh, I would like to connect with you all, uh, here is my LinkedIn. [audience clapping]

80. 19:20

    I can answer your questions. [outro music]