Multi-Agent AI and Network Knowledge Graphs for Change Management and Network Testing

Talk transcript

1. 0:00

   [on hold music] Good afternoon, everyone.

2. 0:16

   My name is Ola Mabadeje. I'm a product guy from Cisco. Um, so my presentation is gonna be a little more producty than techy, but, um, uh, I think you're gonna enjoy it.

3. 0:28

   And so, um, I've been at Cisco, uh, working on, uh, AI for the last three years, and, um, I work in this group called Outshift. So Outshift is Cisco's incubation group.

4. 0:39

   Uh, our charter is to help Cisco look at emerging technologies and see how these emerging technologies can help us accelerate the roadmaps of our traditional business units. And, uh, so, um, by, uh, by training, I'm elect-electrical engineer, um, doubled into network engineering, enjoyed it, and I've been doing that for a while.

5. 0:59

   But over the last three years, focused on AI. Um, our group also focuses on quantum technology, so quantum networking is something that we're focused on. And, um, if you wanna learn more about what we do, uh, uh, with Outshift, uh, at Cisco, uh, you can learn more about that.

6. 1:14

   So, uh, for today, we're gonna dive into this, uh, real quick and, um, like I said, I'm a product guy, so I usually start with my customers' problems, trying to understand what are they trying to solve for, and then from that work backwards towards creating a solution for that.

7. 1:28

   So as part of the process for us, we usually go through this incubation phase where we ask customers a lot of questions, and when then we come up with prototypes, we do A testing, B testing, and then we kind of deliver an MVP into a production environment.

8. 1:41

   And once we get product market fit, that product graduates into the Cisco's businesses. So this customer had this issue that said, "When we do change management, we have a lot of challenges with failures in production.

9. 1:54

   How can we reduce that? Can we use AI to reduce that problem?" So we double-clicked on that problem statement, and we realized it was a major problem across the industry.

10. 2:02

    Uh, I won't go into the details here, but it's a big problem. Now, uh, for us to solve the problem, we wanted to understand, does AI really have a place here or it's just gonna be rule-based automation to, to solve this problem?

11. 2:13

    And when we looked at the workflow, we realized that there are specific spots in the workflow where AI agents can actually help address the problem. And so we, we kind of highlighted three, four, and five, where we believe that AI agents can help increase the value, uh, for customers and reduce the pain points that they were describing.

12. 2:30

    And so we sat down together with the teams. We said, "Let's figure out a solution for this." Um, and so, uh, this solution consists of three big buckets. The first one is the fact that it's a-- it has to be natural language interface where network operations teams can actually interact with the system.

13. 2:47

    So that's the first thing. And not just engineers, but also systems. So for example, in our case, we built this system to talk to an ITSM tool such as ServiceNow.

14. 2:55

    So we actually have, uh, agents on the ServiceNow side talking to agents on our side. Um, the second piece of this is a multi-agent system that sits within the, within this application.

15. 3:04

    So we have agents that are tasked at doing specific things. So an agent that is tasked as doing impact assessment, doing testing, doing, uh, reasoning around, uh, potential failures that could happen in the, in the network.

16. 3:17

    And then the third piece of this is where we're gonna spend some of the time today, which is net-network knowledge graph. So we have a n-a, the concept of a digital twin in this case.

17. 3:25

    So what we're trying to do here is to build a twin of the actual production network, and that twin includes a knowledge graph plus a set of tools to execute test, uh, testing.

18. 3:35

    And so, um, we're gonna dive into that i-in a little bit. But before we go into that, I, I, we, we had this challenge of, okay, we want to build a representati-re-representation of the actual network.

19. 3:47

    How are we gonna do this? Um, because if you know networking pretty well, networking is a very complex, uh, technology. You have a variety of vendors in a customized environment, variety of devices, firewalls, switches, routers, and so on.

20. 4:01

    And all of these different devices are spitting out data in different formats. So the challenge for us was how can we create a representation of this real world network using knowledge graphs in a data schema that can, that can be understood by agents?

21. 4:16

    And so the goal was for us to create this ingestion pipeline that can represent the network in such a way that agents can take the r- the right actions in a meaningful way and predictive way.

22. 4:26

    And so for us to, to kind of proceed with that, we had these three big buckets of things to consider. So we, we, we had to think about what are the data sources gonna be.

23. 4:35

    So if you, again, in networking, there are controller systems, there are the devices themselves, there are agents in the devices, there are configuration management systems. All of these things are all collecting data from network, or they all have data about the network.

24. 4:49

    Now, when they spit out their data, they're spitting it out in different languages, YANG, JSON, and so on. Another set of considerations to have. And then in terms of how the data is actually coming out, it could be coming out in term of streaming telemetry, it could be configuration files in JSON, it could be some other form

25. 5:04

    of, of data. How can we look at all of these three different considerations and be able to se- come up with a set of requirements that allows us to actually build a system that, that addresses the customer's pain point again?

26. 5:15

    And so, um, the team, uh, from a product side, we had a set of requirements. We, we wanted a system that, uh, a knowledge graph that can have multi-modal flexibility, uh, that means it can talk key value pairs, it understands JSON files, it understands, uh, relationships across different entities in a network.

27. 5:34

    Second thing is performance. Uh, if, if a, if an engineer is querying a, a knowledge graph, we want to have instant access to the node, the information about the node, no matter what the, the location of that node is.

28. 5:46

    That was important for our customers. The second thing was operational flexibility. So the schema has to be such that, uh, we can consolidate into one schema framework. Uh, the fourth piece here is where the, the, the RAG piece comes into play.

29. 5:59

    So we've been hearing a little about GraphRAG for, for, for a little bit today. Uh, we wanted this to be a, a system that has ability to have vector indexing in it so that when you want to do semantic searches, at some point you can do that as well.

30. 6:11

    And then in terms of just ecosystems sta- uh, um, stability, we want to make sure that when we put this in the customer's environment, uh, there's not go- no go- there's not gonna be a lot of heavy lifting that's gonna be done by the customer to integrate with their systems.

31. 6:22

    And again, it has to support multiple vendors. So these were the requirements from a product side, and then our engineering teams kind of we started to consider some of the options on the table.

32. 6:30

    Uh, Neo4j obviously, uh, market leader, uh, and various other open source tools. Uh, at the end of the day, the engineering teams decided to kinda do, uh, some analysis around this.

33. 6:39

    So I can s- I'm showing the table on the right-hand side. It's not an exhaustive list of things that they considered, but these were the things that they looked at that they wanted to see, okay, what is the right solution to address the requirements coming from product?

34. 6:52

    And, um, uh, we, we-- they kind of-- we kind of all centered around the first two here, Neo4j and ArangoDB. But for historical reasons, the team decided to go with ArangoDB because we had some use cases that were in the security space, uh, that was kind of a recommendation system, uh, type of use cases that we wanted

35. 7:09

    to kind of continue using. And so, um, but we are still exploring the use of Neo4j for some of the use cases that are coming up as part of this project.

36. 7:17

    So, um, we settled on, on, on ArangoDB for this, and, uh, we eventually came up with a solution that looks like this. So we have this knowledge graph solution.

37. 7:25

    This is an overview of it. Um, on the left-hand side, we have all of the production environment. We have the controllers, the, the Splunk, which is a SIEM system, traffic telemetry coming in.

38. 7:36

    All of them are coming into this ingestion service, uh, which is doing an ETL, transforming all of this information into one schema, OpenConfig. So OpenConfig schema is a schema that is designed around networking primarily, and, uh, it helps us to-- because it-- there's a lot of documentation about it on the internet, so LLMs understand this very well.

39. 7:55

    So, um, this setup is primarily a, a database of, uh, of, uh, networking in- information that has OpenConfig schema as a primary way for us to communicate with it.

40. 8:06

    So, uh, natural language communication through an individual engineer or the agents that are actually interacting with that system. And so we built this in the form of layers, so i- uh, if you, if you're, if you're in into networking again, um, there is a set of entities in a network that you want to be able to interact

41. 8:24

    with. Uh, so we have layered this up in this way such that if, uh, there's a tool call or there's a decision to be made about, uh, a test, for example.

42. 8:32

    Let's say you wanna do a test about, uh, configuration drift as an example. Um, you don't need to go to all of the layers of the graph. You just go straight down to the raw configuration file and be able to do your compari- comparisons there.

43. 8:44

    If you are trying to do like a test around reachability, for example, then you need a couple of layers, maybe you need raw configuration layers, data cont- data plane layers, and control plane layers.

44. 8:52

    So, um, it, it's structured in a way that when the agents are making their calls to this system, uh, they understand what the request is from the, from the, uh, system, and they're able to actually go to the right layer to pick up the information that they need to ex- to execute on it.

45. 9:07

    So this is kind of a high-level view of what the graph, uh, system looks like in layers. Now, um,

46. 9:13

    I'm going to kind of switch geas- switch gears now and go back to the system. Remember I described a system that had agents, a knowledge graph, a natural language digital twin, as well as natural language interface.

47. 9:24

    So let's talk about the agentic layer, and before I kind of talk about the specific agents in, um, in this system on this application, we are looking at how we are going to build a system that is based on open standards for all of the internet.

48. 9:38

    And this is one of the challenge we have within Cisco. We, we are looking at a system, uh, a, a set of a, um, collective, open source collective that includes all of the partners we see down here.

49. 9:47

    So we have, uh, Outshift by Cisco, we have LangChain, Galileo. We have all of these, uh, members who are supporters of this, uh, of this collective. And what we are trying to do is to set up a system that allows agents from across the world, uh, so it's a big vision, uh, that they can talk to each

50. 10:04

    other without having to do heavy lifting of reconstructing your agents every time you want to integrate them with another agent. So it consists of identity, uh, schema framework for defining an agent skills and capabilities, the directory where you actually store these agents, and then how you actually compose the agent, both at the semantic layer and the synthetic

51. 10:22

    layer. And then how do you observe the agents in process? All of these are part of this collective's, uh, vision, uh, as, as, as a group. And if you want to learn more about this, it's on agency.org, and I also have a, a slide here that kinda talks about, um, there's real code actually that you can leverage

52. 10:37

    today. Or if you want to contribute to the code, uh, you can actually go there. There's a GitHub repo here that you can go to and, and, and you can start to contribute or use, use, use the, use the data.

53. 10:47

    Um, there's documentation available as well, and there's sample applications that allows you to actually see how this works in real life. And, uh, um, we know that there's MCP, there's A2A.

54. 10:56

    All of these protocols are becoming, uh, very popular. Uh, we also integrate with all of these protocols because the goal again is not to, uh, create something that is bespoke.

55. 11:05

    We want to make it open to everyone to be able to create agents and be able to make these agents work in production environments. So back to the specific application we're talking about.

56. 11:14

    Based on this framework, we delivered this set of agents. Uh, we built this set of agents a- as a group. So we have five agents right now as part of this application.

57. 11:23

    Um, there's an assistant agent that's kind of the planner that kind of orchestrates things across the globe, a- across all of these agency-- agents. And then we have other agents that are all based on ReAct reasoning loops.

58. 11:33

    There's one particular agent I wanna call out here, the query agent. This query agent is the one that actually interacts directly with the knowledge graph on a regular basis.

59. 11:41

    Um, we had to fine-tune these agents because, um, we initially started by doing a, uh--

60. 11:47

    attempting to use RAG to do some querying of the knowledge graph, but that was not working out well. So we decided that for immediate results, we are gonna fine-tune it, and so we did some fine-tuning with-- of, of the, of, of this agent with some schema information as well as example queries.

61. 12:03

    And so that he- helped us to actually reduce two things, the number of tokens we were burning, because every time we're-- before that, the AQL queries were going through all of the layers of the knowledge graph And in a, in a reasoning loop was consuming lots of tokens and taking a lot of time for it to resolve,

62. 12:17

    to return results. After fine-tuning, we saw a drastic reduction in number of tokens consumed, as well as the amount of time it took to actually come back with the results.

63. 12:25

    So that kind of helped us there. Um, so, um, I'm gonna kind of pause here. I'm talking a lot about s- uh, there's a lot of slide we're here.

64. 12:32

    I want to show a quick demo of what this actually looks like. So tying together everything from the, uh, natural language interface interaction with an ITSM system to how the agents interact, to how that collects information from knowledge graph and delivers results to the customer.

65. 12:46

    Okay. Yeah. So, um, the scenario we have here is a, a network engineer wants to make a change to a firewall rule. They have to do that to accommodate a new server into the network.

66. 12:58

    And so what they need to do is to, first of all, start from ITSM. So they submit a ticket in, uh, ch- in, in their, in, uh, ServiceNow. Now, our system here, the, the V- the UI I'm showing here right here is the UI of the actual system we built, the application we built.

67. 13:14

    We have ingested information about the, uh, tickets here in natural language, and so the agents here are able to actually start to work on this. So I'm gonna play a video here just to make it, uh, uh, more relatable.

68. 13:27

    So the first thing that's happening here is that these agents, uh, the first agent is asking that the inter- for the, for the in- information to be synthesized in a summarized way so that they can understand, uh, what to quickly do.

69. 13:41

    The next action that has been asked here is for it to create an impact assessment. So impact assessment here just means I want to understand, will this change have any implications for me beyond the immediate, uh, target area?

70. 13:54

    And that's gonna be summarized, and we are now gonna ask the agent that is responsible for this particular task to go and attach this information into the ITSM ticket.

71. 14:03

    So I'm gonna say, uh, attach this information about the impact assessment into this, the ITSM ticket. So that's been done. Now, the next step is to actually create a test plan.

72. 14:14

    So test plan is one of the biggest problems that our customers are facing. Um, they, they, they run a lot of tests, but they miss out on the right test to run.

73. 14:22

    So this agent's actually able to reason through a lot of information about test plans across the internet, and based on the intent that was ca- collected from the ServiceNow ticket, is gonna come up with a list of tests that you have to run to be able to make sure that this firewall rule change doesn't make a big

74. 14:37

    impact or create problems in production environment. So as you can see here, this agent has gone ahead and actually listed all of the test cases that needs to be run and the expected results for each of the tests.

75. 14:47

    So we are gonna ask this agent to attach this information again back to the ITSM ticket because that's where the approval board needs to see this information before, before they implement, before they approve the implementation of this change in production environment.

76. 15:00

    So we can see here that that information has now been attached back by this agent to the ITSM ticket. So two separate systems, but agents talking to each other.

77. 15:09

    Now, the next step is to actually run a test, uh, on all of these test cases. So, um, in this case, the configuration file that is gonna be used to make the change in the firewall is sitting in a GitHub repo, and so we are gonna do a pull request of that config file and gonna take that

78. 15:23

    information. So this is the GitHub repo where the, where we're gonna do a pull request. We're gonna take the link for that pull request and paste it in the ticket, an ITSM ticket, and so that when the executor, execution agent starts doing its job, it's actually gonna pull from that and use it to run its test.

79. 15:39

    So, um, at this moment we are, we are, uh, we are gonna start running the test. We're gonna ask this agent to go ahead and actually run the test and execute on this test.

80. 15:48

    And so, um, I have attached the change. Sorry, my, I don't have my glasses. I've attached my, uh, change candidates to the ticket. Can I go ahead and run the test?

81. 15:59

    So what is gonna happen here is if you look on the right-hand side of the screen here, a series of things are happening. The first thing is that the, this agent called the executor agent goes, looks at the test cases, and then it goes into the knowledge graph, and it's gonna go ahead and actually do a snapshot

82. 16:14

    of the most recent visual or most recent information about the network. It's now going to take the pull request that it pulled from GitHub, the snapshot it just took from the knowledge graph.

83. 16:25

    It's gonna compute it together and then run all of the individual tests one at a time. So we can see that it's running the test one test, test one, test two, test three, test four.

84. 16:34

    So all of this is happening in what we call a digital twin. So a digital twin again is a consi- co- combination of the knowledge graph, a set of tools that you can use to run a test.

85. 16:43

    So an, an example of a tool here could be Batfish, uh, or could be Route53, or some other tools that you use for engineering pur-- for network engineering purposes.

86. 16:50

    So once all of these tests are completed, uh, this tool actually is gonna, this agent is gonna now generate a report about the test results. So, um, uh, we give you some time to run through this.

87. 17:00

    It's still running the tests, uh, but one it's, once it concludes all of the tests, it's going to report actually, uh, the, the test results are. So which results, which tests actually passed, which ones failed.

88. 17:11

    For the ones that have failed, it's gonna make some recommendations on what you can do to go and fix the problem. Um, um, I'm gonna skip to the front here to just quickly get this on, uh, done quickly because of time.

89. 17:21

    Um, so, um, it's attached the results to the ticket, and this is the report that it's spitting out in terms of this is the report for the test that were run.

90. 17:30

    So this executor agent actually created a report about all of the different test cases that were run by the system. So, um, very quick short demo here. Uh, there's a lot of detail behind the scenes, but, uh, I can answer some questions offline.

91. 17:44

    Um, the, the, the couple of things I wanna leave us with is that, uh, before I go to the end of this, uh, is that evaluation is very, very critical here for us to be able to, on, to understand how this delivers value to customers.

92. 17:55

    Um, we are looking at a variety of things here, so the agents themselves, the knowledge graph/digital twin, and we are looking at the what can we actually measure quantifiably.

93. 18:05

    Now, for the knowledge graph, we're looking at extrinsic re- re- extrinsic metrics, uh, particularly not intrinsic ones because we want to map this back to the customer's use case.

94. 18:14

    So this is the summary of the, of, of, of what we see in terms of evaluation metrics. Um, we are still learning. This is a, this is for now, for now it's, it's an MVP.

95. 18:23

    Um, but what we are learning so far is that those two key building blocks, the knowledge graph and the open framework for building agents, is very critical for us to actually build a scalable sy- sy- system for our customers.

96. 18:34

    And so, um, I'm gonna stop. It's eight seconds to go. Thank you for listening to me, and then if you have questions, I'll be out there. [clapping] [upbeat music]