Multi-Agent AI and Network Knowledge Graphs for Change Management and Network Testing

How can a network operations team reduce production failures caused by its own changes? That customer question anchors Ola Mabadeje’s walkthrough of work at Cisco Outshift. Outshift incubates emerging technologies, including AI and quantum networking, to accelerate Cisco’s business roadmaps. Its product process moves from customer interviews through prototypes and A/B testing to a production MVP, with graduation into a Cisco business after product-market fit. Here, the recurring customer problem was change management: a planned modification could still create an unplanned outage.

The first design decision was whether AI belonged in the workflow at all. Some problems yield to rule-based automation; others require interpreting intent or reasoning about consequences. The team examined the change-management workflow and identified particular stages where agents might help. The unit of automation was a specific piece of change-validation work, rather than the entire operational process.

The application has three cooperating components:

• Natural-language interaction: Network operations engineers describe their requests, and external systems supply change information. ServiceNow-side agents communicate with agents in the application, so the interface is not limited to a person typing into a chat window.
• Specialized agents: Different agents perform impact assessment, testing, and reasoning about potential network failures.
• A network digital twin: A knowledge graph represents the production network, while a set of tools executes tests against that representation.

The distinction between the graph and the twin matters. A graph supplies network information; the digital twin combines that information with tools that can test a change.

Building that representation is a substantial part of the problem. A customized network can contain firewalls, switches, and routers from multiple vendors, each producing information in different forms. An agent needs a coherent account of those devices and their relationships before it can choose a meaningful action or anticipate a change’s effects.

The ingestion design separates three questions:

These categories are related but not interchangeable. YANG describes data models; JSON represents data. A telemetry stream and a configuration file also have different delivery characteristics, even when they describe the same device. The pipeline must reconcile those differences into a representation agents can use.

The product requirements extended beyond storing edges between devices:

• Multi-model flexibility: Support key-value data, JSON documents, and relationships between network entities.
• Fast access: Let an engineer retrieve node information promptly, regardless of the node’s location. This was a requirement, not a reported latency measurement.
• Operational flexibility: Consolidate heterogeneous information into one schema framework.
• Semantic retrieval: Support vector indexing so semantic search and GraphRAG could become available.
• Practical integration: Minimize customer integration work, use a stable ecosystem, and support multiple network vendors.

The engineering evaluation narrowed toward Neo4j and ArangoDB. The team chose ArangoDB for historical continuity with security recommendation use cases it already wanted to support. That is a reuse decision, not evidence that ArangoDB outperformed Neo4j on a published benchmark. Neo4j remained under exploration for additional use cases in the project.

Production controllers, Splunk/SIEM information, and traffic telemetry feed an ingestion service. Its ETL process transforms those inputs into a common representation based on OpenConfig. More precisely, OpenConfig is a family of vendor-neutral network data models written in YANG, rather than one universal schema or a serialization format. Its models cover configuration and telemetry without attempting to represent every feature. Mabadeje’s rationale for choosing it also includes its public documentation, which he expects to make it familiar to LLMs; that expectation is not a measured model-accuracy result.

Engineers and agents access the resulting network information through natural language, but the underlying graph is organized into layers. A task should retrieve the layers it actually needs. For configuration drift, an agent can go directly to raw configuration files and compare them, without traversing the rest of the network representation.

A minimal AQL example makes that access pattern concrete. In an illustrative raw_configurations collection with one record per device, the query selects a device’s raw configuration directly:

FOR config IN raw_configurations
  FILTER config.device_id == @device_id
  RETURN {
    device_id: config.device_id,
    configuration: config.configuration
  }

The collection and field names here express the example’s data shape. The architectural point is the narrow retrieval: fetch the configuration needed for comparison instead of exploring unrelated graph layers.

Reachability is a different question. Determining whether traffic can get from one point to another may require raw configuration, data-plane information, and control-plane information together.

The request determines the information boundary. Layering gives the agents a way to obtain enough context for a test without treating every request as a query over the entire network.

The next architectural concern is interoperability. The application needs agents in different systems to cooperate without rebuilding each agent for every integration. AGNTCY provides the broader open ecosystem vision behind this work. Mabadeje names Outshift by Cisco, LangChain, and Galileo among the collective’s supporters.

That vision includes agent identity, schemas describing skills and capabilities, directories for storing and discovering agents, composition at both the semantic and syntactic layers, and observability during execution. Agreeing on message structure is only part of the problem: collaborating systems also need to understand what an agent can do and observe what happens when it does it.

The project offers code, contribution opportunities, documentation, and sample applications. Mabadeje also describes integration with MCP and A2A as part of the goal of an open production ecosystem. This is the interoperability direction presented with the MVP; today’s AGNTCY component catalog should not be read as the exact implementation or protocol versions used in the recording.

The application contains five agents. An assistant agent plans and orchestrates their work, while the other agents use ReAct reasoning loops, interleaving reasoning with actions and their observations. The query agent has a particularly important role: it regularly interacts directly with the knowledge graph.

The team initially tried RAG for graph querying, but found that approach unsatisfactory for this application. It then fine-tuned the query agent using schema information and example queries. The fine-tuning supplied knowledge about how the graph was organized and how to ask it useful questions.

Before fine-tuning, AQL queries went through all the graph’s layers inside the reasoning loop, consuming tokens and taking time to return results. Mabadeje reports a substantial reduction in token consumption and response time after fine-tuning the query agent on schema information and example queries. He supplies no numerical benchmark or controlled evaluation conditions for that comparison. The practical lesson is specific to this system: making graph access more selective addressed an expensive part of the agent loop.

The recorded demonstration ties the architecture together with a concrete request: a network engineer needs to change a firewall rule to accommodate a new server. The engineer starts by submitting a ServiceNow change ticket. The application ingests its natural-language information, giving the agents the proposed change’s intent.

The first action synthesizes that information into a summary so the agents can determine what work to perform. Next comes impact assessment: could this change have consequences beyond its immediate target? After producing the assessment, the responsible agent is asked to attach it to the ITSM ticket. The ticket becomes the shared record connecting the engineer’s request to the application’s analysis.

Once the impact assessment is attached, the workflow moves to test planning. The customer problem is not simply a shortage of tests: teams can run many tests while missing the ones that matter. Test selection must follow the change’s intent.

The planning agent uses the intent captured in the ServiceNow ticket, together with test-plan information Mabadeje describes as coming from across the internet, to propose tests for the firewall change. Its output contains both test cases and expected results. Those expected results establish what each test should demonstrate before execution begins.

The agent then attaches the plan to the ITSM ticket. That is where the approval board needs the evidence before approving production implementation. The demonstration shows two systems exchanging work through agents while preserving the ticket as the review surface; it does not show agents autonomously approving the production change.

The proposed firewall configuration lives in a GitHub repository. A pull-request link is placed in the ITSM ticket so the execution agent can retrieve the change candidate. With that reference attached, the user asks the agent to run the tests. This connects the natural-language request and test plan to the configuration that will actually be evaluated.

Execution proceeds through a concrete sequence:

1. Read the test cases associated with the change.
2. Snapshot the latest available network information from the knowledge graph.
3. Retrieve the proposed configuration through the GitHub pull-request reference.
4. Combine the candidate configuration with the network snapshot.
5. Run the individual tests one at a time.

The candidate is evaluated in the context of the modeled network, rather than considered as an isolated configuration file.

This is the digital twin in operation: network information from the graph plus tools capable of running network tests. Mabadeje names Batfish as one possible tool and also mentions Route 53 among examples. The important boundary is that the demonstration runs tests in the twin; combining the candidate with a snapshot is not deployment of that configuration to the production firewall.

After execution, the agent generates a report identifying which tests passed and which failed. For failed tests, it provides recommendations for fixing the problem. Mabadeje advances the recording while tests are running, so the demonstration does not establish an end-to-end execution duration.

The results are attached to the ITSM ticket, and the application displays the executor’s report covering the test cases run. The demonstrated workflow ends with evidence returned to the change record: impact assessment, a test plan, and execution results are available for review before any subsequent production rollout.

Evaluation must cover both the agents and the knowledge graph/digital twin. For the graph, Mabadeje emphasizes extrinsic metrics tied to the customer’s use case rather than intrinsic graph measures alone. The question is whether the representation helps the system perform useful network-change work. The accompanying evaluation table spans agent quality, topology and configuration accuracy, network performance, test completion time, and testing cost. These are evaluation areas, not measured outcomes presented in the walkthrough.

The system is still an MVP, and the team is still learning how to measure its value. Mabadeje identifies two foundations for scaling it: a knowledge graph that makes the network usable by agents, and an open framework that lets those agents cooperate across systems. Together, they support a change workflow in which proposed configuration changes can be assessed and tested, with the resulting evidence returned to the people responsible for approval.