AI Engineer World's Fair 2025
Grounded Reasoning Systems for Cloud Architecture
Read the talk
Grounded Reasoning Systems for Cloud Architecture
Catio’s architecture copilot connects requirements to infrastructure graphs, coordinates specialist agents, and uses human review to turn proposed changes into inspectable design decisions.
From a talk by Iman Makaremi
Before you start: Familiarity with cloud components, vector retrieval, and LLM agents will help; no specific orchestration framework is required.
Reasoning from scattered requirements
How should an architecture change when its requirements, constraints, and current state are scattered across different sources? More users, developers, tools, and expectations create more decisions than a static automation workflow can anticipate. A useful system must understand the situation, consider competing options, justify a choice, and plan the work. This is the problem behind Catio’s cloud-architecture copilot, introduced by Iman Makaremi, its co-founding head of AI.
Architects negotiate trade-offs among requirement clarity, delivery time, and available resources. Much of the context behind those decisions is implicit. Capturing it for AI requires more than inventorying infrastructure: the system must understand how an architect uses that context to make a decision.
Catio divides the work into three connected problems:
- Requirement understanding: Identify where requirements come from, how they are represented, which parts matter, and whether they apply globally or to a particular part of the system.
- Architecture identification: Determine what each component does. Its type alone is insufficient; its position and role in the architecture affect its function.
- Architecture recommendation: Compare the requirements with the current architecture, then propose changes that close gaps or bring the system closer to best practices.
A recommendation therefore depends on both an interpretation of intent and an account of what the infrastructure actually supports.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connecting text, graphs, and questions
The first AI challenge is that the two main sources of evidence have different shapes. Requirements mostly live in text documents; architecture is inherently graph data. The system must connect statements about desired behavior to components and relationships in the deployed design.
The second challenge is decomposition. A broad or vague architecture question rarely maps to one retrieval call: it needs to be broken into smaller questions and a plan for answering them. The third is evaluation. When many moving parts contribute to a recommendation, feedback must help identify which part of the system needs to improve.
Grounding starts with getting the right context into the reasoning process. Translating a natural-language request into a meaningful architecture retrieval task is already difficult; doing it quickly adds another constraint. Catio consequently explored architecture retrieval and requirement retrieval as separate, related problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Enriching retrieval without losing relationships
Catio first improved the information available to its retrievers in several ways:
- Semantic component enrichment: Add relevant descriptive information to each architecture component so vector search can find it more easily.
- Graph-enhanced component search: Use graph algorithms alongside searches for a component or component type to retrieve the appropriate architectural context.
- Requirement scoring: Identify concepts and requirements important to the organization, then score documents against them before retrieval.
Makaremi reports that scoring requirement documents in advance made retrieval faster. He does not provide a measured speedup.
Semantic grounding helped, but it did not solve every retrieval problem. Catio still encountered limits when scaling searches or seeking detailed answers. Soft grounding also depended on prompt design: the agent needed instructions about what to focus on and what to retrieve when the system could not supply an exact structure for the task.
Graph memory supports continuity as well as accuracy. Finding individually relevant nodes is only part of the job. The system also needs to connect those nodes and explain why they appeared together, so subsequent reasoning retains the relationships that made the evidence meaningful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two retrieval pipelines
The early architecture retriever separated storage from querying. Its storage path prepared a searchable representation of each component:
- Break the architecture into individual components.
- Convert each component’s JSON data into natural language.
- Enrich that description with connection data.
- Embed the enriched description and store it in a vector database.
At query time, the system embedded the search request and used it to retrieve matching components. The connection enrichment mattered because the indexed object needed to convey more than its isolated properties.
This pipeline produced useful results, but semantic similarity did not adequately handle the graph-shaped data. Catio first augmented the vector approach with graph searches, then began moving toward a design based more heavily on a knowledge graph. That progression preserves a useful distinction: describing a component’s connections in text is not the same operation as searching the relationships themselves.
The requirements pipeline began with a familiar document-processing sequence: preprocess the input, split it, and embed the resulting pieces. Catio added requirement templates to define the structure of extracted information. The templates made extraction answer to the needs of downstream recommendation agents, rather than merely producing arbitrary snippets of relevant text.
Makaremi reports that requirement templates helped organize business requirements and retrieve useful pieces faster. But as document counts grew, larger searches sometimes lost context. Graph analysis was a possible way to recover relationships across the corpus, extending the same concern that had appeared in architecture retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structured communication and isolated memory
Retrieving evidence does not remove conflicting architectural goals. Good design still requires debate and trade-offs. Catio built a multi-agent system, or MAS, with role-specific agents that could collaborate and converge on justified recommendations. Structured messages made those interactions easier to coordinate across longer chains of work. Early versions generated and parsed XML; Makaremi describes that as a burdensome implementation stage the team had moved beyond.
Multiple levels of interaction required explicit memory boundaries. Catio isolated conversations to avoid spending tokens on unrelated exchanges. Makaremi reports that Catio’s experiments associated larger agent memory with more hallucination. No quantitative effect or evaluation protocol is supplied, but the observation influenced the decision to keep agent memories isolated.
Parallel processing introduced another memory decision: when cloning an agent, what should be copied, and from which point in its history? A clone needed the context relevant to its task without allowing all concurrent conversations to become one growing shared transcript.
Structured output provided clarity and programmatic control over those workflows. Makaremi acknowledges a potential trade-off between output constraints and a model’s reasoning abilities, while reporting good results at Catio’s chosen level of structure. The engineering question is how much structure the workflow needs to remain controllable while leaving room for the agent to solve the task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Dynamic reasoning inside explicit control flow
The orchestration does more than execute a fixed plan. Agents can resolve trade-offs dynamically, and Catio continues to experiment with what each agent does at each interaction step. Makaremi associates more dynamic orchestration with greater creativity in planning. But that flexibility still needs explicit control flow: letting agents converse without governing how they reach a result is not sufficient.
Makaremi describes the recommendation engine as a working production part of Catio’s stack, while identifying the displayed recommendations as using mock data. The interface groups recommendations into categories such as data architecture, messaging and queuing, and API integration.
Each recommendation is a structured design artifact:
| Part | Purpose |
|---|---|
| Description | Explain the recommended change |
| Target state | Describe the intended architecture |
| Gap analysis | Compare requirements with discovered support |
| Recommended actions | Propose how to close the gaps |
The selected recommendation in the screenshot is titled Introduce a Streaming Data Processing Layer. The gap analysis captures requirements whose support the agents could not find in the current architecture. That wording matters: a retrieval result establishes what the system found, so its evidence must remain inspectable before a proposed change is accepted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From specialist recommendations to independent proposals
A chief architect agent oversees the high-level work. Beneath it are ten staff architect agents specializing in domains such as infrastructure, API, and IAM. Two retrieval agents provide the evidence: one accesses requirements, while the other answers questions about the current architecture and its components.
The recommendation workflow has three sequential stages:
- List generation: Produce candidate recommendation topics.
- Conflict resolution: Have the chief architect inspect the combined list, identify conflicts and redundancies, and prune it.
- Design proposal: Expand each retained topic into a full proposal, including gap analysis and architectural improvements.
Resolving conflicts before detailed drafting prevents the workflow from treating every specialist’s initial suggestion as an independently valid final plan.
During list generation, the chief asks a staff architect for recommendations. That specialist queries the architecture-state and requirements agents in parallel, repeating the exchanges within its available budget before returning a candidate list. Multiple staff architects do this concurrently; Makaremi describes tens of calls happening at the same time. After conflict resolution, the chief returns the surviving topics to the specialists for proposal writing.
Proposal writing uses a deliberate fork in memory. Each staff architect is cloned according to the number of recommendations it must develop. Every clone inherits the prior history, but its new conversation history is separate from the other clones. The common evidence survives; proposal-specific work does not leak into neighboring drafts.
For a concrete illustration, suppose one staff architect has gathered evidence about an API and must draft two proposals: add request throttling and add API monitoring. The JSON below represents the memory boundary after each clone receives its assignment. Both retain the same prior evidence; neither receives the other clone’s new assignment. These are proposal-writing tasks, not executed infrastructure changes.
json
{
"staffHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"clones": [
{
"id": "throttling-proposal",
"inheritedHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"currentHistory": [
"Draft a proposal to add request throttling."
]
},
{
"id": "monitoring-proposal",
"inheritedHistory": [
"Requirement: protect the API from request bursts.",
"Architecture: an API gateway routes to the service."
],
"currentHistory": [
"Draft a proposal to add API monitoring."
]
}
]
}
The implementation must preserve that distinction between inherited context and each clone’s subsequent history, regardless of how the histories are stored.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluating the path to a recommendation
A final recommendation alone does not explain whether a many-agent workflow behaved well. There may be repeated conversations behind it, each introducing an opportunity for an irrelevant retrieval, unsupported inference, or mistaken handoff. Catio closed the improvement loop with human scoring, structured feedback, and revision cycles. At the development stage Makaremi describes, LLM evaluations were useful but did not provide the guidance the team needed to make its intended improvements.
The team built an internal evaluation tool called Eagle Eye. For each case, reviewers can inspect the architecture, extracted requirements, inter-agent conversations, and generated recommendations. They score relevance, feasibility, and clarity, then use those judgments to decide which improvements to prioritize. This puts the recommendation beside the evidence and exchanges that produced it.
LLM confidence is not correctness. Human feedback was especially important while building the system from scratch, and evaluation became part of system design rather than a later addition. For every new AI system, Catio began asking how it would be evaluated as it developed. Depending on the task, that infrastructure could be a human review tool, a monitoring dashboard, or an LLM-based feedback loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When a retriever is asked to schedule a workshop
Inspecting intermediate conversations exposed a distinctive failure. In an early case, the network-security staff architect sent the requirements retriever a request to schedule a workshop, complete with possible dates, as part of trying to solve its architecture task. The agent had moved from seeking requirements to requesting an organizational action from a retrieval role.
The example shows why a plausible conversational next step can still violate the workflow’s purpose. A requirements retriever is there to supply evidence; a workshop request does not perform that function. Makaremi identifies prompts and structured outputs as places to address this kind of hallucination. Reviewing the exchanges makes the failure visible at the point where the agent leaves its intended role, rather than only through whatever recommendation eventually emerges.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Coordinating workflows and task-specific memory
The intended scope is larger than answering an isolated architecture question. Makaremi describes a copilot that can reason across thousands or millions of components and many documents, serving stakeholders from developers to CTOs. Those quantities describe the problem the system aims to address, rather than a demonstrated scale benchmark.
Supporting that scope takes roles, workflows, memory, and structure, with experimentation tied to the available data. Graphs are becoming more important in Catio’s designs. How agents interact, and how much autonomy each receives, remain consequential design choices: they determine which evidence can reach a decision and how the system keeps the reasoning process under control.
After experimenting with several frameworks, Catio is converging on LangGraph for some agent workflows, with a manager above them to coordinate multiple workflows. Flyte handles some of the higher-level management. Alongside that orchestration, the team is using graphs to capture as much memory as possible so each task receives the context it needs. The workflow graph and the graph holding architectural knowledge serve different purposes; naming the orchestration framework does not by itself specify the knowledge representation.
Makaremi closes with the view that this work is only beginning, and that systems built around these mechanisms will become a way for AI to design software. The ambition rests on the whole process: retrieving the relevant context, coordinating specialists, preserving useful memory, and making the resulting proposals open to inspection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current documentation for the named framework's thread checkpoints and cross-thread memory stores.
Workflow orchestration project used for higher-level management. The current README presents Flyte 2 and links to Flyte 1.
Further reading
A contemporaneous account of the chief architect, ten specialists, retrieval agents, cloning and EagleEye inspection system.
Makaremi explains Catio's use of conversation limits, hierarchical coordination and conflict resolution.
Updates since the talk
- Introducing GraphQAArticle
Makaremi's later account of combining language-model reasoning with graph algorithms for architectural questions.
Open-source natural-language graph analysis with NetworkX algorithms, installation instructions and examples.
Read the complete timestamped transcript
- 0:01
Hello, everyone. My name is Iman Makaremi. I'm the co-founding head of AI at Cat.io. Today, I'm talking about, uh, grounded reasoning systems, uh, for cloud architecture and, uh, how we use multi-agent orchestration to build, uh, an AI copilot at Cat.io.
- 0:18
Uh, but why now? Uh, cloud architecture, uh, needs reasoning, uh, not just automation. Uh, we see that cloud systems are, uh, growing in complexity, uh, uh, based on people, uh, both users and also developers, and in terms of tools and constraints and com...
- 0:34
expectations are always on the rise. Uh, simplistic tools, uh, don't scale with the diversity of decisions that need to be made around cloud architecture, and we, uh, need systems that understand, debate, and justify and also plan, uh, in order to solve such problems.
- 0:50
And this is not automation. This is, uh, reasoning.
- 0:54
The architecture stack isn't, uh, just technical, it's also, uh, very cognitive. Uh, architects don't just assemble infrastructure. They are constantly negotiating trade-offs based on, uh, how defined the requirements are or how much they...
- 1:11
time they have or what resources are available for building, uh, the application that they are supposed to deliver. And, uh, they rely on this scattered and implicit, uh, context in order to make this decision.
- 1:25
And capturing this for AI requires understanding how architects think. So, uh, where AI meets architecture, there are three challenges that we, uh, capture at, uh, Cat.io to... at a higher level in order to, uh, uh, solve architecture design problems.
- 1:46
Uh, one is about requirement understanding. So, uh, where are these requirements coming from, in what format, uh, and, uh, what are the important pieces? How, uh, global or, uh, specific they are.
- 2:00
Uh, these are the, all the things that need to be understood as, uh, value requirements. On the other side, it's about architecture identification. Uh, there are many, many different components that could exist in an architecture, uh, but, uh, based on the components and based on where, where they are, they could have very different functions.
- 2:17
In order to understand how an architecture works, knowing different functionalities and, uh, and the functions that different components are playing is very important, too. And, uh, putting all these things together is about architecture, uh, recommendation.
- 2:29
Is that, so we know the requirements, we understand what the architecture state is today, what are the recommendation that, uh, need to be pro, uh, provided in order to, uh, either match the requirements or improve the architecture, uh, to, uh, to, to be, uh, matching, uh, best practices.
- 2:49
Uh, to turn these problems into more specific AI-related challenges, uh, what we're dealing with is, is a mix of semantic and graph-based context. On one side, we have requirements, which are mostly text, textual, uh, documents.
- 3:04
And on the other side, architecture, which is inherently graph data. How we can, uh, make these two different sources of data to come together and, and make the right connections in order to, to higher, uh, level of reasoning is very important.
- 3:20
And through that comes the complex reasoning scenarios. Uh, the, the kind of questions that come in could be very complex, very vague, very broad, and, uh, that needs to be broken down into different parts and, uh, be planned properly in order to, to, uh, uh, get the right answer to, to the, uh, question.
- 3:41
And above all of these things, uh, how we evaluate and, uh, provide feedback to such a, uh, uh, large AI system that has many, uh, different, uh, moving parts.
- 3:53
Uh, so, uh, we want to go through, uh, uh, these challenges and, uh, share some of our learnings, um, uh, what, uh, we have experienced, and, uh, what we have done, and, uh, and we are...
- 4:06
what we are doing in order to, uh, improve to, to the next, uh, level. So the first one is about, about grounding ag-agents in a specific context. So LLM, uh, needs proper context, uh, about architecture to reason effectively.
- 4:21
Uh, translating natural language into meaningful architecture retrieval task isn't straightforward, and also providing it fast makes it even harder. Uh, so we, we tried multiple things here, both on architecture retrieval and on requirement, uh, requirement retrieval part.
- 4:38
Uh, one was that, uh, it, it was about semantic enrichment of ar-architecture data. For each component, we, uh, collected, uh, enough relevant semantic information to enrich, uh, that component in order to make it, uh, more searchable and, uh, uh, findable in a vector search.
- 4:58
Another part was about, uh, graph-enhanced component search. When you're searching for a specific component or a type of component within architecture, uh, through graph search, we also, uh, used all sorts of graph, uh, algorithms in order to make sure that we retrieve the right, uh, pieces of information from an architecture.
- 5:20
Uh, for requirements specifically, uh, we did some early score enrichment of requirement documents for faster retrieval because, uh, requirements, uh, could be about many different topics and, uh, uh,
- 5:36
if we want to do, uh, and understand a large corpus of text properly, probably want to, to understand different concept that exists in that, uh, in, in, in that context.
- 5:48
Uh, we did, uh, something, uh, a little simpler in our first iteration, uh, which was about, uh, figuring out what are the important concepts and requirements for us within a large organization.
- 6:00
And based on that, we scored different dec- documents and, and through that, uh, we made retrieval task, uh, faster. What we learned was that semantic grounding improves reasoning, but doesn't always, uh, work.
- 6:15
Uh, we, we, we saw some really good results of architecture retrieval. However, we also saw, uh, where it falls short and cannot be scaled or gets, uh, very detailed, uh, responses.
- 6:30
Prompt design is critical in soft grounding. We, we cannot always provide, uh, the exact, uh, the, the exact structure and, uh, t-telling the agent to, uh, to exactly what to focus on and, and what to retrieve is very important.
- 6:46
And, uh, the other part is the, that, uh, graph memory supports, uh, continuity and not just accuracy. Uh, it is very important that, uh, we, in our graph searches, we are able to, uh, find different nodes within a graph, but also be able to s-connect these different nodes and figure out why these-- we have these different nodes,
- 7:06
uh, pop up in our searches and, and add more context that could help us to do proper reasoning, uh, in the next steps.
- 7:16
So what we see here, for example, is one of our early designs on, uh, architecture retrieval. On the, on the left side, we have, uh, a storing step. On the, on the right side, you have the query step.
- 7:27
And, uh, on the left step, uh, we're basically breaking c- the architecture down by component and turning the, the JSON data into natural language and then enriching it, uh, with connection data and embedding it and putting it into a vector DB.
- 7:43
And on the other side, uh, when the search time comes, a query comes, uh, we embed that search in the vector, uh, DB and find, uh, the component. Uh, we saw some, uh, good results with this, but, uh, as, uh, as one thing we s- uh, we, we, uh, we found was that, uh, semantic search doesn't work
- 8:03
very well for, uh, uh, this type of graph data, and it has its own limitation. And that's why, uh, from here we went more towards, uh, graph-based, uh, searches.
- 8:17
First of all, enrich this approach with, with some graph, uh, searches, and from here we are moving towards, uh, making this more, uh, based on the, the graph, uh, a knowledge graph.
- 8:29
This another design that we did initially for, uh, understanding requirements, and, uh, uh, part of it is, uh, very, uh, familiar, uh, for most of us. It is about, uh, taking an input document, split, uh, uh, preprocessing it, splitting it, embedding it.
- 8:46
Uh, what, what we did extra here was, uh, uh, coming up with, uh, h- first of all, having the requirement templates, uh, that, uh, had s-specific structure on what requirements are.
- 8:59
Uh, in this way, we were able to, uh, basically structure what we are extracting from the documents, uh, in order to,
- 9:08
uh, uh, as we call it, pod, uh, to, to make sure that the, the extracted information is relevant to what we're gonna use downstream.
- 9:18
This helped us, uh, with fast retrieval. It has, uh, the, its own advantage in terms of, uh, finding, uh, uh, structuring the, the business requirements and finding the, the right pieces, uh, for, uh, agents to, to do the further, uh, uh, recommendation tasks.
- 9:36
Uh, however, we also learned that this has also had some limitation, where as we increase the number of documents, we f- we see that we, we sometimes lose the context, uh, uh, in, uh, in the larger, uh, searches.
- 9:51
So, and here is another part that, uh, finding, uh, the right context within larger corpus becomes very important. And, uh, that's where we see some, uh, like graph analysis could also help on this side.
- 10:06
Another part is about complex reasoning scenarios. [clears throat] Good design involves com- uh, conflicting goals, trade-offs, and debates. That's part of the architecture design world. Uh, it doesn't-- It never comes easy.
- 10:19
Uh, and we also need the agents that can collaborate, argue, and converge on, uh, justified recommendations. Uh, so what we tried was, uh, basically building a MAS orchestr- or orchestration, uh, with, uh, role-specific agents.
- 10:34
This multi-agent system, uh, is able to basically, uh, uh, make multiple agents, uh, work, uh, together at the same time and, uh, give them multiple properties. We'll, we'll cover more of that in, uh, in, in the next slides.
- 10:53
Uh, another thing we tried was, uh, having structured message formats. Initially, uh, uh, that we had to start with, uh,
- 11:01
generating XMLs and parsing SM- XMLs, which wasn't fun, and, uh, I'm glad those days are gone. Um, and, uh, but, uh, having the structured message, uh, definitely helped, uh, with, uh, building better workflows and, and h- uh, having multiple agents to work with each other, uh, and having more of our agents also, uh, work with each other
- 11:24
in a, in a longer chain. Um, custom memory management was a, was an important piece because the way that we wanted to, uh, have, have the agents collaborate with each other, because there were multiple levels of int- uh, interaction between agents.
- 11:39
Uh, we wanted to sure-- ma-make sure that the, uh, conversations are isolated to, first of all, make sure that, uh, we don't waste tokens, and plus, we have, uh, done some, uh, have done some experimentations that increasing the memory showed, uh, uh, increase of hallucination in the, in the results.
- 11:59
So we decided to keep the, uh, memory i-isolated, uh, between agents.
- 12:06
Uh, cloning of agents, uh, for parallel processing for certain tasks was a, was an important piece that we did and, and helped us to, uh, uh Speed up some of our processes, um, and, uh, that required also some memory management in order to like what do we, uh, duplicate for, uh, agents, uh, from a certain point at
- 12:27
least how to do the cloning, um, and, and how we manage the memory at that cloning point. And what we learned was that, uh, structured outputs improves clarity and control.
- 12:38
It's super, super important to, to do structured output, uh, for a bit of programming. Uh, we hear a lot also about how structured output could reduce, uh, the, the, the, the reasoning abilities of the, the model.
- 12:55
Um, it is a real trade-off, but however, uh, the-- we are seeing very good results with the level of structured output that we have.
- 13:07
Uh, MAS let, uh, MAS lets agents resolve trade-offs, uh, dynamically and not just, uh, execute static, uh, plans. Uh, we, we let, uh, we, we give a lot of, uh...
- 13:19
Uh, we, we are working also on different ways of orchestration, uh, for, for agents and, uh, what are they exactly doing each, each step that they're taking, uh, as part of their interaction with other agents.
- 13:32
Uh, but, uh, we are learning that the, the more dynamic we make it, uh, the, the higher creativity they're showing in planning and, uh, reaching the results, and that's, uh, what we are trying to, to optimize.
- 13:48
Uh, another thing is that successful MAS orchestration does require control flow, uh, control flows. Uh, it's not possible to
- 13:58
have multiple agents to just work with each other and, uh, uh,
- 14:04
hope for the best that, that, that the result is gonna come at the end. Uh, so, uh,
- 14:10
the major part of our stack, uh, that is in production and it's working is recommendation that is based on multiple agent systems. Uh, as we se-- uh, there's a, here's a screenshot of, uh, some of the recommendations that we have on some, uh, mock-up data.
- 14:27
Uh, but everything else that, that you see here in terms of the categories, uh, that we have, for example, data architectu-architecture, messaging and queuing, API integration, uh, and so on.
- 14:41
Uh, and for each of these, uh, the multi-agent system comes up, uh, with multiple, uh, recommendations, and each recommendation is broken down into different parts, such as description, target state, gap analysis that shows, for example, what the requirement was, what did, what, uh, the agents couldn't find the architecture based on the requirements, and, uh, what are the
- 15:04
recommendation, recommended actions in terms of filling out, uh, the, the discovered gaps.
- 15:12
Um, the system that we have that, that does this is, uh, um, has multiple agents in it. Uh, one is a chief architect that, uh, basically oversees and coordinates, uh, higher-level tasks.
- 15:23
And then-- and there are ten staff architects, uh, each specialized, uh, in a, in a certain domain, for example, infrastructure, API, IM, and so forth. Um, and then there is, uh, there are two retrievers, uh, requirement retriever that, uh,
- 15:42
basically has access to requirements data and the Architecture retriever that understands what's the current, uh, architecture state and can answer questions about what components are there and so forth.
- 15:55
Uh, the MAS workflow that we have is, uh, is like this. That, uh, there are basically three main tasks that, uh, that happen in a, in a sequence in order to, uh,
- 16:09
generate recommendations at the end of the slope. Uh, one is list generation, where, uh, at the end of it, we have a list of possible recommendations. And then there is confluct, uh, conflict resolution, which is about, uh, uh, having the chief, uh, architect looking at the, uh, generated list of recommendations in order to, um,
- 16:34
in order to figure out if they're in conflict or if there are redundancies and so, uh, uh, and so on. And, uh, basically the... prune the list of reco-recommendations in order to, to go further down the workflow.
- 16:50
And the last step is design proposal, which, uh, is, uh, basically writing a full-on, uh, design proposal based on each recommendation topic in, in terms of, again, uh, what are the gap, uh, and analysis that has been done, uh, what are the proposals in terms of improvement of the architecture and so on.
- 17:11
And the way it works is that, uh, there is a chief architect and, uh, the chief architect requests a list of recommendations from a staff architect, uh, and, uh, and the staff architect, in order to do that, it reaches out to, uh, sends requests to the architecture state agent and, uh, requirements agent.
- 17:32
And, uh, these calls happen in parallel. Uh, they, they happen a number of times based on the budgets that exist. And at, at some point, uh, it all comes back to the staff, and the staff, uh, gets back to the chief with a list of, uh, possible, uh, recommendations.
- 17:51
Um, and this also happens among multiple staff architects. So, uh, this could be tens of calls that happen at the same time. And then after re-- uh, after resolving the conflicts, uh, we go again, uh, back to the staff architect in order to, uh, generate those design proposals.
- 18:11
And again, this flow will happen. Uh, and this time, uh, there is a cloning happening, which is an important part of, uh, this whole flow, is that, um Each staff architect is gonna be cloned, uh, by the number of recommendations that's supposed to generate, and each clone has access to the past history and, uh, generates its own
- 18:32
current history, which is separated with-- from the other clones. And this way, uh, the agent has access to all the existing knowledge in order to make sure that, you know, it h- it can write up the best, uh, design proposal at the end of the workflow.
- 18:50
Um, and, um, the la-the last important thing here is evaluation and feedback. Um, and, uh,
- 19:04
the challenge is, uh, how do we know if a recommendation is good? Uh, we build this, uh, h-- multi-agent system that has many, many agents and, uh, there are rounds and rounds of conversations that, that need to be, uh, monitored.
- 19:22
What's gonna work with this here? And, uh, we had to close the loop with human scoring and structured feedback and revision cycles. Uh, what we learned, especially at the stage that we are, uh, the best, uh,
- 19:39
evaluation is human evaluation. Uh, and we noticed that LLM evaluations, uh, are good, but they do not give us what we want in order to make the improvements that we needed to, to.
- 19:54
Uh, for that purpose, we built, uh, our internal human eval, uh, eval tool, we called it Eagle Eye. And, uh, it basically helps us to look at e-every, uh, specific case, uh, the architecture, the requirements that are extracted, uh, the conversations between the agents, and also the, uh, generated requirement, uh, recommendations.
- 20:17
And in this way, uh, we are able to, to look at them and, uh, basically do, uh, relevance, feasibility, the, and clarity studies and give them scores. Uh, and, uh, and based on that, make some decisions on, in terms of what we want to focus next.
- 20:36
And what we learned is that LLM confidence is not correctness. Um, uh, it can help, uh, but it is not, uh, the, uh, uh, what, what, what can be trusted all the time.
- 20:50
And human, uh, feedback is essential early on when, uh, when, when we're building such systems from, uh, from scratch. And, uh, eval must be baked into system design, not added later.
- 21:03
This is a, this is a, uh, um, design thought that, that, uh, that we came down to that is, that is very helpful, is that as soon as we start thinking about, uh, designing a new AI system, we also start thinking about, okay, what are the evaluations that are gonna be around it?
- 21:22
Uh, and we're able to evaluate whatever we have, um, as we go forward. It could be a human eval tool, it could be a monitoring dashboard, or it could be, uh, LLM-based evaluation, uh, feedback loops.
- 21:40
Um, this, uh, is a screenshot of our, uh, uh, evaluation tool. Uh, we basically can see mult-- uh, various things that, about the interactions in the, the multi-agent system and, uh, go through the conversations and read them and make sure that they make sense or not.
- 22:02
And, uh, it actually helps to do that. For example, this is a case of hallucination, uh, early on. Uh, this is, uh, a staff architect network security that sent a message to requirements retriever and, uh,
- 22:18
asked for, to, uh, for scheduling a workshop and, uh, provided some, uh, dates on, uh, when that workshop could be in order to basically solve whatever they were trying to solve.
- 22:30
Um, handling these kind of cases is very important, uh, in, in, in, uh, in the prompts and also, uh, as part of the, the, uh,
- 22:39
structured, uh, output that, that, uh, each LLM, uh, give. And also it's, it's fun, uh, to, to come across some of these hallucinations, uh, uh, mid building such systems.
- 22:54
Um, so to wrap it up, uh, about reasoning systems, uh, is that it's just, it's, uh, it's not assistance, uh, and there, there is a, uh, slight difference there.
- 23:07
Uh, building a copilot, uh, is not generating answers. Uh, it's about designing a system that can, uh, reason. Uh, we want to build a system that, uh, can, uh, have, uh, a good view of a large, uh, amount of data, especially it's, uh, such as, uh, architecture data, uh, where we're talking about,
- 23:32
uh, thousands, uh, or millions of components and, uh, uh, large number of documents, uh, that need to be in scope, uh, in order to answer various kind of questions with, uh, various types of, uh, stakeholders from, uh, developers, uh, to CTOs, uh, that could be interacting with, uh, with, uh, such a copilot.
- 23:58
Uh, so, uh, and, uh, and, and that takes, uh, roles, that takes, uh, workflows, that takes me-memories and structure in order to build, uh, such a reasoning system. Uh, and, uh, a-and, and, and putting all these things together, uh, requires, uh, a lot of experimentation in terms of, uh, what works based on data that exists.
- 24:21
Uh, we are learning that based on data that we have, uh, there are certain patterns that, uh, that work better than the others. Uh, for example, graphs are becoming more and more important, uh, in our, uh, in our designs.
- 24:35
And, uh, you know, the part of it, uh, is, uh, is what kind of, uh, interactions agents can have with each other and, uh, and, uh, what, what level of, uh, autonomy we want to give to each, uh, agent.
- 24:50
Uh, we are learning that, uh, you know, we've experimented with, uh, with various, uh,
- 24:58
uh, uh, frameworks for building multi-agents, and we are coming down to using, uh, uh, LangGraph, uh, for building, uh, some of the agent, uh, workflows. And again, having a, a manager on top of that in order to manage these la-- various, uh, LangGraph workflows.
- 25:18
Uh, and then also using Flyte, uh, to basically to handle some of these, uh, higher level, uh, management and, uh, and, and, and, and using, uh, uh, graphs to capture, uh, as pa- as, uh, as much of the memory as possible in order to, to make sure that the AI always has, uh, the
- 25:43
right context, uh, as, uh, you know, as, as per task.
- 25:49
And yes, and we are just getting started, and we believe this is, uh, this is how AI will, uh, design software. And, uh, if, uh, there is any questions, please feel free to reach me at, uh, [REDACTED:email_address].
- 26:04
Thank you for your attention.