Grounded Reasoning Systems for Cloud Architecture

Talk transcript

1. 0:01

   Hello, everyone. My name is Iman Makaremi. I'm the co-founding head of AI at Cat.io. Today, I'm talking about, uh, grounded reasoning systems, uh, for cloud architecture and, uh, how we use multi-agent orchestration to build, uh, an AI copilot at Cat.io.

2. 0:18

   Uh, but why now? Uh, cloud architecture, uh, needs reasoning, uh, not just automation. Uh, we see that cloud systems are, uh, growing in complexity, uh, uh, based on people, uh, both users and also developers, and in terms of tools and constraints and com...

3. 0:34

   expectations are always on the rise. Uh, simplistic tools, uh, don't scale with the diversity of decisions that need to be made around cloud architecture, and we, uh, need systems that understand, debate, and justify and also plan, uh, in order to solve such problems.

4. 0:50

   And this is not automation. This is, uh, reasoning.

5. 0:54

   The architecture stack isn't, uh, just technical, it's also, uh, very cognitive. Uh, architects don't just assemble infrastructure. They are constantly negotiating trade-offs based on, uh, how defined the requirements are or how much they...

6. 1:11

   time they have or what resources are available for building, uh, the application that they are supposed to deliver. And, uh, they rely on this scattered and implicit, uh, context in order to make this decision.

7. 1:25

   And capturing this for AI requires understanding how architects think. So, uh, where AI meets architecture, there are three challenges that we, uh, capture at, uh, Cat.io to... at a higher level in order to, uh, uh, solve architecture design problems.

8. 1:46

   Uh, one is about requirement understanding. So, uh, where are these requirements coming from, in what format, uh, and, uh, what are the important pieces? How, uh, global or, uh, specific they are.

9. 2:00

   Uh, these are the, all the things that need to be understood as, uh, value requirements. On the other side, it's about architecture identification. Uh, there are many, many different components that could exist in an architecture, uh, but, uh, based on the components and based on where, where they are, they could have very different functions.

10. 2:17

    In order to understand how an architecture works, knowing different functionalities and, uh, and the functions that different components are playing is very important, too. And, uh, putting all these things together is about architecture, uh, recommendation.

11. 2:29

    Is that, so we know the requirements, we understand what the architecture state is today, what are the recommendation that, uh, need to be pro, uh, provided in order to, uh, either match the requirements or improve the architecture, uh, to, uh, to, to be, uh, matching, uh, best practices.

12. 2:49

    Uh, to turn these problems into more specific AI-related challenges, uh, what we're dealing with is, is a mix of semantic and graph-based context. On one side, we have requirements, which are mostly text, textual, uh, documents.

13. 3:04

    And on the other side, architecture, which is inherently graph data. How we can, uh, make these two different sources of data to come together and, and make the right connections in order to, to higher, uh, level of reasoning is very important.

14. 3:20

    And through that comes the complex reasoning scenarios. Uh, the, the kind of questions that come in could be very complex, very vague, very broad, and, uh, that needs to be broken down into different parts and, uh, be planned properly in order to, to, uh, uh, get the right answer to, to the, uh, question.

15. 3:41

    And above all of these things, uh, how we evaluate and, uh, provide feedback to such a, uh, uh, large AI system that has many, uh, different, uh, moving parts.

16. 3:53

    Uh, so, uh, we want to go through, uh, uh, these challenges and, uh, share some of our learnings, um, uh, what, uh, we have experienced, and, uh, what we have done, and, uh, and we are...

17. 4:06

    what we are doing in order to, uh, improve to, to the next, uh, level. So the first one is about, about grounding ag-agents in a specific context. So LLM, uh, needs proper context, uh, about architecture to reason effectively.

18. 4:21

    Uh, translating natural language into meaningful architecture retrieval task isn't straightforward, and also providing it fast makes it even harder. Uh, so we, we tried multiple things here, both on architecture retrieval and on requirement, uh, requirement retrieval part.

19. 4:38

    Uh, one was that, uh, it, it was about semantic enrichment of ar-architecture data. For each component, we, uh, collected, uh, enough relevant semantic information to enrich, uh, that component in order to make it, uh, more searchable and, uh, uh, findable in a vector search.

20. 4:58

    Another part was about, uh, graph-enhanced component search. When you're searching for a specific component or a type of component within architecture, uh, through graph search, we also, uh, used all sorts of graph, uh, algorithms in order to make sure that we retrieve the right, uh, pieces of information from an architecture.

21. 5:20

    Uh, for requirements specifically, uh, we did some early score enrichment of requirement documents for faster retrieval because, uh, requirements, uh, could be about many different topics and, uh, uh,

22. 5:36

    if we want to do, uh, and understand a large corpus of text properly, probably want to, to understand different concept that exists in that, uh, in, in, in that context.

23. 5:48

    Uh, we did, uh, something, uh, a little simpler in our first iteration, uh, which was about, uh, figuring out what are the important concepts and requirements for us within a large organization.

24. 6:00

    And based on that, we scored different dec- documents and, and through that, uh, we made retrieval task, uh, faster. What we learned was that semantic grounding improves reasoning, but doesn't always, uh, work.

25. 6:15

    Uh, we, we, we saw some really good results of architecture retrieval. However, we also saw, uh, where it falls short and cannot be scaled or gets, uh, very detailed, uh, responses.

26. 6:30

    Prompt design is critical in soft grounding. We, we cannot always provide, uh, the exact, uh, the, the exact structure and, uh, t-telling the agent to, uh, to exactly what to focus on and, and what to retrieve is very important.

27. 6:46

    And, uh, the other part is the, that, uh, graph memory supports, uh, continuity and not just accuracy. Uh, it is very important that, uh, we, in our graph searches, we are able to, uh, find different nodes within a graph, but also be able to s-connect these different nodes and figure out why these-- we have these different nodes,

28. 7:06

    uh, pop up in our searches and, and add more context that could help us to do proper reasoning, uh, in the next steps.

29. 7:16

    So what we see here, for example, is one of our early designs on, uh, architecture retrieval. On the, on the left side, we have, uh, a storing step. On the, on the right side, you have the query step.

30. 7:27

    And, uh, on the left step, uh, we're basically breaking c- the architecture down by component and turning the, the JSON data into natural language and then enriching it, uh, with connection data and embedding it and putting it into a vector DB.

31. 7:43

    And on the other side, uh, when the search time comes, a query comes, uh, we embed that search in the vector, uh, DB and find, uh, the component. Uh, we saw some, uh, good results with this, but, uh, as, uh, as one thing we s- uh, we, we, uh, we found was that, uh, semantic search doesn't work

32. 8:03

    very well for, uh, uh, this type of graph data, and it has its own limitation. And that's why, uh, from here we went more towards, uh, graph-based, uh, searches.

33. 8:17

    First of all, enrich this approach with, with some graph, uh, searches, and from here we are moving towards, uh, making this more, uh, based on the, the graph, uh, a knowledge graph.

34. 8:29

    This another design that we did initially for, uh, understanding requirements, and, uh, uh, part of it is, uh, very, uh, familiar, uh, for most of us. It is about, uh, taking an input document, split, uh, uh, preprocessing it, splitting it, embedding it.

35. 8:46

    Uh, what, what we did extra here was, uh, uh, coming up with, uh, h- first of all, having the requirement templates, uh, that, uh, had s-specific structure on what requirements are.

36. 8:59

    Uh, in this way, we were able to, uh, basically structure what we are extracting from the documents, uh, in order to,

37. 9:08

    uh, uh, as we call it, pod, uh, to, to make sure that the, the extracted information is relevant to what we're gonna use downstream.

38. 9:18

    This helped us, uh, with fast retrieval. It has, uh, the, its own advantage in terms of, uh, finding, uh, uh, structuring the, the business requirements and finding the, the right pieces, uh, for, uh, agents to, to do the further, uh, uh, recommendation tasks.

39. 9:36

    Uh, however, we also learned that this has also had some limitation, where as we increase the number of documents, we f- we see that we, we sometimes lose the context, uh, uh, in, uh, in the larger, uh, searches.

40. 9:51

    So, and here is another part that, uh, finding, uh, the right context within larger corpus becomes very important. And, uh, that's where we see some, uh, like graph analysis could also help on this side.

41. 10:06

    Another part is about complex reasoning scenarios. [clears throat] Good design involves com- uh, conflicting goals, trade-offs, and debates. That's part of the architecture design world. Uh, it doesn't-- It never comes easy.

42. 10:19

    Uh, and we also need the agents that can collaborate, argue, and converge on, uh, justified recommendations. Uh, so what we tried was, uh, basically building a MAS orchestr- or orchestration, uh, with, uh, role-specific agents.

43. 10:34

    This multi-agent system, uh, is able to basically, uh, uh, make multiple agents, uh, work, uh, together at the same time and, uh, give them multiple properties. We'll, we'll cover more of that in, uh, in, in the next slides.

44. 10:53

    Uh, another thing we tried was, uh, having structured message formats. Initially, uh, uh, that we had to start with, uh,

45. 11:01

    generating XMLs and parsing SM- XMLs, which wasn't fun, and, uh, I'm glad those days are gone. Um, and, uh, but, uh, having the structured message, uh, definitely helped, uh, with, uh, building better workflows and, and h- uh, having multiple agents to work with each other, uh, and having more of our agents also, uh, work with each other

46. 11:24

    in a, in a longer chain. Um, custom memory management was a, was an important piece because the way that we wanted to, uh, have, have the agents collaborate with each other, because there were multiple levels of int- uh, interaction between agents.

47. 11:39

    Uh, we wanted to sure-- ma-make sure that the, uh, conversations are isolated to, first of all, make sure that, uh, we don't waste tokens, and plus, we have, uh, done some, uh, have done some experimentations that increasing the memory showed, uh, uh, increase of hallucination in the, in the results.

48. 11:59

    So we decided to keep the, uh, memory i-isolated, uh, between agents.

49. 12:06

    Uh, cloning of agents, uh, for parallel processing for certain tasks was a, was an important piece that we did and, and helped us to, uh, uh Speed up some of our processes, um, and, uh, that required also some memory management in order to like what do we, uh, duplicate for, uh, agents, uh, from a certain point at

50. 12:27

    least how to do the cloning, um, and, and how we manage the memory at that cloning point. And what we learned was that, uh, structured outputs improves clarity and control.

51. 12:38

    It's super, super important to, to do structured output, uh, for a bit of programming. Uh, we hear a lot also about how structured output could reduce, uh, the, the, the, the reasoning abilities of the, the model.

52. 12:55

    Um, it is a real trade-off, but however, uh, the-- we are seeing very good results with the level of structured output that we have.

53. 13:07

    Uh, MAS let, uh, MAS lets agents resolve trade-offs, uh, dynamically and not just, uh, execute static, uh, plans. Uh, we, we let, uh, we, we give a lot of, uh...

54. 13:19

    Uh, we, we are working also on different ways of orchestration, uh, for, for agents and, uh, what are they exactly doing each, each step that they're taking, uh, as part of their interaction with other agents.

55. 13:32

    Uh, but, uh, we are learning that the, the more dynamic we make it, uh, the, the higher creativity they're showing in planning and, uh, reaching the results, and that's, uh, what we are trying to, to optimize.

56. 13:48

    Uh, another thing is that successful MAS orchestration does require control flow, uh, control flows. Uh, it's not possible to

57. 13:58

    have multiple agents to just work with each other and, uh, uh,

58. 14:04

    hope for the best that, that, that the result is gonna come at the end. Uh, so, uh,

59. 14:10

    the major part of our stack, uh, that is in production and it's working is recommendation that is based on multiple agent systems. Uh, as we se-- uh, there's a, here's a screenshot of, uh, some of the recommendations that we have on some, uh, mock-up data.

60. 14:27

    Uh, but everything else that, that you see here in terms of the categories, uh, that we have, for example, data architectu-architecture, messaging and queuing, API integration, uh, and so on.

61. 14:41

    Uh, and for each of these, uh, the multi-agent system comes up, uh, with multiple, uh, recommendations, and each recommendation is broken down into different parts, such as description, target state, gap analysis that shows, for example, what the requirement was, what did, what, uh, the agents couldn't find the architecture based on the requirements, and, uh, what are the

62. 15:04

    recommendation, recommended actions in terms of filling out, uh, the, the discovered gaps.

63. 15:12

    Um, the system that we have that, that does this is, uh, um, has multiple agents in it. Uh, one is a chief architect that, uh, basically oversees and coordinates, uh, higher-level tasks.

64. 15:23

    And then-- and there are ten staff architects, uh, each specialized, uh, in a, in a certain domain, for example, infrastructure, API, IM, and so forth. Um, and then there is, uh, there are two retrievers, uh, requirement retriever that, uh,

65. 15:42

    basically has access to requirements data and the Architecture retriever that understands what's the current, uh, architecture state and can answer questions about what components are there and so forth.

66. 15:55

    Uh, the MAS workflow that we have is, uh, is like this. That, uh, there are basically three main tasks that, uh, that happen in a, in a sequence in order to, uh,

67. 16:09

    generate recommendations at the end of the slope. Uh, one is list generation, where, uh, at the end of it, we have a list of possible recommendations. And then there is confluct, uh, conflict resolution, which is about, uh, uh, having the chief, uh, architect looking at the, uh, generated list of recommendations in order to, um,

68. 16:34

    in order to figure out if they're in conflict or if there are redundancies and so, uh, uh, and so on. And, uh, basically the... prune the list of reco-recommendations in order to, to go further down the workflow.

69. 16:50

    And the last step is design proposal, which, uh, is, uh, basically writing a full-on, uh, design proposal based on each recommendation topic in, in terms of, again, uh, what are the gap, uh, and analysis that has been done, uh, what are the proposals in terms of improvement of the architecture and so on.

70. 17:11

    And the way it works is that, uh, there is a chief architect and, uh, the chief architect requests a list of recommendations from a staff architect, uh, and, uh, and the staff architect, in order to do that, it reaches out to, uh, sends requests to the architecture state agent and, uh, requirements agent.

71. 17:32

    And, uh, these calls happen in parallel. Uh, they, they happen a number of times based on the budgets that exist. And at, at some point, uh, it all comes back to the staff, and the staff, uh, gets back to the chief with a list of, uh, possible, uh, recommendations.

72. 17:51

    Um, and this also happens among multiple staff architects. So, uh, this could be tens of calls that happen at the same time. And then after re-- uh, after resolving the conflicts, uh, we go again, uh, back to the staff architect in order to, uh, generate those design proposals.

73. 18:11

    And again, this flow will happen. Uh, and this time, uh, there is a cloning happening, which is an important part of, uh, this whole flow, is that, um Each staff architect is gonna be cloned, uh, by the number of recommendations that's supposed to generate, and each clone has access to the past history and, uh, generates its own

74. 18:32

    current history, which is separated with-- from the other clones. And this way, uh, the agent has access to all the existing knowledge in order to make sure that, you know, it h- it can write up the best, uh, design proposal at the end of the workflow.

75. 18:50

    Um, and, um, the la-the last important thing here is evaluation and feedback. Um, and, uh,

76. 19:04

    the challenge is, uh, how do we know if a recommendation is good? Uh, we build this, uh, h-- multi-agent system that has many, many agents and, uh, there are rounds and rounds of conversations that, that need to be, uh, monitored.

77. 19:22

    What's gonna work with this here? And, uh, we had to close the loop with human scoring and structured feedback and revision cycles. Uh, what we learned, especially at the stage that we are, uh, the best, uh,

78. 19:39

    evaluation is human evaluation. Uh, and we noticed that LLM evaluations, uh, are good, but they do not give us what we want in order to make the improvements that we needed to, to.

79. 19:54

    Uh, for that purpose, we built, uh, our internal human eval, uh, eval tool, we called it Eagle Eye. And, uh, it basically helps us to look at e-every, uh, specific case, uh, the architecture, the requirements that are extracted, uh, the conversations between the agents, and also the, uh, generated requirement, uh, recommendations.

80. 20:17

    And in this way, uh, we are able to, to look at them and, uh, basically do, uh, relevance, feasibility, the, and clarity studies and give them scores. Uh, and, uh, and based on that, make some decisions on, in terms of what we want to focus next.

81. 20:36

    And what we learned is that LLM confidence is not correctness. Um, uh, it can help, uh, but it is not, uh, the, uh, uh, what, what, what can be trusted all the time.

82. 20:50

    And human, uh, feedback is essential early on when, uh, when, when we're building such systems from, uh, from scratch. And, uh, eval must be baked into system design, not added later.

83. 21:03

    This is a, this is a, uh, um, design thought that, that, uh, that we came down to that is, that is very helpful, is that as soon as we start thinking about, uh, designing a new AI system, we also start thinking about, okay, what are the evaluations that are gonna be around it?

84. 21:22

    Uh, and we're able to evaluate whatever we have, um, as we go forward. It could be a human eval tool, it could be a monitoring dashboard, or it could be, uh, LLM-based evaluation, uh, feedback loops.

85. 21:40

    Um, this, uh, is a screenshot of our, uh, uh, evaluation tool. Uh, we basically can see mult-- uh, various things that, about the interactions in the, the multi-agent system and, uh, go through the conversations and read them and make sure that they make sense or not.

86. 22:02

    And, uh, it actually helps to do that. For example, this is a case of hallucination, uh, early on. Uh, this is, uh, a staff architect network security that sent a message to requirements retriever and, uh,

87. 22:18

    asked for, to, uh, for scheduling a workshop and, uh, provided some, uh, dates on, uh, when that workshop could be in order to basically solve whatever they were trying to solve.

88. 22:30

    Um, handling these kind of cases is very important, uh, in, in, in, uh, in the prompts and also, uh, as part of the, the, uh,

89. 22:39

    structured, uh, output that, that, uh, each LLM, uh, give. And also it's, it's fun, uh, to, to come across some of these hallucinations, uh, uh, mid building such systems.

90. 22:54

    Um, so to wrap it up, uh, about reasoning systems, uh, is that it's just, it's, uh, it's not assistance, uh, and there, there is a, uh, slight difference there.

91. 23:07

    Uh, building a copilot, uh, is not generating answers. Uh, it's about designing a system that can, uh, reason. Uh, we want to build a system that, uh, can, uh, have, uh, a good view of a large, uh, amount of data, especially it's, uh, such as, uh, architecture data, uh, where we're talking about,

92. 23:32

    uh, thousands, uh, or millions of components and, uh, uh, large number of documents, uh, that need to be in scope, uh, in order to answer various kind of questions with, uh, various types of, uh, stakeholders from, uh, developers, uh, to CTOs, uh, that could be interacting with, uh, with, uh, such a copilot.

93. 23:58

    Uh, so, uh, and, uh, and, and that takes, uh, roles, that takes, uh, workflows, that takes me-memories and structure in order to build, uh, such a reasoning system. Uh, and, uh, a-and, and, and putting all these things together, uh, requires, uh, a lot of experimentation in terms of, uh, what works based on data that exists.

94. 24:21

    Uh, we are learning that based on data that we have, uh, there are certain patterns that, uh, that work better than the others. Uh, for example, graphs are becoming more and more important, uh, in our, uh, in our designs.

95. 24:35

    And, uh, you know, the part of it, uh, is, uh, is what kind of, uh, interactions agents can have with each other and, uh, and, uh, what, what level of, uh, autonomy we want to give to each, uh, agent.

96. 24:50

    Uh, we are learning that, uh, you know, we've experimented with, uh, with various, uh,

97. 24:58

    uh, uh, frameworks for building multi-agents, and we are coming down to using, uh, uh, LangGraph, uh, for building, uh, some of the agent, uh, workflows. And again, having a, a manager on top of that in order to manage these la-- various, uh, LangGraph workflows.

98. 25:18

    Uh, and then also using Flyte, uh, to basically to handle some of these, uh, higher level, uh, management and, uh, and, and, and, and using, uh, uh, graphs to capture, uh, as pa- as, uh, as much of the memory as possible in order to, to make sure that the AI always has, uh, the

99. 25:43

    right context, uh, as, uh, you know, as, as per task.

100. 25:49

     And yes, and we are just getting started, and we believe this is, uh, this is how AI will, uh, design software. And, uh, if, uh, there is any questions, please feel free to reach me at, uh, [REDACTED:email_address].

101. 26:04

     Thank you for your attention.