The Production AI Playbook: Deploying Agents at Enterprise Scale

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:15

   Um, thank you for joining my session. [clapping] Thank you, mate. Yes. Uh, I'm Sandi. Uh, I'm a Technical Lead, uh, for Data and AI at Databricks. Um, prior to working in Databricks, I worked in Amazon Web Services, uh, for five years as a Principal Architect for Data and AI.

3. 0:35

   Uh, in the past few years, I've worked extensively, uh, building and scaling data and AI platforms using distributed systems and technology. And in the past couple of years, specifically, I've been working with customers trying to figure out what we do with this new AI technology.

4. 0:51

   Uh, when I say new AI, AI has been here for a long time, but we all started experimenting quite exponentially, uh, in, in the past couple of years, right?

5. 1:01

   And I have learnt a great deal of lessons on-- from building demos and how to take those demos to production, working with different customers in, uh, B2B software industries and then, uh, regulated industries like financial, uh, services.

6. 1:18

   So in this session, I want to share a playbook, a framework that I put together from lessons that I have learned working in the trenches, uh, that you can take and apply on when you think about how to put your AI systems into production.

7. 1:36

   And I think this session is nicely placed in the afternoon because what you can do now is, in this framework, you can fit the different, um, knowledge, the knowledge that you have gathered attending these different sessions throughout the day and see where they fit in each of these, you know, elements in the framework.

8. 1:53

   So when I started two years ago, this is the pattern I noticed in every customer conversation, right? So everyone wanted to do something with AI. Uh, there was immense pressure from the top to do something, to build a demo, and every conversation started with, "Let's choose the model," right?

9. 2:09

   And it was nobody's fault because the market was like that. We were talking about models. The models were new technology for us, right? And every conversation started, "Shall we use GPT?

10. 2:18

    Shall we use Claude?" You know, there was huge debate with, with, uh, within organizations. Then you would choose a model. You'll build some features, offsets of-- over what features to build for that application.

11. 2:30

    Uh, you would build that in a controlled environment, so predictable datasets, you know, um, limited scenarios, and then it looked great as a demo. And then leadership would get happy.

12. 2:42

    They would sign it off, and they'll put it into an environment, uh, in, in a production environment. Then after a few weeks, people would start asking questions that, "What the hell is AI doing?"

13. 2:55

    Right? "Why is it not answering the questions the way we expected it to answer when we were doing the demos?"

14. 3:01

    Uh, it would result in not only less-- uh, you know, no realization in return on investment, but also loss of money and effort in building these demos that can never scale to production.

15. 3:15

    Throughout these, uh, meetings, I gathered three insights that connect to everything that you, you, we are talking about when thinking about taking, uh, AI to production. The first one is the observability gap, right?

16. 3:28

    When we use AI and put it into production, if we can't see what it is actually doing, if we can't trace every decision that it's making, it's no use in production.

17. 3:39

    Second is the evaluation ga-gap. A lot of these conversations that we were doing, we were not actually thinking about what is, what is that one thing that we are measuring.

18. 3:48

    Yes, we talk about accuracy, we talk about latency, we talk about groundedness, but we were not defining what is that exact thing like that matters to the business, and how can we build a system that can continuously measure that, whether it's improving, whether it's not improving.

19. 4:06

    Like, what, what is that system that we need to build? And that was that evaluation gap that I, I noticed. And the third is the governance gap. Like, we were not actually thinking, what happens when AI fails in production?

20. 4:16

    Who's accountable? Who do I go to when something happens at three AM in the morning, right? Who needs to own the data assets that feed some, uh, AI responses?

21. 4:27

    What happens if AI, you know, uh, talks, um, uh, um, nonsense to a customer, right? What, what happens, right? So there is no accountability, no governance around it. And these three insights led me to build a framework on how I think AI should be taken to production, and this has been implemented across multiple customer organizations.

22. 4:52

    Uh, and I think this is something that you can pick up from here. These are the five pillars, and these are absolutely what you need to think about even before starting a project, right?

23. 5:01

    Then you start-- build them gradually, preferably in sequence. But in real life, I know that this sequence don't work, but these are the pillars that you have to know about and you have to think about when start building.

24. 5:13

    First one is evaluation. Before touching any code, before discussing about any models, any features, you have to think about when we build this system, how do we measure? What does success look like, and what is that system that will help us continuously measure what success looks like for us?

25. 5:31

    Second is how do we trace each and every decision that AI makes? It's not only important for the performance of the AI system, it is also important for the regulators.

26. 5:42

    In Europe or in a lot of, uh, companies, especially in regulated industry, you cannot even onboard AI into production without having tracing and observability in place. So this is a must-have.

27. 5:54

    The third is the data foundation, right? Uh, I, I, I, I think of data foundation in, in two ways. One is the question data, so that is basically

28. 6:04

    The data needed for the AI to answer questions that users ask to it. So it could be your pre-training data or post, post-training data, data that you use APIs to hook onto and, and get to the answer that the user needs.

29. 6:17

    The other one is the tracking data. Related to the tracing data in observability, but when you think from the data foundation and, and, uh, data strategy perspective, this needs to be handled in this pillar because you need a whole data strategy now with tracing data, especially when you run hundreds of agents in your organization.

30. 6:36

    Fourth is orchestration. One agent would work pretty well. You don't need to think about orchestration. But when you onboard five agents, the, the complexity increases exponentially, right? You will have multiple coordination patterns between these agents.

31. 6:51

    They will need to talk to each other in multiple different ways. They will need to each wait for each other's responses. There's a lot of complexity that comes in, and that's where orchestration patterns and thinking about how you will orchestrate your agents in a particular system becomes really important.

32. 7:06

    Fifth is governance. This is where, uh, you think about what happens when something fails. Who's accountable? How do we govern data? How do we secure it? How do we secure our systems?

33. 7:16

    How-- what do we make su-- How do we make sure that no one injects into our agent and leads, uh, to, you know, misbehavior, right? Or loss of reputation.

34. 7:27

    So in the rest of the session, I will dive a bit deeper into each of these pillars and tell you how, how you can think about when you start working with them, right?

35. 7:34

    The first one is evaluation. Evaluation is basically the specification for your AI system. You define success. As I mentioned, it's not like talking about accuracy. You have to define it with numbers, like what accuracy is, is, is good for your business use case, right?

36. 7:51

    Define it in numbers. Uh, define what kind of, you know, false positives you can handle. What should be the deflection? So this is, this is an example from a, a retail chatbot, right?

37. 8:01

    A banking chatbot, where when you implement a chatbot with an AI agent, one of the main goals is to deflect simple queries, um, to the agent so that a human agent don't need to, uh, deal with them, right?

38. 8:15

    And so you need to, uh, you need to track those queries and track those numbers and put that system in place. Second is building those test, test cases, like the evaluation dataset.

39. 8:25

    You have heard about golden datasets in evaluation. Talk with the domain experts and find what is actually happening in real life on the ground. Like what answer would a support-- human support agent un-- give to a customer on a particular question?

40. 8:39

    Collect those information. What happens in gray areas, in edge cases? Like what happens when a human sees a customer asking a confusing question, right? Collect those into a dataset and then automate your AI testing, right?

41. 8:53

    So you put a question to AI, it answers, take that answer, compare against the test set, and automate this whole pipeline so that when you put AI in production, that pipeline can actually take live responses and evaluate against the test dataset that you're building, and then give you the result, uh, in terms of how AI is performing

42. 9:13

    against those numbers and the goals that you have defined.

43. 9:17

    When we talk about evaluation, there are three main layers that I see, uh, appear, uh, across organization, and this is an architectural decision that you need to make when you build these evaluation systems, right?

44. 9:29

    The first layer is deterministic. These are the easy stuff, like, you know, checking formats, you know, checking, uh, email formats, phone formats, the regular expression things that we have already been doing with our coding systems, right?

45. 9:40

    The s-- uh, the, the other, other is like, you know, you could use, uh, classic ML models for name entity rec-recognition to... for intent classification, for understanding what is a first name, last name, PII detection, et cetera.

46. 9:54

    So th-these, these are easy stuff, cheap stuff. You should get them out of the way. We have already been doing this for years. The second layer is the non-deterministic semantic stuff, right?

47. 10:03

    This is where groundedness comes in. This is where we implement technologies like LLM-as-judges. We all know what LLM-as-judges are, right?

48. 10:11

    Everyone? Okay, I see a lot of nods. So, um, again, this, this is a pretty simple version of how a, uh, how a prompt would look for an LLM-as-a-judge.

49. 10:23

    Um, so with LLM-as-a-judge, you, you w-- you use a separate LLM from the primary LLM to judge the response of the primary model. And when you do that, you tell the secondary, the judge model, on how it should, uh, judge the primary model's output.

50. 10:40

    So it could be around safety, groundedness, you know, relevance to the answer, et cetera, et cetera, right? Again, that can feed from a lot of, uh, the evaluation dataset that you have created, right?

51. 10:52

    To look at what are the expected answers, and then it can check against that. This is a sample prompt on how these things work, but I'm sure you have attended some of these sessions where you have seen vendors doing this automatically at scale.

52. 11:04

    Uh, for example, in Databricks, so in MLflow, you will find automatic LLM-as-judge, uh, where you can create these custom LLM-as-judges that run automatically on traces.

53. 11:15

    That's your second layer. The third layer is behavioral. Like this is where, uh, you think about a tool calls. Like is-- are agents calling the right tool? Are they getting into loops?

54. 11:26

    So for example, um, you know, the first layer, you, you can have a user ask a question, "What is my account balance?" And you could go and check that, okay, this-- there is no deterministic problem with it.

55. 11:37

    The seman-- the agent answered right, that, "Okay, your account balance is this many dollars." And that was right. And you can say, "This is, this is right." But when you go into the behavioral checks, you will see that the agent, uh, actually made three calls to the database to find that answer, right?

56. 11:53

    And that is because it was re-- making duplicate calls for whatever reason. Calls failed, you know, calls did not work. It went and retried and stuff like that. Now, three API calls in demo environment is fine.

57. 12:04

    But in production, when you get thousands of queries from users every day and there's like duplication in API calls, that's an expensive operation. And that's where you need to think about behavioral evaluation.

58. 12:14

    And this layer is very, very important. I see a lot of organizations or a lot of teams miss them when, when talking about this.

59. 12:25

    The second layer is observability, right? Uh, and in this pillar, what we are talking about tracing, right? So you collect all the decisions that an agent is making. So I want to explain this with a scenario here, right?

60. 12:38

    And this is a scenario from, uh, an act-actual project I worked on with a banking-- uh, retail, retail banking chatbot. Now, obviously, if you have seen tracing data, it's not as beautiful [chuckles] as this slide, right?

61. 12:48

    So I've simplified it and made it beautiful for this slide. But what this slide says is basically a user comes in and says, uh, "You know, I have been charged an overdraft fee.

62. 12:56

    Can you waive it for me?" Because the user thinks that-- the customer thinks that that is not legitimate. So the agent does an intent classification. And you all-- You know about this because you have enabled observability, you're capturing traces, and you're actually seeing what the agent is doing, right?

63. 13:11

    What AI is doing. Intent classification. It has done this. It took this many seconds. This was confidence score. Then it goes and connects to the customer's account, maybe in a database, a customer database.

64. 13:21

    It call-calls an API, connects to the customer database, gets the account details. It retrieves policy documents. It checks from a RAG vector database, um, what is, uh, what is the policy around overdraft, right?

65. 13:34

    Is what the customer claiming is legitimate. So it checks for po-policy documents. Then it goes and does a reasoning on what should be, uh, you know, responded to the customer, and then it does fi-final guardrail checks and responds to the customer.

66. 13:49

    Now, if you did not set up a system that helps you look-- visualize all of these traces, when the customer comes to you and raises a dispute, you have no way to check what the AI did,

67. 14:03

    right? You have nowhere to go, and you end up saying that, "I don't have, have any idea. Let's, let's give the customer a discount or something and then make them happy."

68. 14:11

    So this is why you need this, and this is why regulators are, are, are basically mandating, because otherwise there is no production system if you cannot do this kind of stuff.

69. 14:22

    So this is where, um, you know, you, you, you detect this-- the example that I gave around duplicate API calls. This is where you start detecting this stuff. So when you, when you enable these traces, you can actually go and see duplicate calls and then take relevant actions based on that.

70. 14:37

    Not only that, you can actually do that in online monitoring. So when it's happening in production, in on-- uh, you can set up online monitoring, and at that point, if it is doing duplicate calls, you can apply fallback strategies.

71. 14:49

    Or even if it is doing a call that is failing, you can actually go and apply a strategy where it will say, "Okay, go and retry for three times," not more than three times.

72. 14:57

    If it, if it is more than three times, then report somewhere or pass it to a human to take some action.

73. 15:05

    The third pillar is the most important pillar, in my opinion, is the data, da-data, data foundation, right? Uh, in my typical project, projects, I spend sixty percent of my time, uh, and I see, I see a lot of organizations spending a lot of time here.

74. 15:20

    Because no one expected agents to come suddenly in the market and start querying data. Data was always built for humans, and humans are always forgiving. You find the wrong data in a report, you just go and ask someone to correct it.

75. 15:34

    Agents don't forgive you, right? Agents will go find it wrong. They'll give you the wrong answer confidently, right? And you wouldn't know what's happening. And this is why data quality, setting the right data strategy, has become so important for enterprises now.

76. 15:50

    I divide it into two sections. One is the question data, as I was explaining, like, data needed for actually serving the AI's, uh, outcome. And the other one is the tracking data.

77. 16:01

    This is the observability data, the tracing data I was talking about earlier. You need a proper plan on how you collect this tracing data and how you serve it to auditors, to regulators, to do online monitoring, to run LLM-as-judges on the tracing and everything else, right?

78. 16:18

    So there-- it needs a proper strategy on how you structure the schema and everything on the tracing data.

79. 16:24

    Um, on Databricks, um, we create a robust data foundation for our customers using, uh, some of the technologies that we provide. If you don't know Databricks, Databricks has been built on some open source technologies like Apache Spark, MLflow, and Delta Lake.

80. 16:44

    Uh, we provide a bunch of capabilities on top of it. So the blue layer at the bottom is basically your cloud storage. Databricks works on the three major clouds, Google, AWS, Azure. [intercom beeping]

81. 16:57

    Okay. I thought it was for me. [chuckles] So, so once you, once you store raw data on your cloud storage, uh, the data is then, um, um... We, we, we bring in a, a layer called the Delta Lake layer, which, uh, which basically brings in database-like properties on top of your raw data.

82. 17:16

    So you have got images, text files, video files or whatever. We, we help you create this, um, you know, uh, table-like structure on top of it using manifest files, right?

83. 17:27

    And, and we help you to, uh, incrementally load data, do all of those, um, data management tasks in a structured way. On top of that, we bring in Unity Catalog, which is a data catalog.

84. 17:38

    Uh, with Unity Catalog, you can centrally apply permissions on top of the data. You can, um, you can, uh, share the data using, uh, data sharing. But also, uh, what happens with Unity Catalog is, uh, you, you can enable discovery and, um, you know, um, ownership, metadata tagging capabilities at the catalog level.

85. 18:00

    What that means is when you apply table or description-- column description, uh, tag columns or, like, PII columns with metadata, it becomes really easy for AI to then get that context when it queries these tables on top of Unity Catalog.

86. 18:15

    So everything is governed at one layer through Unity Catalog, and on, on top of that, we bring in different, uh, applications. So whether it's AI through Mosaic AI, so to build LLM, tune LLM, or even build AI applications.

87. 18:28

    We bring in, uh, data warehousing capabilities, BI capabilities, and, um, uh, and some of the other text-to-SQL ca-capabilities. We have got Genie that, uh, helps you write natural language to do SQL querying, et cetera.

88. 18:42

    And one application of that in the observability and tracking, tracking data, as I was showing, is, is this. So basically, think about when I was talking about the tracking data strategy.

89. 18:52

    Organizations, especially enterprises, will not be running AI in just one framework. They'll be using different frameworks, QAI, LangChain, et cetera, et cetera. They'll be using different cloud platforms. And once they do that, you need a centralized layer of collecting that tracing data so that you can serve sev-several use cases on the right-hand side.

90. 19:13

    So whether it's for operational dashboarding, for first-line s-support, uh, a, a lot of these, uh, first line, um, first line of defense teams need health monitoring, uh, sort of dashboards, right?

91. 19:25

    These teams can also write SQL using Databricks Genie to do text-to-SQL, but they can also build Databricks apps using coding agents, uh, to create common workspaces or custom, uh, UIs that customers might need for different, uh, different use cases.

92. 19:42

    And then we've got Agent Bricks and A-MLflow that serves you, uh, LLM-- out, out of the box LLM-as-judges and, uh, proactively monitored drift. The idea is no, uh, no, no matter where your AI runs, you can create this kind of strategy, bringing in data in one common place and serving, uh, different teams from one shared location.

93. 20:05

    The fourth pillar is multi-agent orchestration patterns. As I said, one agent is good. Multiple agents increases complexity. That's where you start thinking about, okay, what pattern is good for my use case?

94. 20:17

    The first one I describe here is the orchestrator-worker pattern, where you have one orchestrator which orchestrates all the work, which controls all the work from a centralized plane, and then distributes this work to different agents based on their specialized skills.

95. 20:32

    And then every request goes through the orchestrator, so you have got central control. If something goes wrong, you can go to the orchestrator logs and look into them and see what has happened, right?

96. 20:43

    So that is the orchestration data, uh, pattern. There is this choreography pattern where each agent is independent. They're autonomous. They don't depend on an orchestrator. All of them talk to a message bus, and they listen to the events that they are interested in, right?

97. 20:59

    So think about agents that are independent of each other, right? They can run parallelly, so they are not sequential. Like, one agent is not dependent on another. So they run parallelly.

98. 21:07

    They listen to the message bus for the inter-- uh, for the events that they're interested in. Maybe it's a trigger for, let's say, a mortgage application, and it says, uh, you know, uh, the mortgage application agent, uh, one of the agent, uh, looks customer details, right?

99. 21:22

    The other agent looks at approval details and everything else, right? They can work in parallel, and the advantage it brings you is the latency is reduced because they are not dependent on an orchestrator and sending messages back and forth, right?

100. 21:35

     So this is the choreography pattern. And the third one is human-in-the-loop, which is where when an agent crosses a threshold or serves below threshold, a confidence threshold, then a human is called in the workflow to look into the pattern-- uh, so look into the-- look into what the agent has done and then take action based on that.

101. 21:56

     I have done a deep dive video on multi-agent orchestration pattern, uh, f-for the online track of this conference. Uh, it's already on YouTube, so you can look into it.

102. 22:06

     I talk about the real implications of when you think about multi-agent patterns. One is, uh, state management. The other one is fault tolerance. Like, what happens when things fail?

103. 22:17

     Like, how do you manage them? I talk about different patterns, and then I talk about how, how you think about scaling them in large scale on enterprises.

104. 22:25

     Pillar five is governance, right? Now, here, I'm not talking about data governance at all. That's given. We need that, right? From AI perspective, what, what, what are we thinking about?

105. 22:34

     Regulatory, right? Audit trails. Have we got the trail of every action, every user connection, every request, everything that happens in the system? Are we capturing everything? Are we doing pre-validation of personal information?

106. 22:49

     Are we using named entity recognition? The, the easy stuff, the regex and all of those things, right? In our example, the work that I was doing with the customer that I mentioned, we already detected forty-seven PII breaches during the testing phase by applying this layer.

107. 23:03

     So that's, that's really important. Fourth is, um, prompt versioning. You have to treat prompt versioning as change management in enterprise-grade solution. It cannot be just a change to a prompt and commit to Git.

108. 23:16

     It has to be go-- It, it has to go through proper change management processes as you do with code. So basically treating prompt as code. Third is model change management.

109. 23:25

     So as models change, the model providers upgrade these models. You have to have a system to understand whether that upgraded model will be good for your use case, for your data, right?

110. 23:36

     A model provider will put, will put a, uh, evaluation benchmarks on three, uh, benchmark, uh, boards, but those are not really useful when you put them in your context, in your enterprise.

111. 23:47

     So that's where these evaluation datasets come in handy, where you try these different models on this evaluation dataset and try to understand which one performs better. And that management needs to be done because from a risk perspective, you cannot really rely on one single model.

112. 24:01

     You have to have the flexibility to switch to different models and also test them on your own data. That management needs to be done.

113. 24:12

     Uh, in Databricks, uh, we have taken all of these- These points, these pillars that I have been talking about into agent bricks. We are building agent bricks to make, uh, all of these operations out of the box for you, uh, so that it's easy to implement production-grade AI applications on, uh, on, uh, in, in your enterprises.

114. 24:36

     So I wanted to quickly touch upon a case study just to give you, uh, uh, a flavor of how these things go, right? So, uh, when I was working with this client, um, they were a retail banking-- they were building a retail banking chatbot, you know, one and a half, eighteen months ago.

115. 24:52

     Uh, their, their problem, uh, the pr- the problem they wanted to solve is they had got around twenty thousand odd calls per month from customers on their chatbot. They wanted to deflect...

116. 25:03

     They, they, they saw that there were, like, sixty percent of them were simple queries. What is my account balance? You know, uh, what do I do with my overdraft?

117. 25:09

     And all of those stuff, like, that, that can be answered simply. So they wanted to-- the reliance on human agents for those answers. So they identified those queries, and they wanted to automate them, right?

118. 25:22

     They spent around eighty-five K in six months doing a POC, which did not succeed. When we got involved, uh, we found those insights that I was talking, like, no one knew why things were failing when it was in production, uh, when, when, when, when it was, uh, uh, in, in production.

119. 25:35

     No one could actually, uh, measure, uh, why, why it's not succeeding, and no one could actually understand who is accountable for what when things go wrong, right?

120. 25:47

     So the goal we set for them is AI agent handles sixty percent of user queries, right, which were simple user queries, and then a way to identify and track them.

121. 25:57

     The key difference in this project that we-- when we did is that we selected the model in week seven, like, in the eight weeks POC, right? And this is how it, uh, turned out.

122. 26:07

     For the week one and two, we built the evaluation layer. We collected two hundred cases on their actual human agents answering to their customers on simple queries and understand how they are responding to them.

123. 26:21

     We created that database. Then we defined the success metrics. What does success look like to you? So out of, let's say, hundred queries, you need sixty queries, or the sixty percent of the queries that are simple queries to be, uh, uh, to be handled by the agent, right?

124. 26:36

     They needed some sort of accuracy. So eighty-five percent, it was around eighty-five percent accuracy target. They needed latency, all of the operational targets that you need. They were there.

125. 26:46

     Then we created this automated evaluation pipeline for them. And what I, what I mean by that is an automated system where you can capture a user's res- a user's question and the AI agent's response.

126. 26:57

     You take that, compare that aga- against your evaluation dataset. You rate that, and if the rating is below certain threshold, you get it checked by a human. And you-- if, if something goes wrong, you make sure that you find the solution.

127. 27:12

     So it could be a change to the prompt. It could be change to a tool calling system or something else. Once you have done that, you add that test case in the test dataset, so that when it happens next time, the test cages- cases catch them.

128. 27:24

     So the, the, the, the summary of that story is that your evaluation dataset is a living system. You start with two hundred, maybe. There is no correct number here.

129. 27:33

     But once you start, as you start building in production, this is a living system. This will keep growing. And the b- and the bigger it grows, the better your system will be.

130. 27:45

     In the second week, we talked-- we thought about the foundational layer, right? So the, the question data. We thought, we thought that, okay, if you have to call the database, have you got the API connections right?

131. 27:55

     Have you got a system in place that can trace those API connections? Are those secure, right? We were not talking about MCP at that time, right? It was just direct API calls to database to run queries.

132. 28:07

     Uh, have you got the distributed storage? Uh, have you got the-- have you-- are you collecting traces? And this is where when, when we started testing after building these systems, we could catch those duplicate API calls, right?

133. 28:19

     We could catch why customer satisfaction was dropping and stuff like that.

134. 28:24

     And then comes in week seven to eight, we started talking about models. Now that we had the evaluation dataset, we could run different models on that dataset to see the responses, compare them against the expected responses, and calculate a number on, on, on the accuracy, right?

135. 28:40

     That helped us to de- decide which model to use.

136. 28:44

     Now, that decision didn't take long, right? We, in fr- as I explained in the introduction, like, we would spend weeks debating on which model to use. But when you took the other approach, you can actually, uh, do that in a very quick way.

137. 29:00

     So once, once that's done, we, we stitched everything that I was talking about around observability, evaluation, the layers of evaluation. Once we had that system that can make AI visible, measurable, and accountable, that's when we started launching into production.

138. 29:18

     And that's when, uh... So this is, this is a result, uh, six weeks post-launch. Uh, we, we calculated the operational metrics. Of course, you know, the accuracy, the de- deflection rate, the response time, uh, the customer CSAT.

139. 29:31

     But what's important here is, in few weeks' time, when, uh, there was a problem with, uh... So one of the, one of the things that happened was that the bank changed some, uh, interest rate-related policies.

140. 29:43

     So when that-- when they changed the policy, they actually sent emails to customers or notifications in the application, uh, in the, in the, in the mobile banking app about the policy change.

141. 29:52

     But when the customers came and queried on the chatbot for further questions, they couldn't get the right answers, and they were like, uh, putting thumbs down on the answers.

142. 30:02

     So they were getting this feedback, right? So feedback decreased.

143. 30:06

     The problem with this kind of system, if you did not have this measurement system, is that you couldn't actually know what's happening. But because we had the measurement system in place, these-- the d- the drop in CSAT was detected, right?

144. 30:18

     Because we were getting negative feedback from customers. We could actually look into the tracing decisions and see that the agent was looking at a policy, policy document that was outdated.

145. 30:28

     So the, the new policy document was not updated in the vector database. The embeddings did not come through. Because it did not come through, it was giving it stale answers.

146. 30:37

     And that's when, uh, we went and fixed that. But it, it was all possible because we built that-- those systems that, uh, that led us to, to detect this.

147. 30:53

     Before you go, uh, I, I-- generally in these sessions, I share different, um, artifacts, uh, that you can take away. I have a QR code at the end for you to download, uh, and you'll find multiple ar-artifacts.

148. 31:04

     One of the important artifacts that I want to talk about is the production incident playbook. This is something that a lot of us tend to miss when we st-- work in AI projects.

149. 31:14

     And this playbook is basically a definition of what needs to happen when things fail in production. First, you detect using a eval dashboard, then you diagnose using a tracing, as I explained, then you contain.

150. 31:27

     So basically, you, you, you are versioning your prompts. Is there a, is there is a-- if there is a problem with the prompt, you b- uh, you, you take that prompt out, right?

151. 31:36

     And start the changes, deflect it to a human, or in my multi-agent orchestration video, I've talked about multiple fault tolerance, uh, failure recovery patterns, uh, around Saga pattern, compensation pattern, uh, and, uh, circuit breaker pattern, that even, uh, look into the video, I've explained them in details on how you can handle them.

152. 31:55

     And then you use the test case library to fix. So you look into LLMs judge reports, you look into your evaluation dataset reports, then you, uh, fix your problem.

153. 32:04

     Once you fix your problem, you put those test cases in your dataset, right? And, and create that eval suite that is a living system that will keep growing and, um, and, and you, you keep improving your AI system based on that, right?

154. 32:18

     But this playbook needs to be in place. When it runs in production, you will need to integrate it with your ITSM system so that it alerts the right person at the right time.

155. 32:27

     So, you know, a lot of this organization would have existing, uh, ITSM systems, right? So which, uh, which is used for al- uh, alerting and, uh, you know, making sure that the downstream systems don't get affected, et cetera, et cetera.

156. 32:38

     So, uh, one-once you have this in place, you can go and stitch it together to other systems.

157. 32:46

     So what can you do tomorrow, right? Start with-- If, if you have a project in mind, start with defining success. Success not from the technical sense, from the business sense.

158. 32:56

     What it means, means for the business, right? Come up with a few examples of what good answers look like and, and create a data, a dataset of that, and then build that pipeline using simple Python code.

159. 33:08

     See if you can automate that, so that when you run AI and get some response, it can become-- it can go and compare-- You can go and compare the answer against that dataset, and then that can be delivered, uh, to, to the, to the customer.

160. 33:26

     Now, these are three lessons that I have learnt, uh, while doing these things, which might-- you, you know, might qui-- easily miss. The test case library, as I explained, is a growing system.

161. 33:36

     It will grow over time. And because it grows over time, you need some sort of governance around it. You need a owner, right? You need to un-- You need to figure out which test cases relate to what kind of problem, so that whenever you go back to it, you can, you can relate your answers to those s-sort

162. 33:55

     of problems. Like, if it is a security-- If it's a login-- So you can say that the agent did not ask for login credentials when the, uh, customer asked to answer.

163. 34:02

     And all those kind of issues can be put under a security category within that dataset. So to categorize the rows in your dataset so that you can pick up what changed and compare it with them.

164. 34:12

     The second is prompt versioning. Now, when you start versioning prompts using Git, you know, we all know when you put Git message, commit messages tend to be, uh, simple commit messages.

165. 34:23

     But you have to put governance around what kind of commit messages you are putting in when you're changing these prompts. Because you need to understand when a prompt was changed, for exact what reason it was changed, right?

166. 34:34

     What was the failure that caused this prompt to be changed? What kind of failure would it address, and what would it correct, right, on-- in the next version? That needs to be documented.

167. 34:45

     Otherwise, it becomes difficult because when you go back and look into prompt versioning and look at different versions, and you cannot trace why, why those ch-changes were made, then it becomes difficult to track what's happening.

168. 34:56

     The third, the layer three evals, right? So the behavioral evals that I was talking about around tool calls and stuff like that, they can be really expensive as you grow your eval dataset as well.

169. 35:07

     So when you have a wrong tool call, for example, and you want to correct that system, when you correct it and run it against the eval dataset,

170. 35:16

     you have to basically run it against, let's say, if you've got three hundred, four hundred, five hundred r- uh, rows in the dataset, you have to run it against them.

171. 35:22

     And you do all the testing again and again and again and again. That can cost you a lot of money. So you have to put some governance around that.

172. 35:29

     So for example, when in your continuous integration pipeline, when you do the prompt ver-- uh, change, uh, you can actually, uh, put, uh, some checks around just, just selecting s-a small subset of the eval dataset to do the testing.

173. 35:45

     And you only do the full test when you merge to the main branch. So you can put these kind of decisions in place so that you can reduce cost around, uh, around, um, you know, expensive evals decision.

174. 35:59

     If you scan this QR code, it'll take you to a Google Drive link where I have put some examples on, uh, some of these, how these templates look like, what a evaluation checklist would look like.

175. 36:09

     Uh, I've given you some guide on, uh, set-setting up tracing, uh, with, uh, open source technologies, um, so that you can quickly set up some tracing and start testing in the test environment before you decide on what kind of tools you want to use.

176. 36:26

     Uh, thank you very much for listening to me. Uh, this is-- This QR code will take you to my LinkedIn profile. So I share, uh, I have a newsletter where I share these kind of topics every week.

177. 36:37

     So, um, if you're interested, you can join. It's free. Uh, I basically share what I learn in, in the field working with customers, right? So it might be useful for you.

178. 36:46

     Thank you very much. [audience applauding] [upbeat music]