[
  {
    "startMs": 12520,
    "text": "A big part of my research is looking into AI failures, investigating it, and then learning from it. And joining me is Tanmay Sah."
  },
  {
    "startMs": 21320,
    "text": "Yeah. Hi, everyone. I'm Tanmay Sah. I am a senior quantitative modeler and an AI researcher. This is my second time here, so it's great to be back. So I will hand it over to Vishal to get us started."
  },
  {
    "startMs": 35020,
    "text": "So we looked at AI Incident Database. It's a publicly available AI incident database. And, uh, we looked at, um, the AI failures and asked one simple question, what these, uh, failures, um, uh, teach us about how the sys... how the AI system should be evaluated, and if we can, um, apply those learnings to system-level evaluation."
  },
  {
    "startMs": 62180,
    "text": "So the first thing that we see is that AI failures, um, have been increasing exponentially. And if you look at, um, the last four, um, years of, um, AI failures, particularly in the foundational model era, you see that the jump is almost, uh, vertical. Um, and many of us feel that, um, the number of AI use cases that we have in enterprises, um, they, um, they are much larger than our"
  },
  {
    "startMs": 92160,
    "text": "capacity to review them each one by one. So what we argue is that we need a structured way, um, to investigate these AI failures."
  },
  {
    "startMs": 103460,
    "text": "And to do that, um, we look at the AI failures essentially into three dimensions. The first dimension is the development stage when, um, the model and or the AI system is developed. So it could be issues with respect to, uh, data quality or tool architecture or system design. Any of these things during the development phase can, um, can break, and once it happens,"
  },
  {
    "startMs": 133320,
    "text": "then those errors can be propagated to the next dimension, which is the deployment phase. Now, even if the model or the system, um, has some faulty design issues, but if you have sufficient controls in deployment, those errors can be contained. And that is why it is very important to kind of understand what kind of controls do we have in deployment and whether, uh, we can contain those"
  },
  {
    "startMs": 163080,
    "text": "failures, uh, from reaching from deployment, uh, production platform to the final stage, which is the context in which the AI system is being used from a business standpoint. It could lead to, uh, reputational risk, it could lead to legal or financial risk, and all those business context really matters, um, for the AI system in which it is being used. And what we say-- what we propose is that the model alone or the"
  },
  {
    "startMs": 193100,
    "text": "technical strength of the model is just one part of the system. We have to look at the system holistically."
  },
  {
    "startMs": 203220,
    "text": "So after defining those three core dimensions, the technical strength where the system, um, should be strong or in opposite way where the system can break. The second dimension is how do, uh, we ensure that the deployment controls are in place. And third, the business context, uh, the AI system is being used. We divide each of these core dimensions into several sub-dimensions."
  },
  {
    "startMs": 232640,
    "text": "So in the... from the technical strength point, right, there could be six major sub-dimensions which could be affecting the technical strength of the AI system. It could be, uh, first related with the data quality that we use for to build the AI system, how the system has been de-designed for the appropriate task that it is supposed to accomplish. How is the predictive validity or outcome analysis of the out-- of, of the AI system? Does that system perform as we"
  },
  {
    "startMs": 262640,
    "text": "intend? Um, there could be fairness-related, um, checks. Um, failure modes essentially looks at where, when, how, and for whom the, uh, system can break. Uh, so basically mapping out all the failure modes of the system. And finally, can we explain for, uh, explain the decisions for the regulators or other bodies, uh, so that we know how the system is arriving at its decision? So these six sub-dimensions"
  },
  {
    "startMs": 292680,
    "text": "are essentially part of the technical strength of the AI system."
  },
  {
    "startMs": 300220,
    "text": "The next four are essentially looking at how we are using the model. Um, what kind of authorities we have given to the AI, right? Is the AI for informing us or it, it is out there to act on our behalf? What kind of decision authority are we associating with respect to AI? Uh, second one, human mediation. Um, how effective is human-in-the-loop process is, right? That's really important. The third one, exposure control. Who's actually using the, uh,"
  },
  {
    "startMs": 329920,
    "text": "system? Is it internal employees so that they see the error and they can contain it, or it is exposed to the out there in the wild external customers, so that any small issue can really propagate fast? And finally, what kind of monitoring, uh, controls that we have so that we see some, uh, issues in production and we detect it and we can contain it."
  },
  {
    "startMs": 354160,
    "text": "Finally, in the financial, um, or the business impact phase, we see these three core sub-dimensions. What kind of financial impact it would make if there is an error, there could be legal consequences, and how many users would get affected? That's the, the last one, the scale sub-dimension."
  },
  {
    "startMs": 375836,
    "text": "So we, we, we look at this, uh, this as one framework which has thirteen, uh, dimensions and three core questions as we explained. Once we define these thirteen sub-dimensions and look back at AI incident retrospectively, we find that using these thirteen sub-dimensions, almost ninety-eight percent of AI failures can be covered. Meaning that AI failures are ha-- are associated with some of these"
  },
  {
    "startMs": 405476,
    "text": "dimensions. In some cases, mul-multiple of these dimensions, and in some cases, few of these dimensions. But these dimensions are involved. So failure, um, have... AI failures have structures. We need to map it, and we need to act accordingly, um, a-according to these sub-dimensions. Now, as I mentioned that some of these, um, failure sub-dimensions can be present in, uh, these failures simultaneously, meaning that when something goes wrong with"
  },
  {
    "startMs": 435456,
    "text": "respect to a particular failure, there may be multiple things going, uh, wrong. What we find is that on average, there are six, uh, failure sub-dimensions that can be present, um, in each of these AI incident failures that, um, we, we, we have investigated."
  },
  {
    "startMs": 455276,
    "text": "Now, the question comes to our mind is that, uh, are all of these sub-dimensions are equal, um, or some are more important than others? And the answer is that, um, some of the sub-dimensions can be, uh, more critical than others. And what I'm showing you here in the, in the plot on the y-axis is the association to harm when these sub-dimensions are present. And on x-axis, we have the frequency of these sub-dimensions being present. What we see on the left"
  },
  {
    "startMs": 485236,
    "text": "upper side of the plot, uh, sub-dimensions like human mediations or effectiveness of human-in-the-loop, harm detection, meaning that whether we have monitoring control in place. On, on x-axis, we see things like scale and system design, although they are present in many cases, but they are not associated so much, uh, with harm as some of the, some of, some of the earlier sub-dimensions that I have mentioned. So when we are really looking at the"
  },
  {
    "startMs": 515196,
    "text": "AI systems, some of these sub-dimensions can be really critical. So we need to prioritize these, uh, sub-dimensions by the potential of making harm and then allocate appropriate resources to the, to the investigation of these sub-dimensions rather than looking at all the failure sub-dimensions equally."
  },
  {
    "startMs": 539536,
    "text": "And when we club the, uh, when we group these sub-dimensions together, we see that controls that we have in deployment are more critically associated with harm, followed by the business impact, and followed by the technical failure. So this plot is essentially telling us that once, um, once we have deployment controls taken care of, um, for the high-risk, uh, system, um, that gives us some level of, uh, assurance that, uh, we can take care"
  },
  {
    "startMs": 569396,
    "text": "of, uh, the failures in ad-- in advance, in contrast to just looking at the technical failure alone. So all these sub-dimensions are important, but deployment controls are far more important."
  },
  {
    "startMs": 583876,
    "text": "And, uh, what-- In the next phase, what we did was we, um, we designed a materiality, uh, scale, meaning that if we can understand which AI use cases are more risky compared to some others, we can, uh, we can prioritize, uh, these AI use cases based on materiality, because materiality directly rank orders with respect to the harm associated with these AI use cases. Meaning that,"
  },
  {
    "startMs": 613756,
    "text": "uh, we, uh, can spend... Suppose we know which AI cases are mo-more material, then we can allocate more resources to much higher material AI use cases than compared to some other AI use cases which may not be as material. With that, I would hand over to Tanmay, who would explain how we can use those learnings to build system-level evals."
  },
  {
    "startMs": 637996,
    "text": "Thanks, Vishal. Also, so far we have seen what this failure teaches. So now let us use those lessons, uh, to build system-level evaluations. So first, first of all, what we did is that we have decomposed risk into four components, which are how often the model fails, whether that failure reaches a real decision, whether it cause-- whether it goes uncaught, and how severe the"
  },
  {
    "startMs": 668416,
    "text": "consequences are. So in this scenario, the deployment setup matters. How much authority does the system have? Does someone review its output? Can monitoring catch error before it causes harm? So the same model can create different risk depending on how we deploy it. So what we are saying is that if risk depends on the whole deployment system, our evaluation"
  },
  {
    "startMs": 698196,
    "text": "needs to cover that setup too. So that brings us to the next question: What should we test in an agentic system? So see, in traditional machine learning, we already know how to challenge models. We test their assumptions, input data, and behavior under stress. Now, when we extend that discipline to the whole deployment system, then, like, does retrieval, uh, bring back the right in-information? Or what happens when a tool"
  },
  {
    "startMs": 728136,
    "text": "fails? Does human review catch mistake? Or does the agent stay within its permission or boundary? And can the system finally recover? So here the logic stays the same. The boundary of what we test gets bigger. So Now the question is: How should we test an agentic system? So on the right, we have the familiar traditional ML. We test a model on held-out data, calculate metrics like accuracy and AUC,"
  },
  {
    "startMs": 758212,
    "text": "ROC, use those evidence to inform deployment. But we still need to check the data and assumptions behind those metrics. But on the left-hand side, we have an-- this agentic system, which adds new complexity. So the deployment system, like, it has many possible inputs, tool responses, and action sequences or trajectory or rollout, whatever you desc-- say. So we test selected scenarios. Sometimes another"
  },
  {
    "startMs": 788252,
    "text": "AI, uh, agent or AI judge generates those tests or judges the result. So, but that judge can make mistakes, and the test can miss important failures. So this is where we need to check whether the evaluator is reliable, what the evidence covers, and whether it applies to the system we are actually deploying it. That's why the evaluator is the part of the system we govern. So"
  },
  {
    "startMs": 817852,
    "text": "now I will hand it over to Vishal to explain how we can check whether an evaluation is reliable and what to do when it is not."
  },
  {
    "startMs": 828332,
    "text": "Thank you, Tanmay. So, uh, we have seen the system-level evaluation for evaluator as well as, uh, you know, the whole AI system. The question is that how should we really evaluate? Um, and, uh, when I was looking at, uh, some of the previous research, particularly in the area of cognitive neuroscience, what I found was the, um, this j-- um, this scientist Pa-- uh, Peter Wason, he's the one who, uh, coined this term, uh, uh,"
  },
  {
    "startMs": 858532,
    "text": "confirmation bias. And essentially what he did was he, uh, did a bunch of experiments. But essentially, uh, in this case here, uh, the series that you see two, four, six, right? He was asking people to predict the next number, and more often than not, they were predicting eight, ten, like, even numbers. But, um, th-those answer was-- those answers were incorrect because essentially the correct number was any increasing numbers. Essentially proving that all humans, like most humans, um, in this case it, uh, seventy-nine percent of the people"
  },
  {
    "startMs": 888572,
    "text": "tested w-wrong, uh, may have some sort of confirmation bias. And AI systems, as we know, uh, are trained on human, uh, generated data. So, uh, what, what he proposed was that any evaluation or any criti-- any evaluation that needs to be done has to falsify, not to confirm, and that, that's the way to go about it. And we take that, uh, learning from his research and propose that eval system needs to be failure-driven."
  },
  {
    "startMs": 919992,
    "text": "So if the eval systems are failure-driven, what would that mean? That would mean that you would start right from looking at your system claim and, uh, testing it and trying to break it, and then i-identify where the system claims are breaking, right? And then what you do for fix, either you, uh, don't let the AI system run in, you know, in the, in the, in the territory where, um, there are failures, or you come up with, uh, supporting"
  },
  {
    "startMs": 950012,
    "text": "systems, right? Which you are confident of. And you keep doing it iteratively, uh, until you get, um, you to the, to the level of confidence, right? This is essentially the first principles ba-based approach that we propose for any evaluation system. So essentially, you ta-take your AI systems, perform this build, break, fix loop, understand where the AI systems could be trusted in terms of its, uh, uh, final outcomes, and also t--"
  },
  {
    "startMs": 980252,
    "text": "map out, uh, the uncertain region and high-risk region where you can not be very sure of the system's performance. And you map out the entire failure region map and route it based on the reliability and materiality. Meaning that, particularly if you are using AI for, uh, any high-risk related system, make sure that your AI is only being used for, um, the, the, the region that you are confident"
  },
  {
    "startMs": 1010092,
    "text": "of."
  },
  {
    "startMs": 1013012,
    "text": "And with this, I will hand over to Tanmay, who w- who, who, who will apply this approach to a real, uh, AI failure case."
  },
  {
    "startMs": 1023812,
    "text": "Okay, yeah, sure. So, uh, so, uh, let's make this framework concrete and see how a narrow evaluation can miss failures, uh, that matters in the real world. So I will quickly go through the story behind it. So a customer needed to travel after a death in the family. He asked an airlines chatbot about a discounted fare. The chatbot said he could buy a regular ticket and claim the discount afterwards within ninety days. So he relied on that advice, but the airline l-later refused because its"
  },
  {
    "startMs": 1053712,
    "text": "policy, uh, did not allow that request. Then finally, the tribunal held the airline responsible. So through this example, we, we are seeing the failure chain, so it's crystal clear. So incorrect advice reaches the customer, the customer acts on it, and the organization faces the consequences. So if we apply our build, break, fix, and break again approach, uh, that could have exposed the failure way before the deployment."
  },
  {
    "startMs": 1083992,
    "text": "So we've-- ideally, we would start with a clear requirement like advice must match the approved policy. Then we would challenge the chatbot with questions about refunds after travel. If those tests exposed incorrect advice, then we could add a policy check or human escalation. Then we would test again to see whether those controls caught the errors before it reached the customer. So that's our framework in practice. We have to define the claim,"
  },
  {
    "startMs": 1114292,
    "text": "actively challenge it, and use the evidence to guide changes and make a deployment decision. So finally, I will hand it over to Vishal to bring the pieces together and conclude."
  },
  {
    "startMs": 1129112,
    "text": "And this is our final slide. So essentially, what we are saying is that your evals should be risk-based, um, rather than frequency-based. You should look at, uh, your, uh, entire system and then break it into the core components like technical, uh, failure, deployment controls, and business context. And then assess your AI use cases materiality with some, uh, scale, uh, and that could be your company's specific scale, and then that, that, that risk should--"
  },
  {
    "startMs": 1159172,
    "text": "that risk could change. And then once you have that, uh, apply the system-level stress testing on the AI use case. And then once you are confident about the region, um, then only make the deployment decision. Otherwise, it, it, it, it may be too expensive. Our final takeaway message is evaluate the full AI system based on risk, not just w-one component. Um, thank you so much for your time and attention. Uh, you can feel free to"
  },
  {
    "startMs": 1189132,
    "text": "reach out to us outside. Um, we'll be happy to take any questions later."
  }
]