1,368 AI Failures and What They Teach About Evals — Vishal Srivastava, New York Life & Tanmay Sah

An enterprise can accumulate more AI use cases than its reviewers can assess one by one. Vishal Srivastava and Tanmay Sah approach that capacity problem through a study of 1,368 incidents from the public AI Incident Database. Their starting observation is a sharp rise in recorded failures, particularly over the preceding four years. Incident counts alone do not establish a failure probability per deployment, but they motivate a practical question: how should limited evaluation effort be directed? 0:35

The framework begins with development. Problems in data quality, tool architecture or system design can create a technical fault. That fault can propagate into production, where deployment controls determine whether it is contained. A model can have design weaknesses while the surrounding system still prevents a particular error from reaching the people who would act on it.

The third layer is business context: what follows when an error escapes those controls. Reputational damage, legal consequences and financial loss depend on how the system is being used. Evaluating technical strength therefore leaves an essential question open: what happens to the model’s mistakes after they leave the model?

The three layers become 13 dimensions that an evaluator can investigate. Six concern technical strength:

• Data quality: What weaknesses in the data used to build the system could affect its behavior?
• System design: Does the design fit the task the system is supposed to accomplish?
• Predictive validity and outcome analysis: Does the system produce the intended outcomes?
• Fairness: Are fairness-related checks part of the assessment?
• Failure modes: Where, when, how and for whom can the system break?
• Explainability: Can its decisions be explained to regulators or other bodies that need to understand them?

Four concern deployment controls, where the questions shift from model behavior to how that behavior is used:

• Decision authority: Is the AI supplying information, or can it act on someone’s behalf?
• Human mediation: How effective is the human review process? Having a person in the loop is useful only to the extent that the process catches or contains mistakes.
• Exposure control: Who encounters the output? Internal employees may detect and contain an error; direct exposure to external customers can let it propagate quickly.
• Monitoring and harm detection: Can production monitoring detect an issue in time for the organization to contain it?

Three describe business impact:

• Financial impact: What financial consequences could an error create?
• Legal consequences: What legal consequences could follow from the system’s behavior?
• Scale: How many users could be affected?

Applied retrospectively, these 13 dimensions cover almost 98% of the incidents in the study. An incident has an average of six dimensions present. This is coverage of a classification: several categories can describe the same event. The overlap is the useful finding. An investigation needs to account for combinations of weaknesses across the system, including the fault itself and the conditions that let it become consequential.

The frequency-versus-harm plot asks whether the dimensions that appear most often are also the ones most strongly associated with harm. Its horizontal axis records frequency; its vertical axis records association with harm. Human mediation and harm detection appear toward the upper left. Scale and system design occur frequently, but their association with harm is weaker than that of those deployment controls. 7:35

That separation changes how evaluation time should be spent. A review process organized entirely around common failure categories can underweight controls whose failure has greater consequences. Human review needs to be tested for whether it catches mistakes, and monitoring for whether it detects trouble early enough to contain it. Their presence on an architecture diagram would not answer those questions.

Grouping the dimensions into the three larger layers puts deployment-control failures first in association with harm, followed by business impact and technical failure. These are retrospective associations; they do not establish how much harm a particular intervention would prevent. They do make the deployment configuration a central object of evaluation, especially for systems whose mistakes could be costly.

Materiality turns the findings into a way to prioritize use cases. The proposed scale ranks AI applications by how much is at stake, with greater materiality reported to align with greater associated harm. Its purpose is to allocate review effort: a more consequential application should receive more investigation and resources than one whose errors have limited effects.

Sah then separates risk into four questions:

• Model failure: How often does the model fail?
• Reaching a decision: Does that failure influence a real decision?
• Escaping detection: Does the failure go uncaught?
• Consequences: How severe is the resulting harm?

This decomposition explains why the same model can create different risks in different deployments. Giving it more decision authority changes the route from an error to action. Human review and monitoring can interrupt that route if they work in time. Evaluating the complete setup therefore means testing the model together with the permissions, review processes and monitoring under which it will actually operate.

Traditional machine learning already has a discipline of challenging assumptions, input data and behavior under stress. Extending it to an agent makes more of the deployed system part of the test:

• Retrieval: Does it bring back the right information?
• Tools: What happens when a tool fails?
• Human review: Does the reviewer catch a mistake?
• Permissions: Does the agent stay within the actions it is allowed to take?
• Recovery: Can the system recover when something goes wrong?

The comparison between traditional models and agents asks what evidence a deployment decision rests on. A traditional model can be evaluated on held-out data, using metrics such as accuracy and ROC AUC. Those results still depend on the data and assumptions behind the test; the metric does not remove the need to examine them.

An agent introduces many possible inputs, tool responses and action sequences. Evaluation samples selected scenarios from those possibilities. Another AI may generate the tests, judge the results, or do both. That creates two distinct ways to get misleading evidence: the selected scenarios can miss an important failure, and the judge can misclassify a result that was tested.

A passing set of scenarios therefore needs three further checks:

• Evaluator reliability: Can the evaluator make the judgments being relied on?
• Coverage: Which behaviors and failures does the evidence actually cover?
• Deployment match: Does the evidence apply to the system being deployed, with its particular setup?

The evaluator becomes part of the system that must be governed. An AI judge’s verdict cannot, on its own, establish the reliability of the process that produced that verdict. 13:08

The confirmation-bias example begins with 2, 4, 6. It is easy to infer a rule of even numbers increasing by two and continue with 8, 10. Those continuations also fit the broader rule given in the example: any increasing numbers. Their compatibility therefore cannot distinguish the narrow rule from the broad one. The mistaken confidence lies in treating an example that fits as if it had established the rule.

That is the trap for an evaluation designed around the behavior its builders already expect. More confirming examples can leave the important uncertainty untouched. Failure-driven evaluation deliberately looks for evidence that would falsify the system’s claim. The test has to create an opportunity for the claim to fail, so the result can change what the team believes the system is capable of doing.

Build, break, fix begins with a system claim, then deliberately tests where it breaks. A failed test identifies a condition under which the claimed outcome cannot yet be trusted. The next decision concerns what the system should be allowed to do in that condition.

The framework offers two responses:

• Restrict the use: Prevent the AI from operating in the region where failures occur. Reducing the allowed scope can be a valid fix without changing the underlying model.
• Add supporting controls: Introduce a supporting system that the team has reason to trust, then test the combined behavior again.

The iteration continues until the evidence supports sufficient confidence in the intended outcomes.

The result should be a map of where outcomes are trusted, where uncertainty remains, and where failures carry high consequences. Reliability and materiality describe different properties: confidence in performance and the stakes of getting it wrong. Routing needs to consider both. For a high-risk use case, the proposed rule is to keep AI operation within the region where the team has adequate confidence, while further tests and controls can change that scope.

A customer needed to travel after a death in the family and asked an airline’s chatbot about a discounted fare. The chatbot said he could buy a regular ticket and claim the discount afterward, within 90 days. He relied on that advice. The airline later refused the request because its policy did not permit it, and a tribunal held the airline responsible. 17:03

The failure develops through observable changes: an incorrect policy statement reaches a customer, the customer acts on it, and the organization faces the consequences. The consequential action here is the customer’s reliance on the advice. An evaluation of answer quality needs to account for the delivery and decision process that allows a wrong answer to become a real commitment.

The proposed evaluation starts with a requirement: advice must match the approved policy. Questions about refunds after travel directly challenge that requirement. A response that permits a later discount when the approved policy does not allow it exposes a specific failure. The test now identifies the discrepancy the system must prevent, rather than merely collecting another plausible answer.

Sah proposes adding a policy check or human escalation, then repeating the challenge to establish whether the control catches the error before it reaches the customer. This is a proposed prevention test whose effectiveness would have to be demonstrated. The comparison below asks where the same incorrect answer could be stopped: the intervention belongs before delivery, while a missed error leaves the path to customer reliance open.

The closing deployment decision combines the materiality of a use case with evidence from stress testing the full system. An organization can use its own materiality scale, and the risk assessment needs to accommodate changes in the use case. Technical behavior, deployment controls and business context all remain part of that assessment. Deployment should proceed within the region where the tests establish sufficient confidence. When confidence is missing in a high-stakes region, restrict the scope; discovering the gap through a customer may be too expensive. 18:49