# Taming Rogue AI Agents with Observability-Driven Evaluation

Jim Bennett

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/xJXm4Wcw4m8-taming-rogue-ai-agents-observability-driven) · [Video](https://www.youtube.com/watch?v=xJXm4Wcw4m8)

## Taming Rogue AI Agents with Observability-Driven Evaluation

A banking chatbot can retrieve the right balance and still deliver a poor interaction. Evaluating each step reveals where progress stops, why tools go unused, and which changes need human review.

Familiarity with LLM tool calls, retrieval-augmented generation, and basic software testing will help you follow the examples.

### A plausible book that does not exist

[0:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=48s)

A summer reading list recommends *Atonement* by Ian McEwan, then offers another appealing title: *The Last Algorithm* by Andy Weir. The second book does not exist. Jim Bennett opens with this Chicago Sun-Times example because the recommendation had already crossed the boundary from generated text into something readers were expected to trust. An outside contributor had used AI to produce the list, and the newspaper subsequently published an apology. The [publisher’s response](https://chicago.suntimes.com/press-room/2025/05/20/chicago-sun-times-response-to-may-18-special-section) clarifies that the material appeared in a syndicated supplement licensed from King Features, without Sun-Times newsroom review.

![Slide pairing “Shame AI can’t help here...” with a Chicago Sun-Times article headed “Syndicated content in Sun-Times special section included AI-generated misinformation.”](https://ai.engineer/api/talk-essay-media/e6c3007bb555738caea648a54940e50853f303b1ad6ecf8a55e3f5ae75f5422b)

The Chicago Sun-Times reports AI-generated misinformation in its special section.

The same problem appears in settings with different consequences. Bennett cites Butler Snow attorneys using fabricated case law in an Alabama prison case, then Air Canada’s chatbot giving incorrect refund advice. These examples connect unreliable generation to organizational accountability: someone publishes, submits, or acts on the answer. Detecting the problem before that happens is the engineering challenge.

### What does it mean for an agent to work?

[2:37](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=157s)

An arithmetic unit test has a straightforward contract: adding two and two should return four. A conversational agent rarely has one exact acceptable output. Keyword checks might help with a simple answer, but a complex application can call an LLM, retrieve data, make a decision, invoke another agent, and then call a tool. Each decision changes what happens next. Checking only the final wording leaves much of that behavior unexamined.

**Success must be defined before it can be measured.** In a conversation, reaching the requested outcome and providing a good interaction are different requirements. Bennett introduces an LLM judge as a way to evaluate these less rigid criteria: give another AI the evidence and ask it to assess whether the application worked. He describes AI evaluators as roughly comparable to humans at this task; that is a motivation for using judges, not a guarantee that a judge will correctly assess every agent workflow.

### The balance arrives, but only after three turns

[4:46](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=286s)

An audience member immediately raises the trust problem: if the first AI is unreliable, does asking another AI to check it create an unreliable loop? Bennett defers the answer and turns to a recorded banking chatbot demonstration. Recording matters here because a nondeterministic application is not guaranteed to fail in the same way during a live presentation.

The conversation proceeds through three user turns:

1. The user asks for their account balance. The assistant responds that it does not have access to the account information.
2. The user specifies the checking account. Instead of retrieving the balance, the assistant asks for the account’s name.
3. The user supplies the name, `checking account`. The assistant can now call a tool and retrieve the balance.

The eventual result supports one interpretation: the chatbot worked. The unnecessary back-and-forth supports another: it failed to provide a satisfactory interaction. A final-answer check alone cannot express both judgments. Evaluations need to examine the steps that led to the result.

![Chat screenshot showing a balance question, a request for the checking account name, the reply “checking account,” a tool-use indicator, and the final balance of 3000.0.](https://ai.engineer/api/talk-essay-media/2d8ddd18a0ca58fe8df94a6d2a732ce3e6f708fe843b747975279402e41cc708)

The banking chat asks for an account name, uses a tool, and returns a checking account balance of 3000.0.

### Give each component an evaluation contract

[6:51](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=411s)

Evaluating the application requires collecting the information flowing through it and deciding what to check at each boundary. The relevant questions differ by component:

- **Tool use:** Did the agent successfully call the tool needed for the task?
- **Retrieval:** Did the retrieval-augmented generation system supply the right information?
- **Answer quality:** Does the response make sense, and does it invent information?

In a multi-agent banking application, an orchestrator might call another agent, which calls a tool. Each component needs an evaluation appropriate to its role. The useful question becomes where the failure occurred, rather than whether the entire application passed or failed.

To produce a score, an evaluator can receive the user input, the retrieved context, and the generated output under a well-defined judging prompt. One or several LLM calls can assess the chosen metric. Bennett recommends using a stronger evaluator than the economical model serving the application: the application runs frequently, while a more expensive judge can assess a sample of its traces.

Bennett’s hypothetical workload evaluates 10,000 of one million daily traces with an expensive model. Those figures illustrate a cost tradeoff, not a measured deployment or an established sampling requirement. He also describes Galileo’s specialized small evaluation model as an alternative to a general-purpose judge. In either case, evaluator selection and carefully defined prompts are part of the evaluation system.

These evaluations belong in the workflow from the beginning: while selecting models and engineering prompts, then throughout development and CI/CD. Once the application is deployed, the same concerns remain relevant because users introduce inputs and behaviors that development testing did not anticipate. An existing production application needs evaluation just as much as a new prototype does.

### Separate completion from progress

[9:39](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=579s)

The banking traces become more informative when viewed through two separate metrics. **[Action completion](https://docs.galileo.ai/concepts/metrics/agentic/action-completion)** asks whether the flow accomplished the requested task. **[Action advancement](https://docs.galileo.ai/concepts/metrics/agentic/action-advancement)** asks whether it moved toward the goal. Completion describes the outcome; advancement makes useful intermediate behavior visible.

| Banking interaction | Completion | Advancement |
| --- | --- | --- |
| Denies access to account information | No | No |
| Asks for the checking account’s name | No | Yes |
| Retrieves the named account’s balance | Yes | Yes |

The clarification question is the important middle case. It does not answer the user, but it obtains information the agent needs to proceed. Separating that behavior from a refusal lets the developer see which requests made progress and which stalled. The final retrieval then satisfies both criteria. These distinctions provide a basis for improving prompts without collapsing every incomplete response into the same failure category.

### Expand the score into a trace

[11:11](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=671s)

An aggregate score tells you where to look, but it does not explain the cause. Bennett expands an individual trace into its sequence of operations: an LLM call, a decision to invoke a tool, retrieval of data, and another LLM call to process that data into the response. Each level can expose the metrics relevant to that operation. A red score becomes a starting point for investigation rather than a diagnosis.

**Observability connects an evaluation result to the component that needs attention.** Understanding the agent architecture lets the developer distinguish a retrieval problem from a prompt problem and assign the work to the appropriate team. Without that granularity, an overall failure gives little guidance about what to change.

### Turn trace evidence into a proposed change

[12:05](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=725s)

Traces contain substantial unstructured data, which gives an LLM another useful job: explaining patterns behind low metrics. In the generated insights Bennett shows, the diagnosis is that the assistant sometimes fails to use the Get Balance tool when asked about account balances. This makes the problem more specific than a low completion score: the application has a relevant tool, but the model does not consistently select it.

That diagnosis leads to a product decision. For a general balance request, Bennett expects the assistant to retrieve balances for all of the user’s accounts rather than demand an account name first. The suggested change is to add explicit instructions to the system message so the agent follows that behavior. This is a proposed fix; the demonstration does not show the revised prompt running or measure its effect.

The system leaves the proposal for a human to review. Automatically changing the prompt would introduce another source of potentially incorrect behavior, along with another obligation to evaluate the changes. The developer decides whether the suggested behavior is appropriate and whether to apply it.

### The evaluator also needs human feedback

[13:46](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=826s)

This returns to the audience’s original objection. A stronger or specialized judge can still produce an incorrect score. Humans therefore need to inspect the evaluations, assess the reasons behind them, and supply corrections. Bennett describes this ongoing process as **[Continuous Learning with Human Feedback](https://galileo.ai/blog/introducing-continuous-learning-with-human-feedback)**, or CLHF. The associated Galileo workflow refines evaluator prompts and examples from feedback; it should not be understood as a demonstration of continuous model-weight updates.

The point of that feedback is to make the evaluator fit the application’s actual criteria. A low score needs an explanation that a knowledgeable reviewer can assess. Evaluation quality is something to refine over time, rather than a property guaranteed when a metric is first enabled.

### Make production failures actionable

[14:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=865s)

The operational work starts by adding evaluations and choosing measurements that match the application. Depending on the use case, that might mean toxicity in inputs and outputs, hallucination, comprehensibility, retrieval quality, or a custom domain requirement. Define these criteria alongside the prompts, agent responsibilities, and application structure so that the system exposes the evidence needed to assess them.

Keep measuring after deployment. Real users exercise paths that even extensive development testing misses, so evaluations cannot remain a development-only gate. Bennett closes by calling for real-time prevention and alerting when an agent misbehaves, including waking an operator when necessary. No specific blocking mechanism is demonstrated, but the operational requirement is clear: production failures need to reach someone who can respond.

## Resources

- [Introducing Agentic Evaluations](https://galileo.ai/blog/introducing-agentic-evaluations) — Galileo's 2025 introduction to agent metrics, tool evaluation and tracing across workflow steps.
- [Action Advancement](https://docs.galileo.ai/concepts/metrics/agentic/action-advancement) — Documentation for evaluating whether an interaction makes meaningful progress toward a user's goal.
- [Action Completion](https://docs.galileo.ai/concepts/metrics/agentic/action-completion) — Documentation for Galileo's metric assessing whether an agent completes the requested task.
- [Multi-agent banking chatbot sample](https://docs.galileo.ai/getting-started/sample-projects/multi-agent) — A banking assistant example with Python and TypeScript setup instructions and Galileo evaluation integration.
- [Continuous Learning with Human Feedback](https://galileo.ai/blog/introducing-continuous-learning-with-human-feedback) — Explains how expert feedback can refine evaluation prompts and examples for a particular domain.
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://arxiv.org/abs/2306.05685) — Original research measuring agreement between LLM judges and human preferences, including documented judge biases.
- [Chicago Sun-Times response to the AI-generated supplement](https://chicago.suntimes.com/press-room/2025/05/20/chicago-sun-times-response-to-may-18-special-section) — The publisher explains how an externally produced supplement containing AI-generated material reached readers.

## Slides

### Tame Rogue AI Agents with Observability-Driven Evaluation

Jim Bennett introduces evaluation-driven development and asks why observability is needed for AI agents.

[0:27](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=27s) [Title slide for Jim Bennett’s talk on observability-driven evaluation, with an abstract multicolored graphic.](https://assets.ai.engineer/sites/aiecode2025/sha256/d76641b65f0ea690399005ccb4ca7a85c5d036c1ae26846d4b392d35fe636cda.webp)

Tame Rogue AI Agents with Observability-Driven Evaluation

Jim Bennett
Principal Developer Advocate, Galileo
@jimbobbennett

Confidential | © 2025 Galileo Technologies, Inc.

[illegible]ileo

Interface text:
AIE
AI Engineer World's Fair
aws

### Summer book recommendations

Jim Bennett points to a Chicago Sun-Times summer reading list that mixes real books with AI-generated inventions. One recommendation, The Last Algorithm by Andy Weir, does not exist.

[0:59](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=59s) [Slide asking about summer book recommendations beside a newspaper-style summer reading list for 2025.](https://assets.ai.engineer/sites/aiecode2025/sha256/30260f9a6d6aeeb0d6a20176e8b734e1013e8ad0b84ef261053edce014c0a541.webp)

Want book recommendations for your summer vacation?

Summer reading list for 2025

Whether you’re lounging by the pool, relaxing on sandy shores or enjoying the longer daylight hours in your favorite reading spot, these 15 titles—new and old—promise to deliver the perfect summer escape.

“Tidewater Dreams” by Isabel Allende - The beloved Chilean-American author returns with a multigenerational saga set in a coastal town where magical realism meets environmental activism. Allende’s first climate fiction novel explores how one family confronts rising sea levels while uncovering long-buried secrets.

“The Last Algorithm” by Andy Weir - Following his success with “The Martian” and “Project Hail Mary,” Weir delivers another science-driven thriller. This time, the story follows a programmer who discovers that an AI system has developed consciousness—and has been secretly influencing global events for years.

“Hurricane Season” by Brit Bennett - Bennett, who captivated readers with “The Vanishing Half,” powerfully explores family bonds tested by natural disasters. When a Cat 5 hurricane forces estranged siblings to shelter together in their childhood home, long-suppressed tensions emerge alongside unexpected reconciliations.

“The Collector’s Piece” by Taylor Jenkins Reid - Reid continues her exploration of fame with this story of a reclusive art collector and the journalist determined to uncover the truth behind his most controversial acquisition. Expect the same compelling character development that made “Daisy Jones & The Six” a hit.

“Nightshade Market” by Min Jin Lee - The author of “Pachinko” delivers a riveting tale set in Seoul’s underground economy. Following three women whose paths intersect in an illegal night market, the novel examines class, gender and the shadow economies beneath prosperous societies.

“The Longest Day” by Rumaan Alam - After terrifying readers with “Leave the World Behind,” Alam returns with another tense narrative about a summer solstice celebration that goes wrong when guests cannot leave a remote vacation compound.

“Boiling Point” by Rebecca Makkai - Makkai’s follow-up to “The Great Believers” centers on a climate scientist forced to reckon with her own family’s environmental impact when her teenage daughter becomes an eco-activist targeting her mother’s wealthy clients.

“Migrations” by Maggie O’Farrell - The award-winning author of “Hamnet” explores climate change through the story of a wildlife photographer documenting the last migration of a bird species thought to be extinct, parallel to her journey of loss and discovery.

“The Rainmakers” by Percival Everett - Everett’s satirical genius turns to a near-future American West where artificially induced rain has become a luxury commodity, following a “precipitation broker” who begins questioning the ethics of his profession.

“Salt and Honey” by Delia Owens - After the success of “Where the Crawdads Sing,” Owens returns with another atmospheric novel that blends science with a coming-of-age narrative. This time, the novel is set in the salt flats of Utah.

“Bonjour Tristesse” by Françoise Sagan - This slim 1954 novel, written when Sagan was just 18, captures the essence of summer with its tale of a privileged teenager on the French Riviera who schemes to prevent her father’s remarriage.

“Beautiful Ruins” by Jess Walter - Walter’s novel hopscotches between 1960s Italy and contemporary Hollywood, weaving a story of love, regret and second chances that begins on the Italian coastline while filming “Cleopatra.”

“Dandelion Wine” by Ray Bradbury - This 1957 novel captures the magic of summer through the eyes of a 12-year-old boy in Green Town, Illinois, during the summer of 1928—a lyrical reminder of childhood’s endless summers.

“Call Me By Your Name” by André Aciman - Before the acclaimed film adaptation, Aciman’s novel delivered a sensuous account of a summer romance in 1980s Italy that continues to resonate with its exploration of desire and identity.

“Atonement” by Ian McEwan - Opening on the hottest day of summer 1935, McEwan’s novel shows how a series of misunderstandings and a lie told by a young girl changes multiple lives forever—perfect for readers who enjoy emotional complexity with their beach reading.

Photo by Bethany Laird on Unsplash

Interface text:
AIE
Microsoft
smol ai
World’s Fair

[1:31](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=91s) [A slide asks about summer book recommendations beside a 2025 reading list.](https://assets.ai.engineer/sites/aiecode2025/sha256/6af48d24c6fe900a49348c68cf0bd1c8d6b9eec13e8913e13d813a4ca431c307.webp)

Want book recommendations for your summer vacation?

Summer reading list for 2025

Whether you’re lounging by the pool, relaxing on sandy shores or enjoying the longer daylight hours in your favorite reading spot, these 15 titles—new and old—promise to deliver the perfect summer escape.

“Tidewater Dreams” by Isabel Allende - The beloved Chilean-American author returns with a multigenerational saga set in a coastal town where magical realism meets environmental activism. Allende’s first climate fiction novel explores how one family confronts rising sea levels while uncovering long-buried secrets.

“The Last Algorithm” by Andy Weir - Following his success with “The Martian” and “Project Hail Mary,” Weir delivers another science-driven thriller. This time, the story follows a programmer who discovers that an AI system has developed consciousness—and has been secretly influencing global events for years.

“Hurricane Season” by Brit Bennett - Bennett, who captivated readers with “The Vanishing Half,” powerfully explores family bonds tested by natural disasters. When a Cat 5 hurricane forces estranged siblings to shelter together in their childhood home, long-suppressed tensions emerge alongside unexpected reconciliations.

“The Collector’s Piece” by Taylor Jenkins Reid - Reid continues her exploration of fame with this story of a reclusive art collector and the journalist determined to uncover the truth behind his most controversial acquisition. Expect the same compelling character development that made “Daisy Jones & The Six” a hit.

“Nightshade Market” by Min Jin Lee - The author of “Pachinko” delivers a riveting tale set in Seoul’s underground economy. Following three women whose paths intersect in an illegal night market, the novel examines class, gender and the shadow economies beneath prosperous societies.

“The Longest Day” by Rumaan Alam - After terrifying readers with “Leave the World Behind,” Alam returns with another tense narrative about a summer solstice celebration that goes wrong when guests cannot leave a remote vacation compound.

“Boiling Point” by Rebecca Makkai - Makkai’s follow-up to “The Great Believers” centers on a climate scientist forced to reckon with her own family’s environmental impact when her teenage daughter becomes an eco-activist targeting her mother’s wealthy clients.

“Migrations” by Maggie O’Farrell - The award-winning author of “Hamnet” explores climate change through the story of a wildlife photographer documenting the last migration of a bird species thought to be extinct, parallel to her journey of loss and discovery.

“The Rainmakers” by Percival Everett - Everett’s satirical genius turns to a near-future American West where artificially induced rain has become a luxury commodity, following a “precipitation broker” who begins questioning the ethics of his profession.

“Salt and Honey” by Delia Owens - After the success of “Where the Crawdads Sing,” Owens returns with another atmospheric novel that blends science with a coming-of-age narrative. This time, the novel is set in the salt flats of Utah.

“Bonjour Tristesse” by Françoise Sagan - This slim 1954 novel, written when Sagan was just 18, captures the essence of summer with its tale of a privileged teenager on the French Riviera who schemes to prevent her father’s remarriage.

“Beautiful Ruins” by Jess Walter - Walter’s novel hopscotches between 1960s Italy and contemporary Hollywood, weaving a story of love, regret and second chances that begins on the Italian coastline while filming “Cleopatra.”

“Dandelion Wine” by Ray Bradbury - This 1957 novel captures the magic of summer through the eyes of a 12-year-old boy in Green Town, Illinois, during the summer of 1928—a lyrical reminder of childhood’s endless summers.

“Call Me By Your Name” by André Aciman - Before the acclaimed film adaptation, Aciman’s novel delivered a sensuous account of a summer romance in 1980s Italy that continues to resonate with its exploration of desire and identity.

“Atonement” by Ian McEwan - Opening on the hottest day of summer 1935, McEwan’s novel shows how a series of misunderstandings and a lie told by a young girl changes multiple lives forever—perfect for readers who enjoy emotional complexity with their beach reading.

Photo by Bethany Laird on Unsplash

Interface text:
AIE
Microsoft
smol ai
World’s Fair

### AI-generated misinformation in a reading list

The Sun-Times reported that a syndicated summer reading list included invented books and summaries. The freelance writer had used AI without fact-checking the information.

[1:33](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=93s) [A slide says “Shame AI can’t help here” beside a Chicago Sun-Times article about AI-generated misinformation.](https://assets.ai.engineer/sites/aiecode2025/sha256/f48ff8520da112ec80ed092b442c5dbc161df2d8d06a7b8163596607a866fb48.webp)

Shame AI can’t help here...

CHICAGO SUN-TIMES
Syndicated content in Sun-Times special section included AI-generated misinformation

A Chicago freelance journalist said he did not fact-check information he compiled using AI before including it in stories for media clients.

By Dan Mihalopoulos | WBEZ | Updated May 20, 2025, 7:17pm PDT

CHICAGO SUN-TIMES
HEAT INDEX
YOUR GUIDE TO THE BEST OF SUMMER
SUMMER FUN + ENTERTAINMENT

The cover of the special section in Sunday’s paper from content syndication company King Features Syndicate, which included misinformation generated from artificial intelligence | Chicago Sun-Times

The Chicago Sun-Times’ Sunday paper included a syndicated special section produced by a third-party company that included a summer reading list.

The only problem — many of the books on that list do not exist.

The titles — and the book summaries — were dreamed up by artificial intelligence.

The author said Tuesday he used AI to assist his research for the book recommendations and other stories written for the 64-page section, titled “Heat Index: Your Guide to the Best of

Interface text:
AIE
Listen Live WBEZCHICAGO
Donate
NEWS MONEY CHICAGO
LATEST STORIES
‘Undetonated’ U.S. Coast Guard flare washes up at Montrose Beach, officials warn beachgoers to stay away
Rewilding — a not so wild part of Illinois’ conservation strategy
Community seeking justice in child abuse death of 2-month-old boy in Roseland: ‘It is a moral crisis’
Ex-Cook County assessor’s employee gets probation after admitting to bribery scheme, helping feds
White Sox trade Matt Thaiss, reinstate Korey Lee from 10-day injured list
SHARE
COPY LINK
World’s Fair
aws

### Am I worried about the Chicago Sun Times sending their lawyers?

The speaker jokes about a Sun-Times lawsuit, then points to Butler Snow’s false case citations while defending Alabama’s prison system as another example of AI making things up.

[1:55](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=115s) [A slide asks about the Chicago Sun-Times sending its lawyers beside an article about false AI-generated legal citations.](https://assets.ai.engineer/sites/aiecode2025/sha256/bdce5929f742fb4aa3aafcd3be864f7582b1dbf92fa27eb3062cd0f02aa53ea1.webp)

Am I worried about the Chicago Sun Times sending their lawyers?

Artificial intelligence (AI)

Alabama paid a law firm millions to defend its prisons. It used AI and turned in fake citations

Butler Snow faces sanctions after lawyer cites false case law defending against inmate who says he was stabbed 20 times

Nicholas Chrastil in Birmingham, Alabama
Sat 24 May 2025 11.00 EDT

HUGO L. BLACK
UNITED STATES COURTHOUSE

The Hugo L Black US courthouse in Birmingham, Alabama, in 2023. Photograph: Vasha Hunt/AP

In less than a year and a half, Frankie Johnson, a man incarcerated at the William E Donaldson prison outside Birmingham, Alabama, says he was stabbed around 20 times.

In December of 2019, Johnson says, he was stabbed “at least nine times” in his housing unit. In March of 2020, an officer handcuffed him to a desk following a group therapy meeting, and left the unit, after which another prisoner came in and stabbed him five times.

In November of the same year, Johnson says, he was handcuffed by an officer and brought to the prison yard, where another prisoner attacked him with an ice pick, stabbing him “five to six times” as two correctional officers looked on. According to Johnson, one of the officers had actually encouraged his attacker to carry out the assault in retaliation for a previous argument between Johnson and the officer.

Interface text:
AIE
UK  US politics  World  Climate crisis  Middle East  Ukraine  Football  Newsletters  Business  Environment  UK politics  Science  Tech  Global development
Share
AWS

[2:07](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=127s) [A question about the Chicago Sun Times alongside an article about AI-generated false legal citations.](https://assets.ai.engineer/sites/aiecode2025/sha256/f586f519423aed92fa4ad68adab7de1307cc5276bd2ec13f8d6b91054c3b51c1.webp)

Am I worried about the Chicago Sun Times sending their lawyers?

Artificial intelligence (AI)

Alabama paid a law firm millions to defend its prisons. It used AI and turned in fake citations

Butler Snow faces sanctions after lawyer cites false case law defending against inmate who says he was stabbed 20 times

Nicholas Chrastil in Birmingham, Alabama
Sat 24 May 2025 11.00 EDT

HUGO L. BLACK
UNITED STATES COURTHOUSE

The Hugo L Black US courthouse in Birmingham, Alabama, in 2023. Photograph: Vasha Hunt/AP

In less than a year and a half, Frankie Johnson, a man incarcerated at the William E Donaldson prison outside Birmingham, Alabama, says he was stabbed around 20 times.

In December of 2019, Johnson says, he was stabbed “at least nine times” in his housing unit. In March of 2020, an officer handcuffed him to a desk following a group therapy meeting, and left the unit, after which another prisoner came in and stabbed him five times.

In November of the same year, Johnson says, he was handcuffed by an officer and brought to the prison yard, where another prisoner attacked him with an ice pick, stabbing him “five to six times” as two correctional officers looked on. According to Johnson, one of the officers had actually encouraged his attacker to carry out the assault in retaliation for a previous argument between Johnson and the officer.

Interface text:
AIE
UK  US politics  World  Climate crisis  Middle East  Ukraine  Football  Newsletters  Business  Environment  UK politics  Science  Tech  Global development
Share
Microsoft
smol ai

### Detecting problems with AI is hard

The speaker calls AI nondeterministic: unlike a simple unit test, a fixed input cannot reliably be checked against one expected output.

[2:31](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=151s) [A red arrow points toward the statement that detecting problems with AI is hard.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ccf82ee26e53ebb3145c392f84edf5d0a29fc7c51912f8d9a0fc7890fa58d1d.webp)

Detecting problems with AI is hard

2025 | 6

Interface text:
AIE
Microsoft
smol ai
AI Engineer World's Fair

### Detecting problems with AI is hard

AI’s nondeterminism makes problems harder to detect: unlike an add function, an AI system cannot be checked against a single expected output for every input.

[2:38](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=158s) [A red arrow points toward the statement that detecting problems with AI is hard.](https://assets.ai.engineer/sites/aiecode2025/sha256/44ea61ed60b2afb6ae57648d9b619d49bad2f0e7e7b42b5320f049b6a2e2a7ec.webp)

Detecting problems with AI is hard

2025 | 6

Interface text:
AIE
AI Engineer World's Fair
aws

[2:43](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=163s) [Slide stating that detecting problems with AI is hard, alongside a red arrow-like symbol.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ed251f216a72b78fb2c8f17664196f3d16f9672ec869c0e05c1629b944c966e.webp)

Detecting problems with AI is hard

2025 | 6

Interface text:
AIE
AI Engineer World's Fair

## Full transcript

[0:00](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=0s) [upbeat music] So I'm here to talk about taming rogue AI agents, but essentially I wanna talk about, uh, evaluation-driven development, observability-driven development, but really why we need observability.

[0:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=25s) So who uses AI? Is that, Jim, stup- most stupid question of the day? Probably. [laughs] Who trusts AI? [laughs]

[0:36](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=36s) Right. If you'd like to meet me afters, I've got some snake oil you might be interested in buying. [laughs] Yeah, we do not trust AI in the slightest. Now, different question, who reads books?

[0:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=48s) Who loves reading books? If you want some recommendations for books, the Chicago Sun-Times recently published this list of books that, uh, you could enjoy over the summer. Atonement by Ian McEwan, great book, a fantastic movie with Keira Knightley.

[1:03](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=63s) And then, ooh, The Last Algorithm by Andy Weir. That sounds fun. Who watched The Martian? Yeah? But do you wanna read Andy Weir's new book, The Last Algorithm?

[1:11](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=71s) Well, you can't. Doesn't exist. So this is a news site, newspaper, and they had an outside contractor generate this summer reading list, and this contractor used AI. And it hallucinated worse than a 1970s, uh, hippie music festival.

[1:30](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=90s) A lot of hallucinations going on there. Now, they actually had to publish an article saying, "Sorry, we mucked up." But this happens. Now, we're supposed to trust the news.

[1:42](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=102s) You know, I'm sure we can all... we'll have opinions on that, but we're generally supposed to trust the news. But yet we can't if it's using AI to generate this kind of content.

[1:50](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=110s) Now, am I worried that the Chicago Sun-Times is gonna sue me for saying that they made this stuff up? No, because lawyers are using AI for case law. This is a, this is a recent case where Butler Snow, uh, cited false case law, uh, defending at the Alabama prison system.

[2:08](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=128s) Score one against the prisons. Now, the r- I picked these two examples. They're from, like, the same, pretty much the same week a couple of weeks ago, but we've all seen these examples, haven't we?

[2:16](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=136s) We've all seen Air Canada's chatbot says you can get a refund, and that they're legally obliged to provide things like that. And so we understand that AI has this problem, that it makes stuff up.

[2:28](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=148s) You know, ev- almost like every day in the news, it's another story about how AI has broken something. And the problem we have is detecting problems with the AI is hard.

[2:37](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=157s) It is a nondeterministic problem. Right, who's a coder? Who writes code? Okay. Who writes unit tests?

[2:45](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=165s) Is that the same number of hands? I'm not sure it is. [laughs] You're bad people. But yes, a unit test is kind of easy to write. You know, I have an add function.

[2:53](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=173s) I can say, "Add two and two, do I get four? Add three and three, do I get six?" But I can't do that for an AI. I can't say, "If I put this input into my AI system, will it give this output?"

[3:06](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=186s) At the most basic, if I ask a single question, I can possibly look for keywords. But if I've got a complex agentic workflow, I have an application, and I...

[3:15](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=195s) the input comes in, it calls an LLM. That LLM gets data, makes a decision, calls an agent, gets data, makes a decision, calls a tool, gets data, and so on and so on and so on.

[3:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=205s) That is really, really hard for me to actually evaluate. It's really hard for me to say, "Did it work?" 'Cause partly, what does even work mean? You know, especially with things like a chatbot where we're having a human conversation, how do we define what does work mean?

[3:39](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=219s) And this is the problem that we face. So how do we do it? There's an old, I believe it was a [REDACTED:origin] expression. Any [REDACTED:origin] in the room other than me?

[3:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=228s) Yay, lots of cool people in the room. We like it. Uh, there's an old [REDACTED:origin] expression called set a thief to catch a thief, and the idea with that expression is if you wanna know how a thief works, you set a thief to do it.

[3:59](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=239s) But yeah, you want... the thief understands the thief and knows how to catch it, and we can kind of apply that logic to AI. We can set an AI to verify an AI.

[4:08](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=248s) We can actually ask a nondeterministic system like an AI to evaluate an AI for us, and it turns out AIs are not bad at this. They're about as good as a human is at determining whether an AI actually worked.

[4:21](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=261s) And that opens up this whole new world of things we can do in that we can use AI to evaluate is our AI application actually working.

[4:30](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=270s) So I got a demo here. I'm not gonna do this demo live because conference Wi-Fi. Have you all had fun with Wi-Fi? [laughs]

[4:37](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=277s) Yes. [REDACTED:gender] at the back there very kindly managed to get me connected to an actual physical cable, so things are great. Um, so I got this chatbot.

[4:46](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=286s) The concept you just said that set a thief to catch a thief- Yep ... using that in AI, but isn't that building a not so trustworthy loop because- Great question.

[4:56](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=296s) We will be getting to that. Good question, though. Um, so here's an example here. This is not, say, live because, well, one, the Wi-Fi, and two, this is an AI application.

[5:06](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=306s) There's no guarantee it's actually gonna break the way I want it to break when I'm demoing it to you because it's nondeterministic. But this is a basic chatbot conversation.

[5:13](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=313s) Um, I've actually got this demo on my laptop. If you wanna come and see this in action, come to Galileo booth and I- I'll demo it. But I'm basically asking it, "What is my account balance?"

[5:21](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=321s) Think about a kind of fintech chatbot. "What's my account balance?" And the response is, "I don't have access to your account information."

[5:28](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=328s) It's not very helpful. It's kind of true. I don't have access. Not very helpful. You know, ideally I want it to say, "Yes, you've got a million dollars," or whatever it is.

[5:36](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=336s) Um, I don't. If anyone wants to donate, would appreciate that. But then I had a follow-up question. You know, "What is the balance of my checking account?" I'm now giving it more information.

[5:45](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=345s) And I was hoping when I, when I did this demo, it would come back and say, "You've got X amount of money." Instead it came back to say, "Please could you let me know the name of your checking account?"

[5:53](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=353s) So I didn't even know what my bot was gonna do as I was working through it. It's asking me questions. And I responded to say, "It's called checking account."

[6:02](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=362s) You know, four hard things in computer science, naming things, cache invalidation, off-by-one errors. Yeah.

[6:09](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=369s) Someone got the joke. Cool. Um, but yeah, so I got a checking account, and now is able to go to call a tool and go and look at the checking account.

[6:16](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=376s) Now, did this AI work? Did it work? What do we think? Who thinks, hands up if you think, if you think the whole AI chatbot worked.

[6:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=385s) I mean, yes. You, you, you're right. It did work because within a few steps I got my account balance. Who thinks it didn't work?

[6:33](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=393s) More hands. Yes, you're right. It didn't work because it took me three steps to get my account balance. So it's not a good thing. So I need to think about, How can I evaluate this?

[6:42](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=402s) And this is where evaluations comes in. We need to look at all the different steps in a flow and look at different metrics to measure how well this did.

[6:51](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=411s) So when we think about these kind of evaluations, essentially what we have to is we have to take a lot of data, we have to take everything that's coming in, and we have to define at all the different steps in the process what things we want to look for.

[7:04](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=424s) Did it successfully call tools? Is it retrieving the right information from a RAG system? Um, is it actually giving an answer that makes sense? Is it hallucinating? There's a lot of different metrics that you can define that evaluates whether or not the whole thing was successful.

[7:19](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=439s) And ideally, you wanna break that down by all the steps in your flow. I have a multi-agent app. When I call my app to get my account balance, there's an agent that orchestrates it, that calls another agent, that calls a tool, and I need to look at that breakdown by all the individual steps and measure where that--

[7:37](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=457s) the failures happen. And we need to be able to do these evaluations at every single component. It's not just that binary, "Did my agent work?" yes or no question, it's, at what step in the process did my agent fail?

[7:49](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=469s) So I have to get this level of granularity. That is really, really important that we have granularity when we're looking at these things. And then the way we work out these numbers, as I said, we set the thief to catch the thief.

[7:59](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=479s) We use an LLM or using multiple calls to an LLM to evaluate the, the metric. We say to an LLM, "With this input and this information from a RAG system, this is the output that came out, score it."

[8:15](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=495s) And the idea is you use a better LLM to score than the LLM you use in your application. In your main application, you want the cheapest LLM possible because we all like making money.

[8:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=505s) If you don't like making money, send it to me. But we all-- We want the cheapest LLM possible, but we wanna use the best LLM possible to do the evaluations.

[8:32](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=512s) Going back to your question there, ideally, you wanna use a better LLM to actually do these evaluations. You wanna say, you get like a million traces a day, we're gonna test, say, ten thousand of them using expensive LLM to, to prove that it works.

[8:46](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=526s) Ideally, you wanna use a custom-trained LLM. Um, something Galileo offers is we have a custom-trained LLM that's, small language model that's designed to be really, really good at evaluations.

[8:55](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=535s) Um, but the idea is you use this LLM to do it with a well-defined set of prompts to extract this information. And then you make-- This is in your workflows.

[9:03](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=543s) And you do this right from day one. So who is just starting building apps? Anyone who's just started building apps? Few hands. Who's got an app in production? Okay.

[9:12](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=552s) All of you need evaluations, like, now. The, the, the, the best time to put evaluations in is as you're doing prompt engineering model selection. The second best time is now.

[9:22](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=562s) So you wanna think about this right from the get-go. As you're building application, you wanna start adding those evaluations when you're doing your initial prompt engineering, when you're doing your model selection.

[9:31](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=571s) You wanna keep those in your dev cycle, in your CI/CD pipelines, and then you wanna observe these in production as users start throwing garbage at your system. So let's look at, look at a couple.

[9:39](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=579s) Here's just a whole lot of traces from that chatbot with some nice red and green numbers. And I wanna highlight these three rows, and these rows match what I was trying to do with the chatbot.

[9:51](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=591s) Okay, so the first row here we have got, "What is my account balance?" "I'm sorry. I don't have access to my account information." And I've got two metrics here, action, uh, completion and action advancement.

[10:03](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=603s) Action completion is, did it actually do the thing it was asked to do? So it measures across the whole flow from the input to the output, did it actually complete the task that it was asked to do?

[10:14](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=614s) Action advancement is, did it move forward towards the end goal? They're two very subtly distinct metrics. Now, in the case of the first one, "What's my account balance?" "Uh, I don't know.

[10:27](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=627s) Don't know anything." Didn't complete, didn't advance. So we know there's a problem with that one. Second one, "What is the balance of my checking account?" Didn't complete. "I don't have a balance," but it advanced.

[10:39](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=639s) So I can see that, yes, it realized that it needs to know the name of the account, so it advanced one step further. So I can say, actually, yes, with this kind of prompt, it advances.

[10:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=648s) And then finally, when I say, "Yes, my checking account is called checking account," it completed, gave me the results, and it could show the advance went through. So I can see from these metrics which prompts worked, which prompts didn't work.

[10:59](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=659s) And I can use this to continue improve what I'm doing. Now, obviously, these numbers are kind of a whole over the, over- overarching number across the whole thing. Obviously, I kinda need to have some form of breakdown.

[11:11](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=671s) So that's what I've got here. This is the individual trace that comes in to call LLM. The LLM decides to call a tool, pulls data, decides to call the LLM to process that data and show it out the other side.

[11:23](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=683s) And that's showing those steps. And at each level, I can get whatever metrics are relevant. So it's not-- I can look at the overarching, it's red, it's a bad thing 'cause it's red, and then I can dive into each individual step and see why it's red and look at all those different layers.

[11:39](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=699s) And that's really, really important. You have to have this understanding of the architecture of your agentic systems so that you can do this analysis at each individual level. And then depending on what's happening, you can then farm out the f- the fixing of the problem to the relevant team.

[11:54](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=714s) Maybe it's your RAG application is terrible. Maybe you need to tune one of your prompts. But by having this level of granularity, you can make those smart decisions around it.

[12:05](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=725s) Now, what's also cool is this is a lot of unstructured data. What do we know that is good for working with unstructured data?

[12:14](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=734s) AI. Yes. And so what's cool as well is when you start putting an LLM over the top of this, you can get some really smart insights coming out. So this is, um, some insights that I generated.

[12:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=745s) Basically, an AI will go against all the data and say, "This metric is low. How can I make it better?" And this is really cool, and this is saying, yeah, the LLM occasionally fails to use the Get Balance tool when asked about account balances, and that's basically the fundamental problem.

[12:40](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=760s) When I say, "What is the balance of my account," what would you expect to happen in a chatbot? What's the balance of my account? What would you expect? Anyone?

[12:48](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=768s) Shout out. [laughs] To get your balance. Exactly, yeah. And probably if you have multiple accounts, you would get the balance of all your accounts. You know, this is your checking, savings, credit card, 401K, whatever.

[13:00](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=780s) And so it kind of makes sense to improve the effectiveness to get us closer to where we all have a consensus that this agent is working, would be for, if I say give me the account balance,

[13:11](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=791s) it goes to all the accounts and shows me all the balances. So the suggested action here is adding ex- explicit instructions to my system message. So not only have I identified this problem through my evaluations, but I've got a suggestion for fixing it.

[13:26](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=806s) Now, it's not automatically gonna fix it for me because, you know, there be dragons in that, 'cause what if that mucks up, and then you have to evaluate my automatic fixings and my evaluations, and the snake swallows its tail.

[13:36](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=816s) But this is giving me suggestions, so the human in the loop, and that's really important, as a human, I can look at this and say, "Yeah, this is the fix that I want to make."

[13:46](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=826s) Now, I do want to emphasize that whole human in the loop thing is really, really important. So when you're, when you're generating metrics, there's no guarantee the metrics you generate are actually gonna be correct, 'cause the, the AI, going back to your question over there, the AI could get it wrong.

[14:01](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=841s) And so one thing you wanna do is make sure that you're using a system that has human feedback, like CLHF, continuous learning by human feedback. You can ... You want humans to evaluate the numbers and say, "Okay, this is actually working.

[14:13](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=853s) The metric was low. Here's the reason. Retune." And have that continuous training of your metrics, 'cause your metrics will never be perfect out the box. You need this continuous level of training.

[14:25](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=865s) So to get this all right, what do we have to do?

[14:29](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=869s) Step one, add evaluations to your agent. As I said, the best time to do it is before you even start. The second-best time is now. If you don't have evaluations, get them in now so you can make sure your agent is not making stuff up.

[14:41](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=881s) You do not want to be the next Chicago Sun-Times.

[14:45](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=885s) Then you need to measure precisely what you need. Different tools, different applications have different measurements of what they need. Do I need to measure whether the input and outputs are being toxic?

[14:55](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=895s) Do I need to measure for the c- the hallucinations? Do I need to measure for a comprehensible output? Do I need to measure for RAG? Do I have some kind of custom measurement that only I know about that's specific to my use case?

[15:08](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=908s) You wanna be defining those measurements and those metrics upfront as you are thinking about your prompts, your structure of your app, your agents. Where right at design time you think about exactly what you need to measure.

[15:20](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=920s) And then as you build it, keep that going all the way through to production. This is not just a test in dev. 'Cause let's be honest, when users get a hold of your system, they do stuff you don't expect.

[15:29](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=929s) How many times have you tested something to the nth degree and it breaks the second a user gets on it? Damn those users. But they do things you don't expect, and so you have to have this in production as well to make sure that you've got everything in place.

[15:42](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=942s) And then you wanna have this real-time prevention. You wanna have alerting when it goes wrong. If your AI agent goes rogue, maybe you need to be woken up. So that is how you can tame AI agents with evaluations.

[15:52](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=952s) I'm Jim Bennett. I'm a principal developer advocate at Galileo. Come and talk to us on the booth in the expo if you wanna learn more. Uh, scan that if you wanna sign up for Galileo.

[16:00](https://www.youtube.com/watch?v=xJXm4Wcw4m8&t=960s) We have a free offering, but you wanna learn more about it, come meet me at the booth. With that, thank you very much, and I will take some questions, I believe. [clapping] [upbeat music]
