# Why Your Agent Disagrees With Itself (And What To Do About It)

Diane Lin, Datadog

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/wEc9aG7cRQc-why-your-agent-disagrees-itself-what-do) · [Video](https://www.youtube.com/watch?v=wEc9aG7cRQc)

## Why Your Agent Disagrees With Itself (And What To Do About It)

Repeated verdict changes can expose missing policy or evidence. Active learning, semantic memory, and episodic memory turn that disagreement into targeted feedback.

Familiarity with LLM classification, model evaluation, and the distinction between training and inference will help.

### The same input, a different decision

[0:19](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=19s)

What should you do when the same model receives the same input and changes its decision? Diane Lin approaches this problem through both learning research and security operations. Her background spans continual learning at Imperial College London, one-shot learning with Josh Tenenbaum at MIT, and work as one of the first three applied scientists on Alexa’s question-answering team. She later researched zero-shot transfer at Vicarious before spending five years applying machine learning to cybersecurity at Zscaler. That experience led her to co-found Culminate to automate security-alert triage. At the time of the recording, she reports that Datadog had acquired Culminate earlier that year and that she was leading development of self-evolving agents. The practical problem is **semantic inconsistency**: different verdicts, not merely different wording.

![Background slide showing education and research affiliations alongside a career timeline through Amazon Alexa, Vicarious, Zscaler, Culminate and Datadog.](https://ai.engineer/api/talk-essay-media/748945cd28e48f24e1a16f0234ae85931c8b412666cf26154d5aed222103d90d)

Diane Lin’s research and industry background.

Consider a hotel-review classifier whose only permitted labels are `positive` and `negative`. Today's LLMs can perform this familiar task well, yet repeated runs of the same review through the same model can occasionally produce opposite labels. Stochastic generation is an obvious initial explanation, but it does not tell you which inputs are vulnerable or how to improve them. A single evaluation run therefore gives an incomplete picture: repeat the evaluation and average its results across runs.

In security operations, the same behavior creates a more consequential problem. An analyst must decide whether an alert describes malicious activity requiring intervention or a benign false alarm that can be ignored. Lin’s example is a failed login to a Gmail account from a suspicious IP. Across repeated agent runs, some alerts remain consistently benign, others remain consistently suspicious, and a third group flips between the two.

A customer facing those changing verdicts must decide which result to trust. In a proof-of-concept vendor comparison, an agent that repeatedly changes its answer can lose credibility against one that gives stable decisions. The engineering question becomes more specific: **where does the disagreement concentrate, and what information would resolve it?**

### The gray zone needs a policy and the right evidence

[6:21](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=381s)

Lin observes that unstable examples tend to sit near the **decision boundary**, in a gray zone between classes. An obviously enthusiastic hotel review remains positive across runs. A more ambivalent review can support either interpretation: some wording sounds mildly positive, while other wording sounds negative. Even human experts may disagree.

The missing ingredient can be the hotel’s purpose for assigning labels. If a complaint concerns an experience the hotel cannot improve, one hotel may see no reason to classify it as negative. Another may want that dissatisfaction captured so it can pursue improvements. The label is partly an operational preference, not simply a property of the sentence. Until that preference is explicit, asking for a binary answer leaves the boundary underspecified.

Security triage adds another distinction: an attacker trying to enter an environment is different from an attacker who has entered it. A few blocked attempts may be routine noise for an enterprise that constantly receives such activity. Escalating every attempt could flood the alert queue. But if an attacker guesses the password, passes MFA, and gains access, the response changes substantially. Similar initial behavior can lead to very different outcomes.

Two questions must therefore be resolved: does the customer want notification about blocked attempts, and is there evidence that access succeeded? The first supplies a policy; the second supplies discriminating information. Lin’s broader diagnosis is that these cases also trouble human experts and traditional classifiers. The agent may be exposing ambiguity that already exists in the task. Finding and resolving that ambiguity is the next step.

![Red and blue circles lie on either side of a dashed diagonal boundary, with overlapping red and blue circles inside a shaded band labeled Gray Zone.](https://ai.engineer/api/talk-essay-media/42918e7e3bac5a40ae84d53775bc538832d83c7c97565fef1f46753a385d9fc3)

Ambiguous data points sit in a gray zone near the decision boundary.

### Spend review effort where the model can learn

[10:59](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=659s)

**Active learning** addresses a practical constraint: you have many predictions to assess and too little human attention to inspect them all. An initial model might run in production in monitor mode, producing predictions on unlabeled data. The selection strategy identifies cases likely to be problematic and directs review toward them.

The traditional cycle proceeds as follows:

1. Train an initial model and predict labels for an unlabeled dataset, potentially drawn from production.
2. Select informative examples. In binary classification, uncertainty sampling can prioritize probabilities near `0.5`. A committee can instead prioritize examples on which competing models disagree.
3. Ask a human to inspect the selected examples, correct labels, and determine whether additional information is needed.
4. Add the reviewed labels—and any new features—to the next training round, retrain, and repeat.

The value comes from concentrating effort on cases that teach the model something, rather than reviewing every prediction equally.

For LLM agents, Lin changes two parts of that loop: selection and learning. In her team’s experience, model uncertainty scores were unreliable; confidence did not establish correctness. Disagreement across models or repeated runs provided a more useful signal for requesting human guidance. Repeated sampling of one LLM is her adaptation of the committee idea, rather than a requirement to maintain several separately trained models. Reviewers then label the disputed examples and provide feedback.

A minimal selection function can operate on normalized verdicts rather than generated explanations. Here, the failed-Gmail-login alert is selected because its repeated verdicts differ; a consistently benign alert is left out of this disagreement queue.

```python
from collections.abc import Mapping, Sequence


def select_disagreements(
    verdicts_by_alert: Mapping[str, Sequence[str]],
) -> list[str]:
    return [
        alert_id
        for alert_id, verdicts in verdicts_by_alert.items()
        if len(set(verdicts)) > 1
    ]


verdicts_by_alert = {
    "failed-gmail-login": ["benign", "suspicious", "benign"],
    "stable-alert": ["benign", "benign", "benign"],
}
review_queue = select_disagreements(verdicts_by_alert)
```

This selects a place to investigate; it does not decide which verdict is correct.

After review, fine-tuning remains an option, but Lin describes it as expensive. Her lighter-weight alternative is to augment the agent with **semantic memory** and **episodic memory**, allowing feedback to influence later decisions without requiring a new training cycle for every update.

### Turn ambiguity into explicit domain knowledge

[16:21](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=981s)

Semantic memory stores knowledge that explains how to decide. Return to the hotel review: if the missing information is company preference, add that preference to the knowledge base. Lin’s example policy classifies complaints about matters outside the hotel’s control as positive because the hotel does not want those complaints to demand attention. The rule supplies the operational meaning that the binary labels lacked.

For the security example, the key distinction is whether authentication ultimately succeeds. [Password spraying](https://attack.mitre.org/techniques/T1110/003/) specifically tries one or a small set of passwords across many accounts. Lin’s example customer policy treats a spray without a successful login as benign for triage, and one with a final successful login as malicious. Here, `benign` expresses the customer’s response policy; failed attempts can still be evidence of an attack.

| Evidence in the example | Customer’s triage label |
| --- | --- |
| Password spray, no successful login | `benign` |
| Password spray, final successful login | `malicious` |

The rule identifies the evidence needed to separate two superficially similar cases. Storing it as domain knowledge sharpens the agent’s decision boundary and gives human labelers the same explicit basis for making consistent decisions.

![Semantic Memory: Domain Knowledge slide with a security alert example and a highlighted rule stating that password spray without successful login should be benign, while password spray with a final successful login should be considered malicious.](https://ai.engineer/api/talk-essay-media/5edbd9889214f99acbe03c0c461b5f24b20d70815d9f9a5af3ba9821425fa0cd)

An explicit security rule distinguishes password sprays by whether login succeeds.

### Reuse past decisions before asking for another review

[18:02](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1082s)

Sometimes a human has already labeled a case but has not distilled its reasoning into a reusable rule. **Episodic memory** preserves the case and its decision. When a similar case arrives, the agent can retrieve the earlier example and use its label as a reference. The human intervention happened previously, so online decisions need not wait for someone to write new semantic knowledge.

That mechanism is especially useful for recurring security false positives. Similar noise appears repeatedly, making past reviewed cases valuable references. A new case that falls between familiar groups, however, may still require human attention. Automation handles the recurring cases and leaves review capacity for the examples memory cannot resolve.

| Memory | What it preserves | How it helps |
| --- | --- | --- |
| Episodic | Similar cases and past labels | Reuses earlier decisions |
| Semantic | Explicit policies and domain rules | Explains the decision boundary |

The two forms are complementary. Use episodic memory for recurring situations; if no useful reference exists, or the agent still disagrees after consulting one, send the case to a human. The reviewer can then distill the missing domain knowledge into semantic memory for future decisions.

### What the feedback loop changes

[20:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1238s)

Lin connects this loop to three operational benefits:

- **Consistency:** Explicit knowledge and relevant past decisions help stabilize cases near the boundary, supporting user trust.
- **Efficient quality control:** Reviewers inspect a selected subset of outputs and provide missing information or clearer labels where that effort can be most useful.
- **Customer adaptation:** Accumulated feedback teaches the agent how a particular customer wants its environment handled.

The review process becomes a way to improve the product while using it, rather than a separate exercise in checking every answer.

Lin reports evaluating 93 cybersecurity alerts three times each. Approximately 25% changed verdict without the proposed solution; approximately 10% remained inconsistent after episodic memory. She describes roughly 15% becoming consistent, a breakdown consistent with an approximately 15-percentage-point reduction across the alert set, although exact counts are not supplied. The measured outcome is repeated-run verdict consistency, not correctness.

The remaining cases either lacked a similar past reference or still produced disagreement after consulting one. Those cases went to human review for additional knowledge to disambiguate the decision and adapt it to the customer’s environment. Lin does not report a numerical consistency or accuracy result after that review.

The practical response to a changing verdict is therefore to investigate the label and the available evidence before assuming the model itself is the problem. Disagreement points to a place where the system can learn. Fine-tuning is one option, but explicit policies and reusable reviewed cases offer another path: resolve what the agent does not know, preserve that resolution, and let subsequent decisions reflect the customer’s environment.

## Resources

- [Active Learning Literature Survey](https://burrsettles.com/pub/settles.activelearning.pdf) — Burr Settles’s survey explains active-learning selection strategies, including uncertainty sampling and query by committee.
- [MITRE ATT&CK: Password Spraying](https://attack.mitre.org/techniques/T1110/003/) — Defines password spraying and describes detection signals and mitigations for attacks across multiple accounts.
- [Bits Security Analyst overview](https://www.datadoghq.com/blog/bits-ai-security-analyst/) — Datadog’s overview of automated Cloud SIEM investigations, supporting evidence, verdict recommendations, and activation options.

## Slides

### Why Your AI Agent Disagrees With Itself and What to Do About It

Dianhuan (Diane) Lin introduces a session on AI agent inconsistency and what to do about it.

[0:00](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=0s) [Title slide for Dianhuan (Diane) Lin’s talk on AI agent inconsistency.](https://assets.ai.engineer/sites/aiecode2025/sha256/efb09307eb17b341224400c354a35d58a322fc4fa54e2efacfb227ee0db5b3f5.webp)

Why Your AI Agent
Disagrees With Itself
and What to Do About It

Dianhuan (Diane) Lin

DATADOG

### Background

Diane Lin moved from machine learning research at Imperial and MIT through Alexa and Vicarious to cybersecurity at Zscaler. She then co-founded Culminate, which Datadog acquired.

[0:18](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=18s) [Timeline of the speaker’s education and professional background.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f069cafee5b6a755fbeebbb4152c7639a0b1b631af362057ad9dc5f6f373162.webp)

Background

Education & Impact
• PhD in Machine Learning, Imperial College London
• Research affiliations at MIT's CSAIL

Amazon (Alexa Team)
Founding applied scientist

Vicarious
(acquired by Google DeepMind)
Senior Researcher

Zscaler
Director of Machine Learning
zscaler

Culminate
(acquired by Datadog)
CTO & Co-founder
CULMINATE

Datadog
Tech Lead
Self-Evolving AI Agents for the Modern SOC

### Agenda

The talk moves from LLM inconsistency in practice to its sources, solution trade-offs, and experimental results.

[2:12](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=132s) [Agenda covering inconsistency in practice, its sources, solutions, and experimental results.](https://assets.ai.engineer/sites/aiecode2025/sha256/b93c86f149d15b4d77770ccf8eaeaa7e8c23e404a585eb611ef078b5c8072c0d.webp)

Agenda

• The Problem: LLM inconsistency in practice
• Where Does Inconsistency Come From?
• Solutions
• Experimental Results

DATADOG

### LLM Inconsistency in Practice: Sentiment Analysis

A glowing hotel review sits beside a mixed one. The speaker says repeated runs of the same model can yield different verdicts, so a single evaluation run may miss that variation.

[3:08](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=188s) [Two hotel reviews illustrate a positive-or-negative sentiment classification task.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e9346f77e1081763f1bbe4ea9eb2363a292dfd43fa7d37f70bbcf92f9df461e.webp)

LLM Inconsistency in Practice

Sentiment analysis: classify hotel reviews as positive or negative

Review A
“Fantastic hotel in the perfect location. It's just a five-minute walk to all the major museums and restaurants, yet tucked away on a quiet street so you don't hear any city noise at night. The concierge, David, gave us the best recommendations for local dining that weren't tourist traps. Everything about this place is top-notch. Highly recommend to anyone visiting the city.”

Review B
“It's an okay hotel. It's clean and safe, which are the most important things, but it entirely lacks any charm or personality. The check-in process was automated via a kiosk, so we never actually spoke to a human being during our entire stay. The room had all the basics but felt very sterile, almost like a hospital room. If you just need a cheap, clean place to crash before an early flight, it works fine. Just don't expect a memorable hospitality experience.”

DATADOG
5

### LLM Inconsistency in Practice: Security Alerts

A failed Gmail login from a suspicious IP illustrates security alert triage. The speaker says an AI agent can classify similar alerts consistently across runs, while other cases flip between benign and suspicious.

[4:21](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=261s) [A security alert about a failed login from a suspicious IP introduces a triage example.](https://assets.ai.engineer/sites/aiecode2025/sha256/22ae9b1fb784fedef84097f12ef684b5a593a91e74d9c90ad9b219864f13de0b.webp)

LLM Inconsistency in Practice

Triage cyber security alerts:
"Failed login attempt to gmail account detected from suspicious IP”

DATADOG
6

### Gray Zone: Sentiment Analysis

Review A is clearly positive; Review B mixes cleanliness and safety with a sterile, impersonal stay. The speaker says Review B’s label can flip and may depend on a hotel’s preferences.

[6:34](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=394s) [Two hotel reviews contrast clearly positive and mixed sentiment.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f8a5e371e1121806151daa33f7f85da1904f542f1d27cec388a6f9139f557cf.webp)

Gray zone: data that sit near the decision boundary

Sentiment analysis: classify hotel reviews as positive or negative

Review A

"Fantastic hotel in the perfect location. It's just a five-minute walk to all the major museums and restaurants, yet tucked away on a quiet street so you don't hear any city noise at night. The concierge, David, gave us the best recommendations for local dining that weren't tourist traps. Everything about this place is top-notch. Highly recommend to anyone visiting the city."

Review B

"It's an okay hotel. It's clean and safe, which are the most important things, but it entirely lacks any charm or personality. The check-in process was automated via a kiosk, so we never actually spoke to a human being during our entire stay. The room had all the basics but felt very sterile, almost like a hospital room. If you just need a cheap, clean place to crash before an early flight, it works fine. Just don't expect a memorable hospitality experience."

DATADOG 8

### Gray Zone: Decision Boundary

Red and blue points separate across a dashed boundary. Points in the gray zone can flip between classifications and need more clarity to disambiguate.

[9:51](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=591s) [Red and blue circles lie on either side of a dashed boundary, with overlapping points in a gray zone.](https://assets.ai.engineer/sites/aiecode2025/sha256/6cab4d3cf704061273d265d3e53671f52dfd66e2b5bd29136408c137a456a96b.webp)

Gray zone: data that sit near the decision boundary

Gray Zone

DATADOG
10

### Semantic Memory: Hotel Review Examples

Review A is enthusiastic. Review B praises cleanliness and safety but criticizes the hotel's sterile, impersonal experience, making its sentiment harder to classify.

[16:12](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=972s) [Two hotel reviews illustrate a sentiment classification task.](https://assets.ai.engineer/sites/aiecode2025/sha256/c69135509f7a3e2ed0d4048fb791d91ca91714c1490e7582987d9ce5e2a4a25c.webp)

Semantic Memory: Domain Knowledge

Sentiment analysis: classify hotel reviews as positive or negative

Review A

"Fantastic hotel in the perfect location. It's just a five-minute walk to all the major museums and restaurants, yet tucked away on a quiet street so you don't hear any city noise at night. The concierge, David, gave us the best recommendations for local dining that weren't tourist traps. Everything about this place is top-notch. Highly recommend to anyone visiting the city."

Review B

"It's an okay hotel. It's clean and safe, which are the most important things, but it entirely lacks any charm or personality. The check-in process was automated via a kiosk, so we never actually spoke to a human being during our entire stay. The room had all the basics but felt very sterile, almost like a hospital room. If you just need a cheap, clean place to crash before an early flight, it works fine. Just don't expect a memorable hospitality experience."

DATADOG
5

### Semantic Memory: Hotel Classification Policy

A company can add its sentiment policy to the knowledge base: classify a review as positive when the complaint concerns something outside the hotel's control.

[16:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=998s) [Hotel reviews accompany a policy for classifying complaints about matters outside the hotel's control.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb7783b40ff1f02b1280c6296ac55e9f715b02705511c00cbe02d60b24040be3.webp)

Semantic Memory: Domain Knowledge

Sentiment analysis: classify hotel reviews as positive or negative

Review A
“Fantastic hotel in the perfect location. It's just a five-minute walk to all the major museums and restaurants, yet tucked away on a quiet street so you don't hear any city noise at night. The concierge, David, gave us the best recommendations for local dining that weren't tourist traps. Everything about this place is top-notch. Highly recommend to anyone visiting the city.”

Review B
“It's an okay hotel. It's clean and safe, which are the most important things, but it entirely lacks any charm or personality. The check-in process was automated via a kiosk, so we never actually spoke to a human being during our entire stay. The room had all the basics but felt very sterile, almost like a hospital room. If you just need a cheap, clean place to crash before an early flight, it works fine. Just don't expect a memorable hospitality experience.”

“If the customer complains about something outside the hotel's control, classify the review as positive.”

DATADOG
15

### Semantic Memory: Security Alert Triage

In this triage example, the speaker's rule labels password spraying benign without a successful login and malicious when the final login succeeds. The login outcome helps distinguish the alerts.

[16:59](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1019s) [A security alert example contrasts password spraying with and without a successful login.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e52e9bd629ba615247f9ac4fedc739f4e248b4c754fa784b02811147e81d8d5.webp)

Semantic Memory: Domain Knowledge

Security Analysts triage alerts
“Failed authentication attempts detected from suspicious IP”

"Password spray without successful login should be benign. In contrast, password spray with final successful login should be considered malicious.”

DATADOG
18

### Semantic Memory: Domain Knowledge and Triage

A triage rule treats password sprays without a successful login as benign and those ending in a successful login as malicious, helping models and reviewers distinguish alerts consistently.

[17:47](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1067s) [A security alert example is followed by takeaways about decision boundaries and consistent human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/258d0c7102f39f156e9cf8486fb35185bea032744a89d56a5ce8a7fe45def0f3.webp)

Semantic Memory: Domain Knowledge

Security Analysts triage alerts
“Failed authentication attempts detected from suspicious IP”

"Password spray without successful login should be benign. In contrast, password spray with final successful login should be considered malicious.”

• Domain Knowledge Sharpens the Boundary
• Helps human reviewers make more consistent decisions too

DATADOG
16

### Episodic Memory: Past Similar Cases

Episodic memory compares a new case with previously labeled similar cases and uses their decisions as a reference. Recurring cases can be resolved automatically, leaving ambiguous cases for human review.

[18:01](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1081s) [An embedding-space diagram connects a new case to similar past cases.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3e95923645b9e5f2cf2883294d00ffa6e9c47b8c889825bb0cc4a4f7cb59637.webp)

Episodic Memory: Past Similar Cases

Embedding Space (Visualization)

New Case
Similar Past Cases (Benign)
Similar Past Cases (Malicious)
Other Cases

DATADOG
17

[19:23](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1163s) [Embedding-space diagram connects a new case to similar past cases; a callout says recurring cases can be resolved automatically while ambiguous cases receive human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/026b0cc4bdafd701e8b88ca3a90195e55262fd839b4c5c29928d7873ae1d4f8c.webp)

Episodic Memory: Past Similar Cases

Embedding Space (Visualization)

New Case
Similar Past Cases (Benign)
Similar Past Cases (Malicious)
Other Cases

Episodic memory resolves recurring cases automatically.
Humans review only the genuinely ambiguous cases.

DATADOG
17

### Semantic Memory vs Episodic Memory

Episodic memory handles recurring cases automatically. Cases that still flip or lack a past reference go to human review, which adds distilled domain knowledge to semantic memory for future use.

[20:00](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1200s) [Bullets explain that episodic memory handles recurring cases, remaining flip cases receive human review, and review findings contribute to semantic memory.](https://assets.ai.engineer/sites/aiecode2025/sha256/a3343e3d8f60ab67d2ea8078caa6f1b20533e7ba2d8c0382b8b727650b155a59.webp)

Semantic Memory vs Episodic Memory

Complement each other:
• Episodic memory handles recurring situations automatically
• Remaining flip cases => exactly the ones that deserve another human review
• Human review distill the domain knowledge to be put into Semantic memory

DATADOG
18

### Experimental Results

Across 93 cybersecurity alerts run three times, inconsistent verdicts fell from 25% to 10% with episodic memory. Human review addresses the remaining ambiguous cases.

[22:22](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1342s) [Results for 93 alerts run three times show 25% inconsistent verdicts before the solution and 10% remaining inconsistent with episodic memory; ambiguous cases receive human review.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7c2cb050c52daea92c5418bf93de3b70382bdeefe44cc28091440a8ed8e120d.webp)

Experimental Results

EXPERIMENT SETUP
93 ALERTS
3x RERUNS

BEFORE: 25% FLIP-FLOP VERDICTS
Inconsistent Verdicts
25%
25%
Consistent Verdicts

AFTER: SYSTEMIC IMPROVEMENT
EPISODIC MEMORY:
10% REMAINING INCONSISTENT
10%
90%

HUMAN REVIEW:
AMBIGUOUS CASES
10%

DATADOG
20

## Full transcript

[0:03](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=3s) Hello. This session is about why do your AI agents disagree with itself, and what to do about it. And this is Die Huang Lin, and you can call me Diane.

[0:19](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=19s) A little bit about my background. My journey with AGI started from my PhD at Imperial College London, where I was working on continual learning. Later, I worked with Professor Josh Tenenbaum at MIT on one-shot learning, meaning learning from one single examples.

[0:42](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=42s) Afterwards, I was lucky that Alexa was launching. I was the first three applied scientists on the question answering team at Alexa.

[0:53](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=53s) Later, attracted by a, a startup working on AGI at Silicon Valley called Vicarious. So today, Vicarious is part of the Google DeepMind. There, I got to work on the exciting frontier research on zero-shot transfer learning.

[1:13](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=73s) After working on AGI for a few years, I decided to pivot to the applied world at Zscaler, which is cybersecurity company.

[1:25](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=85s) There, I got to apply a variety of different machine learning model to address the cybersecurity challenge.

[1:34](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=94s) There, for five years, uh, I decide to co-found my own company because the pain point I have seen at Zscaler.

[1:44](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=104s) So Culminate is building AI agent to auto triage your security alerts

[1:50](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=110s) and which make your SOC more efficient. Uh, we are super proud that Culminate has been acquired by Datadog earlier this year. So now I'm part of the Datadog and leading the development of self-evolve agent.

[2:14](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=134s) So today, I'm going to share with you the problem you're probably seeing in your day-to-day building AI agent, the inconsistency.

[2:24](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=144s) And then we'll discuss about where this inconsistency come from in order to figure out the solutions.

[2:33](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=153s) And we will discuss a few trade-off among the different solutions, and then eventually we'll show you the experimental results to show how feasible they are.

[2:46](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=166s) You probably see this. Same model, same input, but different output. Here, I don't mean the wording different, I mean semantically different output.

[3:01](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=181s) You might be thinking that's the stochastic nature of LLM.

[3:07](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=187s) Hold on to that thought. Let's look at a few concrete examples.

[3:13](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=193s) So here, imagine you're giving a task for sentiment analysis, where we are supposed to label each review of a hotel either positive or negative.

[3:26](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=206s) This is typical NLP task, and today's LLM is really good at this.

[3:33](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=213s) However, there's one problem. When you pipe this input to the same LLM model,

[3:42](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=222s) occasionally, you will see different verdict coming out of it when you run multiple times.

[3:49](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=229s) That means a single evaluation run didn't tell you the whole story. You need to repeat your evaluation multiple times in order to get a holistic picture

[4:05](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=245s) by the average over the different runs results.

[4:10](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=250s) And this sounds like a technical inconvenience, but the reality is, it's way more problematic than just inconvenience. Let me show you an example in a different domain in cybersecurity.

[4:26](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=266s) Imagine you're a security analyst, where your day-to-day job is triage the security alerts and decide whether it's malicious, that you need to take certain actions to stop the attack, or it's false alarm, meaning it's actually benign, which you could ignore.

[4:47](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=287s) So here, example alert you got is there's a failed login attempt to Gmail account detected from suspicious IP.

[4:58](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=298s) And if we give such similar tasks to this one to AI agent and run it different times,

[5:06](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=306s) you will see sometimes it stay consistently benign, sometimes it stay consistently suspicious across different runs. However, there are cases where it flip-flop between benign, suspicious.

[5:24](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=324s) Now, your customer will have a difficulty using such product because they will wonder which one should I trust?

[5:34](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=334s) So the inconsistency is causing a trust issue in your product.

[5:42](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=342s) Imagine if you were in a POC bake-off, one vendor gave the consistent verdict all the time, while the other

[5:55](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=355s) vendor have this flip-flop verdict. You can imagine which one will win the deal.

[6:04](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=364s) Now, I'm pretty sure you want to solve this problem

[6:12](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=372s) First, we ha- we have to figure out where the inconsistency come from.

[6:21](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=381s) The good news is the data points tend to flip-flop, actually concentrate around

[6:29](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=389s) the decision boundary, the so-called gray zone. Let me illustrate with the earlier example. Here, in the sentiment analysis, we have two reviews. The first one, actually, no matter how many times you run through the LLM model, it will always give you the positive label.

[6:54](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=414s) In this case, it's pretty obvious it's... the review is super positive.

[7:00](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=420s) However, the second case is the one tend to flip.

[7:05](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=425s) If we look closer, it's kind of on the boundary. Sometimes this certain word is kind of, uh, relative positive, but sometimes it's a little bit negative.

[7:20](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=440s) In this case, you will find that even human experts will have a disagreement on these cases. In fact, there's no right or wrong answer. It really depends on particular company's policy and their preference.

[7:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=458s) For hotel thinking that the... such a review is not something I can do to improve the memorial kind of experience, [laughs] memorable experience, then that mean not to label as negative because there's nothing I can do.

[7:55](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=475s) But some hotel might care about it and want to label them as negative so they can improve. So that comes down

[8:04](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=484s) your preference. Let's look the other cybersecurity example.

[8:11](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=491s) Here is another case on the boundary, and why it's in the boundary is here there's a few attempts.

[8:22](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=502s) Indeed, there's attacker knocking on the door, but

[8:28](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=508s) attacker might not get in yet. So for enterprise customers who always have attacker knocking on the door, while if they haven't get in in environment, they don't want to worry about it, otherwise they'll be flooded by this type of alerts.

[8:45](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=525s) There's nothing they need to do because they are blocked outside already.

[8:52](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=532s) However, if attacker did manage to guess the correct password and pass MFA and they manage getting the environment, then it become very seriously and you need to take certain actions.

[9:04](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=544s) So almost similarly at the beginning behavior, but very different outcome and different response needed

[9:14](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=554s) depends on whether it's truly attack or attacker is still outside. So here,

[9:23](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=563s) whether we label them malicious or benign depends on company preference, whether they wanted to be notified about such a situation.

[9:33](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=573s) So again, this is an example where it depends on your preference

[9:41](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=581s) and additional information to disambiguate between the two different scenarios.

[9:50](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=590s) So from these two very different domains, you will see that

[9:55](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=595s) actually the data points tend to flip-flop are the one close to the decision boundary, where it need your clarity to disambiguate which side it should really belongs to.

[10:14](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=614s) So these data points are where human experts tend to make a mistake, and the traditional machine learning classifier will also struggle with. So in other words, it's not your AI agent's fault.

[10:31](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=631s) Your ag- AI agent simply point out the ambiguity that already exist.

[10:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=638s) So now we know where the root cause or the inconsistent is. The next step is identify where exactly they are, who are they, and fix them.

[10:59](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=659s) Surprisingly and not surprised, the solution to identify the gray zone actually exists already in machine learning, and it's called active learning.

[11:13](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=673s) So the idea of active learning is you have a lot of data point to be labeled. Imagine you have in your initial model and you put it in production, maybe in the monitor mode,

[11:29](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=689s) and you wanted to check the quality, but you don't have bandwidth to check all of them. Otherwise, it lose the point of having agent to do the work.

[11:39](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=699s) But you still want to identify where the model didn't do well.

[11:46](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=706s) And the active learning is to find a way to select the data points where tend to make a mistake,

[11:56](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=716s) and that way tells you where to pull your attention.

[12:03](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=723s) So let's go over the traditional, uh, machine learning models, active learning pipeline, and see how it's different in the case of the new LLM or AI agent era.

[12:16](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=736s) So first, you train your initial model, and then you make predictions on the unlabeled data set. It could be the production online data.

[12:29](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=749s) And then from there, you see there's certain cases where the model wasn't sure because the probability is close to point five. And those data points are the high informational one that your model is going to learn the most.

[12:46](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=766s) The uncertainty is a type of a signal to tell you where the model tend to make the mistake.

[12:53](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=773s) Another popular type of way to figure out the problematic one is query by committee, meaning that it, by a few different models or even the same model multiple times, and then you identify the disagreement, and that's where the model tend to make the mistake as well.

[13:13](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=793s) And that's where you need human clarification on the label, so the model can learn it from.

[13:23](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=803s) The... after you surface these problematic data points and for human to take a second look to make sure the label is actually correct or do it, does it need additional information to disam-ambiguate those cases, meaning adding new features.

[13:42](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=822s) And then you can add those label data set or potentially additional features to the next round of training. Then you retrain the model and proceed, uh, continue the cycle.

[13:54](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=834s) So then you can see in this particular active learning cycle, you spend very little, uh, amount, or at least another way, it's very efficient way to identify the problematic data points and spend your attention on those point where your model tend to learn the most.

[14:18](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=858s) So that's the traditional one. The good news is the LM way is not very different.

[14:27](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=867s) There's a major two part which we need some refinement. One is on this selection strategy, the other part is about retraining.

[14:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=878s) On the selection strategy one, um, here we suggest you use the one about identify disagreement. At least from what we have tried so far, the uncertainty score from LM is not very reliable.

[14:55](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=895s) It's kind of, uh, the LM model doesn't know what it doesn't know. So when it's very confident, it doesn't mean that it was to-- it w- is correct. So that, however, the, the disagreement from different runs or from different models actually give you a more reliable signal where the model actually not sure about its verdict, and it

[15:19](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=919s) need human guidance. And then after you identify this group or the, uh, disagreement and n-need attention, you label them and provide the feedback like what you did before.

[15:37](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=937s) Then afterwards, you might be thinking, "Oh, time to retrain the model." Yes and no. Yes, one option is retrain the model, but fine-tuning model is expensive. Here we are proposing a more lighter weight solution, easier to iterate.

[16:01](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=961s) It's about augment your agent with semantic and episodic memory.

[16:21](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=981s) Let's use the previous example to illustrate what I mean here. Um, as we talk about, in this case, they tend to flip. It's mainly missing the information

[16:33](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=993s) about y- particular company policy or preference. Here we can add it to the, um, the, the knowledge base. If this customer complain about something outside the hotel's control, then classify as positive because you don't want to pay attention to this.

[17:00](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1020s) And then in the cybersecurity example, something similar, you can put in, in that the password spray, meaning that the attacker trying to guess your password and try diff- with different times, even fail and continuously.

[17:16](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1036s) So if without successful login, then that should be benign. But in contrast, if it succeed with final login, then that should be malicious. So here you actually identify the additional information to help disambiguate the two and then label them separately.

[17:36](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1056s) And that's a clarity that help your model to learn, also help your human experts to label more consistently.

[17:48](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1068s) So this type of domain knowledge sharper your decision boundary and also help the human beings. Uh, this type of knowledge is like factual knowledge, is part of actually belong to your semantic memory.

[18:02](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1082s) And this is in contrast with the episodic memory. Episodic memory is in the case where

[18:09](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1089s) you haven't got around to distill the reasoning behind the earlier case, why it should be on malicious or the benign side. Instead, you said that I have seen these similar cases before, and it was labeled as benign, and let me leverage those past similar cases, and they made the decision accordingly.

[18:33](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1113s) And the advantage of the, this episodic memory way is, it's relative automatically. It has less human intervention. I mean, the human intervention was done in the past already. When's it-- During-- When you're doing a job online, um, you can reference them, at least your AI agent can reference them automatically and can making decisions without waiting for you

[18:57](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1137s) to distill the reasoning and putting in the semantic knowledge. So the idea is that the-- identify a similar case to the one under examination and then leverage their past decision as a reference.

[19:13](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1153s) And then imagine there's a new case which is kind of in the middle, doesn't belong to any past group, and that's the one you still need human attention. And this is also, um, where episodic memory resolve recurring case

[19:31](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1171s) automatically. And this particularly useful in the case of cybersecurity alerts because a lot of noise, especially those false positive noise, are the one keep recurring, and that's the one are the low-hanging fruit you can automate away.

[19:48](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1188s) And with that human review, which is a precious attention, the bandwidth you have, now you can concentrate on those where episodic memory fail to address. And this is also how

[20:03](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1203s) we chose between semantic memory and episodic memory. The two are actually not contradicting. In fact, they are complementary. Basically, you use episodic memory to address those recurring situations automatically.

[20:18](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1218s) And then for what's remain, still flip-flopping after the reference from the past or no reference to be able to use, then you pass on to human review. And then human review distill the domain knowledge and put it in the semantic memory to be used next time.

[20:38](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1238s) So here we talk about a solution actually brings three benefits. First,

[20:45](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1245s) we identify this data point near the decision boundary and use the domain knowledge in semantic memory and this past similar case in the episodic memory. We're able to improve the incon-- the consistency significantly.

[21:04](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1264s) So now you have a more trustworthy AI agent for your user to use. And two other by-product of this solution is you have a very efficient quality control.

[21:19](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1279s) You don't need to check every agent output, but instead, a subset of them which

[21:28](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1288s) your algorithm from active learning think is potentially problematic, and you check them, either provide additional information or clarify the label, and your model will learn from there.

[21:44](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1304s) So most importantly, your labeling effort is very small but high return.

[21:56](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1316s) Last but not least, now you are gathering your customer feedback along this way. You not only have a trustworthy, consistent model and high efficacy one, you also have one adapt to your customer environment.

[22:12](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1332s) I'm pretty sure your customer will love your product if you, your AI agent listen to them and adapt to them. So here are some experiment results from, uh, real data points.

[22:26](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1346s) So here we collect, uh, ninety-three alerts, cybersecurity alerts, and then we run them three times.

[22:35](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1355s) If without the solution we are proposing here, a quarter of them will flip-flop the verdict.

[22:43](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1363s) In contrast, after applying our solution or using episodic memory, about fifteen of percent of them become consistent. However, there's still ten percent of them remaining inconsistent.

[22:59](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1379s) They still flip, sometimes because there's no reference, similar cases in past to be ref-- used. Sometimes even there's a reference to be used, they still disagree after the second thought.

[23:15](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1395s) But not a problem. The episodic memory do the automatic knockdown of the inconsistency for fifteen percent, the remaining ten percent, human review and provide the additional knowledge to disambiguate those tricky cases.

[23:34](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1414s) After that, you also adapt to the customer environment that you want to learn.

[23:46](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1426s) So the three takeaways that I hope you have after this talk is, number one, inconsistency isn't usually your model problem. Stop blaming your model, but instead focus your energy on label issue and potentially insufficient information and help your AI agent with this

[24:11](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1451s) additional clarity. Second, the model disagreement isn't a bug, but a feature.

[24:22](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1462s) Treat each disagreement as a opportunity for your model to learn.

[24:29](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1469s) Third, fine-tuning isn't your only option. Arm your agent with semantic and episodic memory.

[24:40](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1480s) I hope you enjoy seeing your model keep improving and adapting to your customer environment.

[24:50](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1490s) Last but not least, I want to thanks to my colleagues and wonderful friends at Datadog. Special thanks to Zhichun, Santhosh, Anna Chu, and Stefan, Sai, and Matty to give me a lot of enjoyable discussions and helped me to clarify some of, um, the concept.

[25:14](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1514s) And also thank you for everyone in the BIDS Security Analyst team to make this journey enjoyable.

[25:22](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1522s) And thank you for listening. Hope you're enjoying training your model and having it improving.

[25:31](https://www.youtube.com/watch?v=wEc9aG7cRQc&t=1531s) Thank you. Bye.
