Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo

What does a human reviewer actually contribute after a model produces an answer? In the simplest human-in-the-loop design, the sequence is model output → human → decision. The human participates in operation, supervision, or decision-making to supply accuracy, safety, and ethical judgment where automation falls short. For Angel Ortmann Lee, who introduces her work as security engineering for the Duolingo English Test, the question is whether the interaction actually elicits that judgment.

Everyday tools make deference convenient. Phone numbers live in contacts rather than memory. A GPS chooses a route that the driver may never examine. Search adds another layer: search for COVID-19 symptoms, and a Gemini summary can become the final answer instead of a starting point for consulting the CDC or WHO. As these small acts of reliance accumulate, trust can increase while caution decreases. A person remains present, but some of the deliberation has disappeared.

Lee introduces cognitive surrender through a Wharton study: people adopt AI output as their own with minimal scrutiny. Participants answered reasoning questions with access to AI assistance. The important distinction is between AI supplementing a person's thinking and replacing it—a transition the person may not notice.

On an adapted Cognitive Reflection Test, Lee summarizes the accuracy change relative to unaided reasoning as approximately +25 percentage points with correct AI assistance and −15 percentage points with faulty assistance. That reversal matters: the same willingness to incorporate AI output can amplify both correct and incorrect answers. Better performance when the assistant is right does not, by itself, demonstrate better independent reasoning.

Lee describes roughly 80% acceptance of incorrect AI answers; the study's denominator is trials in which participants consulted faulty AI, not all participants. The result concerns what happened after people sought the assistant's advice: they frequently followed it even when it was wrong.

The Duolingo English Test team investigated the same problem in When Machines Mislead: Human Review of Erroneous AI Cheating Signals. The DET is a high-stakes English proficiency exam taken online, on the test taker's own machine, with remote proctoring. At the time of the talk, Lee says 6,000 programs worldwide trusted DET results. Its security workflow combines identity verification, a locked-down testing environment, and AI-assisted monitoring, followed by human proctors reviewing exam footage and AI results.

The experiment focused on copy-typing: transcribing material being read rather than composing a response. Those activities produce different typing patterns. A custom detector examines anomalies in keystroke behavior and flags unusual sessions for review. Lee describes a conservative system that prioritizes fairness to test takers; alerts are uncommon, and trained proctors examine the relevant portions of the recording.

The test was straightforward: would experienced reviewers reject a false alarm when it appeared inside their usual workflow?

1. Select legitimate sessions with no cheating.
2. Attach fabricated copy-typing alerts pointing to specific moments.
3. Present the sessions to proctors as ordinary review work.
4. Observe whether reviewers uphold the alerts.

These were historical sessions, and the experiment did not affect test takers' results.

Lee reports that reviewers consistently exceeded 90% on their accuracy calibration metrics, yet accepted 50% of the fabricated alerts. The latter figure is an experimental false-alert acceptance rate, not the production rate of false cheating accusations. Lee interprets the result as automation bias: reviewers deferred to the signal without finding corroborating evidence. That failure is consequential in an exam whose results inform college admissions and visa decisions, even though these experimental judgments never reached customers.

Lee reports a 1% false-positive rate for the copy-typing model. The experimental sessions had originally received negative model predictions; the researchers, not the detector, introduced the misleading alerts. The reported rate should remain distinct from the fabricated-alert experiment and should not be read as an independently established, stable production rate. With both a conservative detector and experienced reviewers, the team targeted the interaction between them.

The intervention changed the proctoring guidelines in two specific ways:

• Decision authority: An AI signal is a preliminary alert; the proctor makes the final decision.
• Evidence requirement: Before upholding a flag, the proctor must find independent evidence in the video footage.

This changes the reviewer's task from confirming a model output to investigating whether the evidence supports it.

The study's model-implied rejection rate for fabricated alerts rose from 50% to 71%, a gain of 21 percentage points. Lee describes this as a 21% increase, but the endpoints establish a percentage-point change. The full paper compares sequential studies rather than simultaneous randomized guideline groups. It also reports genuine-alert rejection rising from 9% to 13%, so better rejection of fabricated alerts alone does not establish overall production accuracy. Lee says the new result more closely matched production sessions.

The team changed the instructions, not the model or the UI. A copy change made the reviewer's responsibility and evidentiary standard explicit. The result illustrates why adding human oversight is incomplete unless the system also specifies what the human must independently establish.

The linear picture—model → human → decision—leaves out what happens next. In a working AI system, model output enters an interaction, the interaction elicits human behavior, and that behavior produces data for evaluation and subsequent model development. Engineers cannot directly control how someone thinks, but they can change the interaction that invites, discourages, or structures that thinking.

Intentionally structured interactions produce labeled signals that can serve both evaluation and training. That makes the interface part of the data collection system. When the interaction captures a useful decision in a useful form, the team spends less effort reconstructing intent or cleaning ambiguous logs before the next iteration.

The feedback cycle can compound in either direction:

In the first cycle, apparent confirmation makes the model more confident while the human continues to defer. In the second, the system makes room for disagreement and preserves it as evidence. A high approval rate is useful only if approval represents the judgment the system needs.

Headphone detection exposes how a single yes/no control can hide two different judgments. The original interaction told a reviewer that headphones had been detected and asked whether to flag the session. But that bundles a perception question—did the model correctly identify headphone-like pixels?—with an enforcement question—does this warrant a violation?

In Lee's example, a hearing aid counts as a valid headphone-like visual detection, but wearing it should not trigger a violation. Under the combined control, a reviewer selects no to avoid penalizing the test taker. If that answer is also used as detector feedback, the system records the wrong lesson: it treats a policy exception as a detection error. Separating the decisions preserves both the detection judgment and the fair enforcement outcome.

A TypeScript record can express that separation directly. For the hearing-aid example, keep the model alert as input and record the reviewer's two answers independently:

type DetectionReview = {
  alert: "headphone-like object";
  observedObject: "hearing aid";
  detectionCorrect: boolean;
  violationWarranted: boolean;
};

const review: DetectionReview = {
  alert: "headphone-like object",
  observedObject: "hearing aid",
  detectionCorrect: true,
  violationWarranted: false,
};

const detectorFeedback = {
  alert: review.alert,
  correct: review.detectionCorrect,
};

const policyDecision = {
  violationWarranted: review.violationWarranted,
  reason: review.observedObject,
};

The record represents a review decision; it does not apply a penalty. Its essential property is that violationWarranted: false cannot silently become detectionCorrect: false.

The next example moves from security to tutoring. Lee asked an LLM to act as a writing tutor and critique a short passage deliberately written with English-learner mistakes and awkward sentences. Lee describes receiving 400 lines of response: praise, feedback, and an unsolicited complete rewrite. The response was hard to parse, and its advice was difficult to connect to particular parts of the original paragraph. Asking a friend to review a passage would rarely produce such an exchange; the volume and format made the feedback less usable.

The alternative is a Duolingo-style tutor mockup that marks up the passage itself:

• Green: Strengths, with feedback available on hover.
• Yellow: Awkward wording or something slightly off.
• Red: Direct mistakes.

Hovering reveals concise, actionable feedback, and suggestions can be accepted inline. Each comment has an identifiable target in the text. This resembles a classmate marking up an essay, giving the learner a local problem to understand and an incremental change to make. The comparison concerns how feedback is surfaced, rather than a demonstrated improvement in the underlying model.

Coding agents often create two opposite-looking versions of the same approval problem:

Both can turn the developer into a rubber stamp. The developer accepts the work, perhaps inspects it later on GitHub, and then owns the debugging, re-prompting, or manual repair. Neither interaction reliably creates a useful moment for deliberation.

Lee's comparison is a junior developer. A useful colleague neither disappears for a day and returns with an enormous pull request nor interrupts constantly with small questions. They plan, ask useful questions, document design decisions, produce a workable specification, and divide implementation into meaningful review units. For an agent, that means surfacing assumptions and proposed decisions while the developer can still correct them, with manageable steps and a reasonable number of interruptions.

The interaction also changes what the system can learn. Rapid skimming produces block-level accepted/rejected events, usually skewed toward acceptance. Developer steering can instead identify an incorrect assumption, a disputed trade-off, a stylistic preference, or a preferred approach. Those signals belong to distinct stages of the work and explain more than a final approval click. A reviewable workflow improves the developer's ability to guide the task while creating more specific feedback for future iterations.

Interaction design starts with the reasoning the task requires. If independent judgment matters, make the human an investigator: ask them to establish something, rather than merely validate a finished output. For a long-running task, expose assumptions early enough for sign-off to prevent downstream rework. When a model recommends an approach, present the alternatives and the reasons behind the choice so the user can weigh the trade-offs before implementation commits to them.

Match friction to the stakes. In the DET, a review decision can affect a person's opportunities, so deliberate slowing is desirable. Structured gates should require the thought that the decision deserves. The design question is where a checkpoint adds clarity and sustained attention, rather than noise. A casual AI conversation with little need for oversight has different requirements: unnecessary stopping points can undermine the experience, so the interaction should remain relatively seamless.

Interactions already contain potential labels. An approved plan or accepted suggestion can indicate that the output matched the user's intent; a modification or override can reveal where it fell short. Designing for those signals can reduce the need to reconstruct a clean dataset through separate annotation. But the meaning of an event depends on what the system observes around it.

Consider the sequence Lee highlights: a user accepts an output, then manually changes it or deletes it. If logging stops at acceptance, the system records success while losing the correction that would explain the failure. Capture the diff between what the AI proposed and what the user kept. Otherwise, approval followed by repair can enter the dataset as a false positive and reinforce the very behavior the user corrected.

Questions are feedback too. Requests for explanations and follow-ups may reveal low trust, an error, or a missing piece of the answer. Their content and sentiment help explain the interaction in ways that a binary acceptance event cannot. These are signals to interpret, rather than automatic proof that the output was wrong.

Evaluation should begin before the interface is finished. Define what success means for the whole AI system, choose concrete metrics, and identify the data needed to improve it. Those requirements should shape the interaction itself. The goal is to obtain evidence for the next iteration while helping users act as partners who can guide and correct the system.

That approach becomes concrete in the input and output format. Forms can capture specific user decisions. Tables can expose alternatives. Writing markup can attach feedback to the exact passage it concerns, while a design interface can let users identify individual elements. These structures give the interaction a precise target instead of relying on a wall of text and a general reaction.

Core assumptions should be visible before they shape later decisions. Confirming them early can avoid correction cycles and wasted tokens. Review gates then belong at the moments that require deliberate thought, encouraging users to take ownership of the decision. Finally, collect explicit feedback at the relevant touchpoint, with enough nuance to distinguish what the user accepted, challenged, or changed. A thumbs-up or thumbs-down often cannot carry that information.

Sometimes the fix is not a better model or more oversight. It is an interaction that makes the existing human judgment real: design for discernment, not approval.