AI Engineer World's Fair 2026
Build AI Systems for Discernment, Not Approval - Angel Ortmann Lee, Duolingo
Read the talk
Build AI Systems for Discernment, Not Approval
A Duolingo English Test experiment shows how human review can amplify false AI signals—and how interaction design can recover independent judgment and better feedback data.
From a talk by Angel Ortmann Lee
When a human in the loop stops thinking
What does a human reviewer actually contribute after a model produces an answer? In the simplest human-in-the-loop design, the sequence is model output → human → decision. The human participates in operation, supervision, or decision-making to supply accuracy, safety, and ethical judgment where automation falls short. For Angel Ortmann Lee, who introduces her work as security engineering for the Duolingo English Test, the question is whether the interaction actually elicits that judgment.
Everyday tools make deference convenient. Phone numbers live in contacts rather than memory. A GPS chooses a route that the driver may never examine. Search adds another layer: search for COVID-19 symptoms, and a Gemini summary can become the final answer instead of a starting point for consulting the CDC or WHO. As these small acts of reliance accumulate, trust can increase while caution decreases. A person remains present, but some of the deliberation has disappeared.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Correct answers can conceal dependence
Lee introduces cognitive surrender through a Wharton study: people adopt AI output as their own with minimal scrutiny. Participants answered reasoning questions with access to AI assistance. The important distinction is between AI supplementing a person's thinking and replacing it—a transition the person may not notice.
On an adapted Cognitive Reflection Test, Lee summarizes the accuracy change relative to unaided reasoning as approximately +25 percentage points with correct AI assistance and −15 percentage points with faulty assistance. That reversal matters: the same willingness to incorporate AI output can amplify both correct and incorrect answers. Better performance when the assistant is right does not, by itself, demonstrate better independent reasoning.
Lee describes roughly 80% acceptance of incorrect AI answers; the study's denominator is trials in which participants consulted faulty AI, not all participants. The result concerns what happened after people sought the assistant's advice: they frequently followed it even when it was wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Testing whether skilled reviewers catch false alerts
The Duolingo English Test team investigated the same problem in When Machines Mislead: Human Review of Erroneous AI Cheating Signals. The DET is a high-stakes English proficiency exam taken online, on the test taker's own machine, with remote proctoring. At the time of the talk, Lee says 6,000 programs worldwide trusted DET results. Its security workflow combines identity verification, a locked-down testing environment, and AI-assisted monitoring, followed by human proctors reviewing exam footage and AI results.
The experiment focused on copy-typing: transcribing material being read rather than composing a response. Those activities produce different typing patterns. A custom detector examines anomalies in keystroke behavior and flags unusual sessions for review. Lee describes a conservative system that prioritizes fairness to test takers; alerts are uncommon, and trained proctors examine the relevant portions of the recording.
The test was straightforward: would experienced reviewers reject a false alarm when it appeared inside their usual workflow?
- Select legitimate sessions with no cheating.
- Attach fabricated copy-typing alerts pointing to specific moments.
- Present the sessions to proctors as ordinary review work.
- Observe whether reviewers uphold the alerts.
These were historical sessions, and the experiment did not affect test takers' results.
Lee reports that reviewers consistently exceeded 90% on their accuracy calibration metrics, yet accepted 50% of the fabricated alerts. The latter figure is an experimental false-alert acceptance rate, not the production rate of false cheating accusations. Lee interprets the result as automation bias: reviewers deferred to the signal without finding corroborating evidence. That failure is consequential in an exam whose results inform college admissions and visa decisions, even though these experimental judgments never reached customers.
Lee reports a 1% false-positive rate for the copy-typing model. The experimental sessions had originally received negative model predictions; the researchers, not the detector, introduced the misleading alerts. The reported rate should remain distinct from the fabricated-alert experiment and should not be read as an independently established, stable production rate. With both a conservative detector and experienced reviewers, the team targeted the interaction between them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Require evidence before upholding a flag
The intervention changed the proctoring guidelines in two specific ways:
- Decision authority: An AI signal is a preliminary alert; the proctor makes the final decision.
- Evidence requirement: Before upholding a flag, the proctor must find independent evidence in the video footage.
This changes the reviewer's task from confirming a model output to investigating whether the evidence supports it.
The study's model-implied rejection rate for fabricated alerts rose from 50% to 71%, a gain of 21 percentage points. Lee describes this as a 21% increase, but the endpoints establish a percentage-point change. The full paper compares sequential studies rather than simultaneous randomized guideline groups. It also reports genuine-alert rejection rising from 9% to 13%, so better rejection of fabricated alerts alone does not establish overall production accuracy. Lee says the new result more closely matched production sessions.
The team changed the instructions, not the model or the UI. A copy change made the reviewer's responsibility and evidentiary standard explicit. The result illustrates why adding human oversight is incomplete unless the system also specifies what the human must independently establish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The interaction becomes the next model's data
The linear picture—model → human → decision—leaves out what happens next. In a working AI system, model output enters an interaction, the interaction elicits human behavior, and that behavior produces data for evaluation and subsequent model development. Engineers cannot directly control how someone thinks, but they can change the interaction that invites, discourages, or structures that thinking.
Intentionally structured interactions produce labeled signals that can serve both evaluation and training. That makes the interface part of the data collection system. When the interaction captures a useful decision in a useful form, the team spends less effort reconstructing intent or cleaning ambiguous logs before the next iteration.
The feedback cycle can compound in either direction:
| Interaction | Human behavior | Recorded signal | Effect on improvement |
|---|---|---|---|
| Confident output invites approval | Rubber-stamping | Approval treated as truth | Reinforces unchecked predictions |
| Review requires independent judgment | Agreement and disagreement | More faithful positive and negative labels | Exposes specific model failures |
In the first cycle, apparent confirmation makes the model more confident while the human continues to defer. In the second, the system makes room for disagreement and preserves it as evidence. A high approval rate is useful only if approval represents the judgment the system needs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate what the model saw from what policy requires
Headphone detection exposes how a single yes/no control can hide two different judgments. The original interaction told a reviewer that headphones had been detected and asked whether to flag the session. But that bundles a perception question—did the model correctly identify headphone-like pixels?—with an enforcement question—does this warrant a violation?
In Lee's example, a hearing aid counts as a valid headphone-like visual detection, but wearing it should not trigger a violation. Under the combined control, a reviewer selects no to avoid penalizing the test taker. If that answer is also used as detector feedback, the system records the wrong lesson: it treats a policy exception as a detection error. Separating the decisions preserves both the detection judgment and the fair enforcement outcome.
A TypeScript record can express that separation directly. For the hearing-aid example, keep the model alert as input and record the reviewer's two answers independently:
typescript
type DetectionReview = {
alert: "headphone-like object";
observedObject: "hearing aid";
detectionCorrect: boolean;
violationWarranted: boolean;
};
const review: DetectionReview = {
alert: "headphone-like object",
observedObject: "hearing aid",
detectionCorrect: true,
violationWarranted: false,
};
const detectorFeedback = {
alert: review.alert,
correct: review.detectionCorrect,
};
const policyDecision = {
violationWarranted: review.violationWarranted,
reason: review.observedObject,
};
The record represents a review decision; it does not apply a penalty. Its essential property is that violationWarranted: false cannot silently become detectionCorrect: false.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put writing feedback where the writing is
The next example moves from security to tutoring. Lee asked an LLM to act as a writing tutor and critique a short passage deliberately written with English-learner mistakes and awkward sentences. Lee describes receiving 400 lines of response: praise, feedback, and an unsolicited complete rewrite. The response was hard to parse, and its advice was difficult to connect to particular parts of the original paragraph. Asking a friend to review a passage would rarely produce such an exchange; the volume and format made the feedback less usable.
The alternative is a Duolingo-style tutor mockup that marks up the passage itself:
- Green: Strengths, with feedback available on hover.
- Yellow: Awkward wording or something slightly off.
- Red: Direct mistakes.
Hovering reveals concise, actionable feedback, and suggestions can be accepted inline. Each comment has an identifiable target in the text. This resembles a classmate marking up an essay, giving the learner a local problem to understand and an incremental change to make. The comparison concerns how feedback is surfaced, rather than a demonstrated improvement in the underlying model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make coding agents reviewable collaborators
Coding agents often create two opposite-looking versions of the same approval problem:
| Pattern | Immediate burden | Likely review behavior |
|---|---|---|
| One giant diff across many files | Understand everything at once | Approve, then inspect later |
| Approval for every file or function | Respond to constant interruptions | Repeatedly click yes |
Both can turn the developer into a rubber stamp. The developer accepts the work, perhaps inspects it later on GitHub, and then owns the debugging, re-prompting, or manual repair. Neither interaction reliably creates a useful moment for deliberation.
Lee's comparison is a junior developer. A useful colleague neither disappears for a day and returns with an enormous pull request nor interrupts constantly with small questions. They plan, ask useful questions, document design decisions, produce a workable specification, and divide implementation into meaningful review units. For an agent, that means surfacing assumptions and proposed decisions while the developer can still correct them, with manageable steps and a reasonable number of interruptions.
The interaction also changes what the system can learn. Rapid skimming produces block-level accepted/rejected events, usually skewed toward acceptance. Developer steering can instead identify an incorrect assumption, a disputed trade-off, a stylistic preference, or a preferred approach. Those signals belong to distinct stages of the work and explain more than a final approval click. A reviewable workflow improves the developer's ability to guide the task while creating more specific feedback for future iterations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the reasoning, then place the friction
Interaction design starts with the reasoning the task requires. If independent judgment matters, make the human an investigator: ask them to establish something, rather than merely validate a finished output. For a long-running task, expose assumptions early enough for sign-off to prevent downstream rework. When a model recommends an approach, present the alternatives and the reasons behind the choice so the user can weigh the trade-offs before implementation commits to them.
Match friction to the stakes. In the DET, a review decision can affect a person's opportunities, so deliberate slowing is desirable. Structured gates should require the thought that the decision deserves. The design question is where a checkpoint adds clarity and sustained attention, rather than noise. A casual AI conversation with little need for oversight has different requirements: unnecessary stopping points can undermine the experience, so the interaction should remain relatively seamless.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Capture what happens after approval
Interactions already contain potential labels. An approved plan or accepted suggestion can indicate that the output matched the user's intent; a modification or override can reveal where it fell short. Designing for those signals can reduce the need to reconstruct a clean dataset through separate annotation. But the meaning of an event depends on what the system observes around it.
Consider the sequence Lee highlights: a user accepts an output, then manually changes it or deletes it. If logging stops at acceptance, the system records success while losing the correction that would explain the failure. Capture the diff between what the AI proposed and what the user kept. Otherwise, approval followed by repair can enter the dataset as a false positive and reinforce the very behavior the user corrected.
Questions are feedback too. Requests for explanations and follow-ups may reveal low trust, an error, or a missing piece of the answer. Their content and sentiment help explain the interaction in ways that a binary acceptance event cannot. These are signals to interpret, rather than automatic proof that the output was wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design the evidence into the system
Evaluation should begin before the interface is finished. Define what success means for the whole AI system, choose concrete metrics, and identify the data needed to improve it. Those requirements should shape the interaction itself. The goal is to obtain evidence for the next iteration while helping users act as partners who can guide and correct the system.
That approach becomes concrete in the input and output format. Forms can capture specific user decisions. Tables can expose alternatives. Writing markup can attach feedback to the exact passage it concerns, while a design interface can let users identify individual elements. These structures give the interaction a precise target instead of relying on a wall of text and a general reaction.
Core assumptions should be visible before they shape later decisions. Confirming them early can avoid correction cycles and wasted tokens. Review gates then belong at the moments that require deliberate thought, encouraging users to take ownership of the decision. Finally, collect explicit feedback at the relevant touchpoint, with enough nuance to distinguish what the user accepted, challenged, or changed. A thumbs-up or thumbs-down often cannot carry that information.
Sometimes the fix is not a better model or more oversight. It is an interaction that makes the existing human judgment real: design for discernment, not approval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The DET study of fabricated cheating alerts and revised proctoring guidelines, published in October 2025.
Further reading
Technical companion research on behavioral signals for copy-typing detection and cheating-ring identification.
Read the complete timestamped transcript
- 0:00
Hello, my name is Angel Ortmann Lee, and I'm a software engineer at Duolingo. I work on security for the Duolingo English Test. Today's talk is The Human in Your Loop Isn't Thinking: Build AI Systems for Discernment, Not Approval.
- 0:15
So let's get started with some background. What is human-in-the-loop AI? Human-in-the-loop AI is a framework where a system or process is actively involving a human, uh, who is participating in the operation, supervision, or decision-making for an automated system.
- 0:31
Humans are involved because they're able to ensure the accuracy, safety, or ethical decision-making for that system. Uh, they are a piece of that decision-making where, uh, AI normally falls short.
- 0:43
You can kind of think of this as a linear process where a model provides some sort of output, and a human sees that and makes the decision.
- 0:52
Now, in the age of AI, trust is a big part of the conversation. Recently, as technology has developed, uh, a lot of things that used to be manual cognitive tasks are now part of technology, so no one really memorizes phone numbers anymore.
- 1:06
They're just sitting in our contacts list on our smartphones. When you're driving over somewhere, you just put the directions into your GPS, and you don't really think about the details of that route.
- 1:15
Instead, you just trust the GPS to come up with the optimal way for you. When you're searching something in the search engine, you used to look at the top results, and now as AI is becoming more increasingly integrated into our day-to-day life, you might be seeing some AI summaries at the top instead.
- 1:32
Here's an example of Googling, uh, COVID-19 symptoms, and you see that Gemini summary at the top. And instead of looking at the CDC website or the WHO website, you might actually look to that as your final answer to the question that you are asking to the search engine.
- 1:49
And as these pieces of AI are becoming more integrated into your day-to-day life in these atomic little things like searching for an answer or getting a result from an app, your trust of AI systems will increase and caution will decrease.
- 2:04
And this is something that's happening across all of society. A study at Wharton was looking exactly that. How is AI reshaping human reasoning, and what does it mean for us as we continue to use it day to day?
- 2:17
They've observed an interesting phenomenon called cognitive surrender, when a human forgoes deliberation and adopts AI output as their own with minimal scrutiny. They saw this in a study where humans were asked to look at reasoning exams, and they were given AI resources to answer those questions.
- 2:35
They realized that AI can either supplement or supplant a human's thinking, and the human might not even know it's happening. For example, for questions where the AI was right, the human performance increased by twenty-five percentage points, whereas when the AI was wrong, it decreased by fifteen.
- 2:53
This suggests that the humans were taking in that AI cognition and not really thinking critically about what that answer is, and instead using those answers as part of their own, uh, reasoning, uh, together with their human instinct and deliberate reasoning, amplifying those results.
- 3:13
Most interestingly, they saw that eighty percent of participants were an- accepting those AI answers even when it was wrong. So they were lowering that barrier to entry and just trusting the AI, uh, even if they weren't really critically examining the correctness of that result.
- 3:31
So at the Duolingo English Test, we wanted to look at the same thing, and we published some research titled When Machines Mislead. It was a case study on the English exam, specifically in AI-human interaction.
- 3:45
So for context, what is the DET? The DET is a high-stakes exam for-- that measures your English proficiency, and it's fully online and remotely pro- proctored. You can take it on your own machine in the comfort of your home, and it still provides high-quality results that six thousand programs worldwide are trusting day to day.
- 4:06
Uh, as you can imagine, with a fully online assessment, there are some interesting things that, uh, from a technological standpoint that have to happen to ensure good quality security.
- 4:16
This includes identity verification, a lockdown testing environment, as well as a variety of ways in which AI-assisted monitoring is happening to predict different types of cheating. Lastly, we have a final te- uh, round of, uh, scrutiny where we have human proctors review all of the video footage of the exam as well as all of those AI results.
- 4:40
Specifically for the study, we wanted to target one of our AI che- cheating detection, detection systems, which is copy-typing. Copy-typing is a form of cheating where you're writing down, uh, information that you're reading currently as opposed to something that you're thinking right now at this moment.
- 4:57
So as you can imagine, your typing patterns differ when you're transcribing versus composing, and this is something that our custom model is measuring. It's looking at anomalies in keystroke patterns, and it's flagging them for, uh, sessions that are unusual.
- 5:13
So our model is highly conservative, and so we're prioritizing fairness for our test takers. Uh, so this is not a very common flag. However, when humans are-- human proctors are taking a look at these, they're well trained to, um, properly examine those segments and see if this is a flag.
- 5:34
So for our experiment, we wanted to answer the question: Would a skilled reviewer catch a false alarm, or would they just rubber-stamp it? So we know that our proctors are highly accurate at detecting various forms of cheating.
- 5:46
We wanted to see specifically if we give them a fake signal, how well would they be able to discern that this is a totally legit session? The way we did that is by selecting sessions that had no cheating whatsoever.
- 5:59
They were totally okay, and we made it look like they have this AI signal that says, "Hey, check if there's copy-typing behaviors present at a specific moment." We presented them to the proctors as part of their normal workflow.
- 6:14
They thought it was just another session that they were taking a look at, um, and this did not impact our test takers in any way. The results were quite interesting.
- 6:23
So we found that despite the fact that our human reviewers are consistently scoring above 90% on their accuracy calibration metrics, they actually accepted 50% of these fake signals, meaning they were falsely accusing people of cheating half the, half the time.
- 6:41
And this coin flip rate is something that is a strong suggestion of automation bias. This means that they're deferring their judgment to AI, uh, without giving a second thought to whether or not that there's evidence to corroborate this, and if this is a true, uh, cheating session.
- 7:00
Uh, again, to reiterate, these were not sessions that were actually going to our test takers. These were historical sessions, so there was no impact to our customers. However, this is something that was definitely very alarming to the team because in a high-stakes exam where these results are going to influence college admissions and visa decisions, this is not
- 7:20
something that we can take, uh, we can just accept. We knew that the problem was not the model. Our model has a 1% false positive rate, and also these were sessions that were negatively predicted.
- 7:32
We knew that our problem was not the people. Our people are highly skilled and very experienced with detecting cheating, so we wanted to take a look at the interface, and that's where we targeted our solution.
- 7:44
So we decided to take a look, a look at the human-AI interaction loop. Specifically, our proctors are given a series of guidelines about how to interact with the system that they have, and we wanted to just update that proctoring guideline to emphasize two things.
- 8:00
First, the AI signal is just a preliminary alert. They're the final decision-maker. And second, when they're making that decision, they must find independent evidence in the video footage before upholding a flag.
- 8:12
This simple copy change led a 21% increase in rejection rates, meaning that from 50%, uh, only half of se- sessions that were rejected, uh, the majority, 71, were now determined to be totally okay.
- 8:29
And this is really good because it is more closely matching, uh, sessions in production.
- 8:36
And for an engineer, this means that we didn't have to tweak the model, and we didn't have to tweak, uh, the UI in some way. We simply had to take a look at the copy change, and this really changed the way that people were interfacing with that AI result.
- 8:51
So what are the implications of this? As an AI engineer, your interaction loop determines how effective your AI system is and also what you can learn from it. The way that you measure efficacy and the way that you can...
- 9:07
uh, and the data that you collect from that to then drive your next iteration are really important. Earlier, we talked about the, uh, human-in-the-a- human-in-the-loop AI as a linear interaction where a model goes to a human, goes to a decision.
- 9:23
However, in reality, that process isn't linear, it's cyclical. Models y- provide some sort of output that goes into an interaction, that goes into a human behavior, which yields data for your evals and then later on your model.
- 9:38
You can't really change the way a human behaves unless you are that human, but what you can do is tweak that interaction such that it can elicit different results from that human behavior.
- 9:48
And that data is golden. It leads to better analytics, model improvements, and future iterations, including your next generation of your model or maybe some different product.
- 10:00
Now, zeroing in on that interaction por- portion, we can look at how to evaluate your system. Ask yourself the question, "How can I measure and improve the efficacy of my AI system?"
- 10:11
Looking at structured interactions that were designed intentionally, you can see that those interactions yield good data that are labeled signals that can then become training data and evals for a better model.
- 10:24
And that is a flywheel that compounds. As you have the better model, you can have better data that then, uh, has better interactions with your human, which then continues to cycle.
- 10:34
Think of your structured interactions as a system property that specifically yields high-quality data. That high-quality data is unlocking meaningful insights and quicker development iterations. You're not gonna be spending, uh, days cleaning your data or trying to find useful insights.
- 10:51
Instead, you're gonna have structured data that is directly impacting your next iteration loop.
- 10:58
And this is a compounding effect, uh, and it's cyclical. If you are not being intentional with that design, you can get stuck in a vicious cycle where your model is making confident calls, and because your in- interface is not actually, uh, eliciting that deliberation from a human, they just end up rubber-stamping it, and that positive signal is
- 11:19
logged as truth. So over time, your model becomes more confident, and the human is not encouraged to think further, so they continue to defer to the AI, and the AI becomes the person in the driving seat.
- 11:31
Uh, instead, what you want is a virtuous cycle. Uh, if you have an interface that forces independent judgment, uh, this allows the human to really think critically about the decisions they're making and have real disagreements surface.
- 11:44
That means that you have true positive and negative labels that are honest and get logged to then continue to have model improvements that are targeting exactly where the model goes wrong.
- 11:56
So let's take a look at some examples.
- 11:59
For example, we have other cheating flags that are driven by AI. As an example, we have headphone detection. On the left, you can see, uh, a bad pattern that we had where we were saying, "Hey, the model detected headphones at this moment."
- 12:14
Can you flag it here? And it was just asking a simple yes or no question. We realized that there's actually two questions hidden in this single, uh, single CTA.
- 12:25
Uh, one, we're asking, "Headphones were detected. Is that true or false?" Uh, did the model correctly predict that at this video segment there are some set of pixels that look like headphones?
- 12:37
And secondly, are we applying a flag that this person had a violation for i- not correctly using headphones at this moment? Why is this important? For example, if there's somebody who has a hearing aid, that means that the model correctly predicted that there are headphones or earbuds detected.
- 12:56
Uh, that is a true signal. However, we don't actually want to penalize this person for wearing, uh, hearing aids, so we don't want to flag them for a violation.
- 13:06
Previously, as to not falsely accuse somebody of cheating, we would select no, but that would be a bad signal to give to our model. So breaking this up into two pieces is really important because we actually get more data and it's better quality, and we're not harming the model in the long run.
- 13:23
Another example is a quality tutor. On the left here, you see a pattern where the interaction's overwhelming, and on the right, delightful. So this is an example of me talking to an LLM.
- 13:36
I asked it to o- act as a writing tutor and provide feedback for a passage that I wrote. The passage was specifically written as if it was a English-language learner, so it had a few mistakes or awkward sentences.
- 13:48
It was a relatively brief paragraph, and instead, uh, this LLM decided to give me 400 lines of text and, uh, it was very overwhelming and hard to parse. Uh, first it decided to praise me, then give some sort of feedback, and then completely rewrite my passage even though I never asked it to.
- 14:07
The feedback was not direct. It was hard to tie feedback to specific parts of the input, and it was so large that it would be difficult to actually improve your writing through this sort of inter- interaction.
- 14:20
And most importantly, this did not feel natural. If you were to send an email to your friend saying, "Hey, can you take a look at this thing that I wrote?"
- 14:28
You would not expect them to send you an essay double the length telling you all the things that you did right and wrong and then also give you a whole new version.
- 14:35
Instead, you would probably want it to look something like this. Here you have a screenshot of a Duolingo-style writing tutor where the text is directly marked up. Uh, the green is simpl- uh, signifying that this is something that was good and also providing some sort of feedback when you hover over it.
- 14:53
Yellow signifying something is awkward or a little bit off, and red being direct mistakes. When you hover over all of these markups, you can get direct, actionable feedback that's concise, and you can accept the suggestions in line.
- 15:07
Most importantly, all the feedback is directly tied to a portion of text. This is more closely mimicking human behavior. Specifically, like in an academic setting, if you are handing your essay to a classmate, you would expect feedback in line as they mark up your text rather than an essay back.
- 15:26
This is useful because over time, if you get these sorts of feedback points, you can improve m- uh, in incremental steps in your writing, and this is really important for surfacing the exact same AI output.
- 15:41
Lastly, and probably most applicably to an engineering audience, is a coding agent. Now, there are a lot of coaching... coding agents out there, and there's a lot of different UIs.
- 15:51
People have their personal preferences. But I've seen that there's two patterns that a lot of, uh, out-of-the-box coding agents are falling into. So one pattern is that if you ask it to do something, it's gonna touch a lot of files and give you a giant diff, basically doing the thing that you requested all in one go.
- 16:09
This leads you probably to approve all the changes and then take a look at them maybe on GitHub, uh, so that you can see all the things that it did.
- 16:18
Another approach is that it will ping you a notification every single time it's changing some sort of file or function, and you have to keep clicking yes, yes, yes just so that you can get to the result.
- 16:30
Either way, in both of these cases, you're just becoming a rubber stamp. You're accepting all of the changes and then you're taking a look at them all in one go and debugging later.
- 16:40
Anything that went wrong is your responsibility to then identify, fix, re-prompt, or do manually. And it's not a very delightful experience. Instead, you probably want a coding agent that acts like a junior developer.
- 16:54
You wouldn't want a junior developer to come onto your team and just be like, "Oh, sure, I can do that," and disappear for a day and then give you a 1,000-line PR.
- 17:03
You also probably don't want a junior dev who's asking you a question every five minutes at your desk. Instead, you want someone who's able to plan, ask good questions, and document those design decisions, giving you a good spec and breaking up their PRs in meaningful and reviewable ways.
- 17:20
This means that it's delightful for you as a mentor or, in this case, uh, a developer who is partnering with a coding agent to experience things in a way that is easily reviewable.
- 17:33
You can highlight assumptions and maybe answer questions about them and you can easily see, uh, a plan and the things that... the decisions that it's making before things go wrong.
- 17:44
Also, the steps are manageable and you're not being pinged too many times.
- 17:50
And then going a little bit into the data of... side of this, if you have a agentic co- coder and it's leading an interaction where a developer is encouraged to just quickly skim everything and not really discern, this means that you're just collecting these, uh, binary signals of accepted and rejected, usually skewing towards accepted, and, uh, that
- 18:11
doesn't really give you that much information because it's just on some coding block. Instead, if an agent is acting as a partner with a developer steering, you can get rich data because it's structured specifically around different parts of the development cycle, and it's capturing nuanced and structured decisions about the different parts of, uh, the tasks that it's
- 18:32
doing. For example, it can look at assumptions that were bad, the trade-offs that it made, stylistic preferences, and approaches that the developer is taking. And all of those pieces of data can then continue to make your experience as a s- like, as your AI system, coding system better.
- 18:51
So let's look at some design principles. First, you can engineer the reasoning. Think about what reasoning pattern do you want to elicit from a human, and how does your interface a ch- challenge that?
- 19:03
So if you have a system that is specifically optimizing for independent judgment, reframe your system to think of the human as an investigator, not just a validator. Don't just give a, "Hey, is this looking good?"
- 19:16
But instead, have it be a more thoughtful, engaged effort.
- 19:21
Surfacing assumptions is something that's very valuable when you care about, a lot about the quality, uh, of a long-running task. If the model is stating them earlier on and asking for a sign-off, you can prevent any future miscommunication or having to go back and fix certain assumptions.
- 19:37
Weighing trade-offs is also very important because very often, um, an LLM might give you some sort of option, and it thinks it's best without having to actually weigh the different op- uh, the different, uh, trade-offs, the design decisions that it's making in the background.
- 19:52
Presenting those options and the reasoning for why allows the user to continue to be in control over the output, and they can opine earlier on so that they can get the thing that they actually want.
- 20:05
Lastly, for sustained attention, you want to build in friction exactly where the stakes are high. If you want the person to slow down and think deliberately, friction is your friend, which leads us to the next principle, to match the friction to the stakes.
- 20:19
In a high-stakes ex- uh, example like ours, where the Duolingo English Test really matters, uh, for people and their livelihood, we want our human reviewers to be deliberately slow and very thoughtful with the decisions that they make.
- 20:33
This means that we want to be adding review gates that are well structured and friction so they can slow down and think about these things so that they can never become a rubber stamp.
- 20:43
You want these people, the people to slow down, and you want the output to remain high quality. You don't want noise, and you want clarity. So this means that you have to be actively thinking about what friction, um, like, what speed bumps do you want to put along the road, and how and what checkpoints.
- 21:02
For systems where you have low oversight, for example, something delightful where you're just chatting with an AI, you want to make sure that it's relatively frictionless. Your design has to optimize for a happy, someone who's happy with that output and doesn't really feel like there's a lot of stopping points or scenarios where it's kind of difficult to
- 21:25
interact with the system. Instead, you want it to be absolutely seamless.
- 21:31
Next principle is every interaction is already a label. Instead of having to select a portion of your data and have human annotators, um, get you a better, cleaner data set, you can kind of already start thinking about that specific interaction piece as already providing you labels and signals for your next iteration.
- 21:53
So if the agent plan was approved or a suggestion was accepted, this means that you did something right and your output matched the intent of the user. However, if the output was modified or a recommendation was overridden, that probably signifies something bad.
- 22:08
However, a lot of systems don't really capture that diff, and the model falls short because what you get on that decision is a yes or no, and the human clicked yes, then manually went in and changed something or erased it completely.
- 22:23
If you don't capture that diff, which could be something f- the A- the AI falling short or maybe completely being wrong, you actually capture a false signal that then can pollute your data sets.
- 22:35
Instead, think about how can you measure that diff.
- 22:39
Lastly, there are also questions that a human might be asking, for example, explanations or follow-ups, and this could mean that the human has a low trust in the system or there was something wrong with the AI.
- 22:51
Uh, so take a look at the kind of questions that are being asked and what sentiment they have.
- 22:58
Another principle is to stop asking how to evaluate the model. Instead of building your system and then being like, "Hey, what kind of data can I capture to see if this is good?"
- 23:09
Proactively think about what defines success for your AI system. Think about how you can be measuring that with concrete metrics and what data will you need to improve the system from the start.
- 23:21
Those decisions will be informing the way you design that interaction such that you can ensure that you have the hard evidence for that next step. Every interaction should be helping the user to teach them how to improve the system and be a partner with the AI.
- 23:40
This leads me to my final point, which is all of those things are leading you to engineer that interaction. There are different ways to engineer that interaction, which I've covered briefly, but some of the principles are this.
- 23:53
First, you can have structured inputs and outputs. That way you don't just have vibes or giant walls of text. Instead, you have specifics. You can think of this in terms of forms that the user is filling out, structured output in forms of tables, markup UIs, for example, that, uh, writing tutor or maybe it's a design thing where
- 24:13
you can highlight specific elements. All of those ways are having targeted interactions between the human and the AI, ensuring a better result. Next, highlighting that, uh, those assumptions. If you are having assumptions being made by the model, if you surface them proactively and ask if it's legit, that's gonna save you some tokens down the line so that
- 24:33
the human is not having to fix or argue or correct with the, correct the model output. Instead, it's being proactive and h- having those things that are core, uh, pieces of information that are gonna inform later decisions.
- 24:48
Next is building in friction and review gates. That deliberate slowing down so that you can have good quality thought and dec- uh, re- required for good quality decision-making and good quality data is all important because if you're bringing in that friction and review gates, people are going to start thinking clearly as if it's really their own thing
- 25:10
rather than just something that AI is sup- uh, supplanting.
- 25:14
Lastly, collecting explicit feedback is really important because as we've covered, every one of those interactions is a piece of data, so all of those signals are going into your next generation of the model, and collecting explicit feedback, not just a thumbs up, thumbs down, but instead feedback at the correct touchpoint, uh, with the correct amount of nuance
- 25:36
is something that can really drive improvements directly.
- 25:41
So yes, uh, you should be designing for discernment. Sometimes the fix is not a better model or more oversight. It's just engineering the interaction itself. Thank you.