Voice AI Where One Wrong Question Breaks the Rules - Shiv Indap, Upstart & Brooke Hopkins, Coval

Upstart handles thousands of lending calls a day. When Shiv Indap, who leads voice AI there, launched an agent to automate verification calls, even a courteous follow-up could create a regulatory problem. For Brooke Hopkins, founder of Coval and formerly the lead for evaluation infrastructure at Waymo, the problem was familiar from self-driving cars. Two journeys can reach the same destination by different routes. In a lending call, it matters what the agent did along the way.

About 90% of Upstart applicants received an instant decision. The remaining 10% had to call in to verify information. First the agent had to establish that the application belonged to the caller. Then it had three questions to cover. Would the applicant’s income sources continue for the next three months? How did they plan to use the funds? Did they have outstanding obligations such as alimony or wage garnishments?

The question about the funds sounds straightforward until the applicant answers. Two answers called for quite different behavior under Upstart’s rules.

• Medical expenses. If the applicant said the money was for medical purposes, the agent could accept that answer. Asking how much they planned to spend was prohibited.
• Travel to a graduation. Here the agent needed one more detail. Was the applicant traveling to their own graduation, their child’s, or someone else’s? Whose graduation it was affected how Upstart understood the use of the funds.

Applicants had questions of their own, especially about application status and when the money would arrive. Compliance wanted this agent to stick to verification. Even a qualified estimate of up to three business days went beyond its job. The agent had to acknowledge the question politely and steer the conversation back to verification.

At launch, a human had to review every verification call. The operations team recorded its findings in a Google Sheet. In a few weeks, five columns became ten, then about fifteen. Reviewers added fields as they went and used them inconsistently. When the team wanted to compare calls, it had to work out what people had recorded in each column before it could use their observations to change the agent.

Coval helped Upstart give reviewers a more specific job. After a call ended, a large language model acting as a judge answered a yes-or-no question about whether the call met the compliance requirements. It also explained its answer. A reviewer could inspect what the judge thought had happened and record where they agreed or disagreed. The explanation made a verdict worth challenging even when its yes or no looked plausible.

Upstart initially tracked a long list of measures, including call time. Over time it concentrated on accuracy, clarity, completeness, policy alignment, whether the agent escalated correctly, and customer satisfaction (CSAT). Indap’s advice was to let a metric go when it stopped helping the team make decisions. A short call, for instance, tells a reviewer little about whether the agent should have asked whose graduation it was.

Each scored call entered a review queue. How often would someone who knew the lending rules reach the same verdict as the model? At launch, the human reviewer and the judge agreed on compliance only 33% of the time. Either a shared pass or a shared fail counted as agreement. The percentage measured how closely the judge matched the reviewers, leaving the proportion of compliant calls a separate question.

The disagreements led back to three places. First, experienced staff knew rules that had never made it into the agent. Upstart encoded that knowledge, and agreement began to rise. Then the team discovered that human reviewers disagreed with one another on cases where their criteria were unwritten. Making those criteria explicit gave reviewers a common basis for a decision. Finally, a wider range of cases exposed errors in the judge itself, prompting the team to revise it. By the time of the talk, Indap reported 92% agreement.

A process that gets one version of an agent approved can still leave engineers stuck when they need to change a prompt or swap a model. Hopkins starts the repeatable evaluation with three questions. Did the call accomplish its task? Did the agent follow the required steps to get there? What was the conversation like for the caller? In Upstart’s case, an agent could collect all three answers, sound perfectly clear, and still fail the second test by asking about the amount of a medical expense.

Human reviewers first label the behavior they care about. The team can then turn those criteria into automated checks using an LLM judge, rule-based checks, or audio processing. Starting with a readily available measure such as word error rate would answer a narrower question about transcription. A perfectly transcribed prohibited question is still a problem for the applicant.

Once those checks run on production calls, monitoring can look for two kinds of trouble.

• Known failures. A check can flag where the agent missed a required piece of loan verification or used the wrong wording. A review queue tied to that metric collects the calls that need inspection.
• Unanticipated patterns. Clustering groups similar cases, while anomaly detection picks out unusual ones. Both can give reviewers something to investigate before the team knows enough about a failure to write a dedicated check for it.

At thousands of calls a day, people can concentrate on flagged conversations while automated checks score every call. The reviewer decides whether a flag identifies a real failure. Who acts next depends on what happened. In Hopkins’s routing example, an off-script or incorrect answer could go to the agent team for a fix. A compliance violation could go directly to compliance staff so they could act on the incident itself.

A confirmed failure gives an engineer a case to reproduce. Once there is a proposed fix, Coval calls the revised agent and runs a voice-to-voice simulation. The simulated caller can speak with a different accent, interrupt the agent, or call against background noise. Because the exchange happens aloud, the team can also assess voice quality. Those conditions exercise parts of the conversation that a written exchange cannot reproduce.

Did the change fix the cases that prompted it? Hopkins points to improved cases in the simulation report. That comparison lets engineers check a specific failure before exposing another applicant to it. The result applies to the cases exercised in simulation. Scoring live calls after release is how the team can find out what still goes wrong with real callers.

Patterns across reviewed failures help engineers decide what to fix first. Hopkins sees a further possibility for self-healing agents. An agent could inspect evaluations and propose fixes, or a team could use that feedback for reinforcement learning or fine-tuning a language model. Those are ways to automate parts of the work. They depend on a reviewer making one consequential distinction first.

Suppose a reviewer opens a compliance flag. Which system needs to change? If the judge misread the conversation, the reviewer’s correction is an example for improving how future calls get scored. If the agent really did ask a prohibited question, the call is an example for changing its behavior and testing the proposed fix in simulation. Treating every flag as an instruction to change the agent would give an incorrect verdict influence over what the next applicant hears.

Upstart keeps changes behind a feature flag and requires human sign-off. Engineers usually approve changes to judges. When a change affects compliance, compliance staff participate in approving how the agent will behave. A simulation can show how the revised agent handled a medical-expense answer. Deciding that it is appropriate to enable that behavior for applicants remains someone’s named responsibility.