I Monitored Crime Audio. Voice Agents Scare Me More. — Sumanyu Sharma, Hamming AI

Sumanyu Sharma, founder and CEO of Hamming, begins with an unusual reference point for voice-agent safety. At Citizen, his team listened to thousands of hours of police radio and sent millions of alerts across San Francisco, New York, LA, Chicago, Baltimore, and other cities. The incidents ranged from grim reports to someone stealing bags of ice cream or taking a ladder while a man was still on the side of a house. Interpreting audio at scale already meant dealing with consequences outside the recording.

Voice agents make him nervous because they are moving from demonstrations into production. Since he began working on voice reliability in early 2024, infrastructure and orchestration have improved enough to make a convincing experience much faster to build. His informal description—perhaps 60% good in a short time—captures the gap between getting started and handling the long tail.

Teams are experimenting with hybrid architectures that combine voice-to-voice capabilities and cascading stacks, seeking reliability while keeping latency low. The talk does not specify how those components divide the work. The consequential change is clearer: connections to calendars, customer relationship management systems, electronic health records, and reservation systems let agents take actions. A good conversation now needs to produce the right change in another system.

The first failure example concerns a customer seeking trade-in information. Confusing answers leave the customer unsure whether they are talking to AI or a person, and the business needs to call back to repair the situation. Natural speech has made the interaction plausible without making its information dependable. The cost includes the original confusion and the human work needed to restore trust.

The appointment example makes the distinction concrete. Sharma believed a voice agent had booked a physician appointment. He arrived, the front desk found that he was absent from the schedule, and he was turned away. He lost two hours. The observable failure was a mismatch between the outcome he understood from the call and the appointment record; the account does not establish which internal step failed. 3:37

Changing the patient or purpose changes the severity without changing the defect. A missing booking for a parent or grandparent, especially for a procedure rather than a checkup, could cost much more than wasted time. This is why conversational confidence is an inadequate success criterion: the caller acts on the promise, while the downstream system determines whether the service actually exists.

Sharma contrasts what he describes as declining crime with growing voice-agent deployment. He estimates at least a trillion phone calls each year and forecasts that conversational agents will handle a majority within five years. The arithmetic is conditional: one trillion interactions at a 1% error rate would produce 10 billion incidents annually. That calculation explains the stakes of volume; it does not establish the adoption forecast.

Across the 10,000 agents Hamming monitors, Sharma reports an error rate closer to 10%. The recording does not define the error denominator, sampling method, or measurement window, so this is an observation about Hamming’s monitored population rather than an industry-wide rate. Its examples span several mechanisms:

• Skipped prerequisites: An agent says it found the right policy while omitting eligibility or verification steps.
• Unauthorized actions: An agent applies a discount it was not supposed to offer.
• Understanding and information errors: An agent mishears the caller or supplies incorrect information.
• Uncompleted actions: An agent claims it booked an appointment that does not exist.

Frequency alone cannot rank these failures. Repetition and misunderstandings can be annoying; mishandling a hypothetical drive-through order for a vegan burger from someone with peanut allergies can create a safety risk. The relevant question is what the caller needs the system to preserve, and what happens when that requirement is lost.

The crime comparison then turns from frequency to topology. A local robbery or vehicle theft affects the people involved in that event. A voice deployment shares prompts and architecture across calls. One change can therefore alter the experience of millions of downstream users. Centralized control makes an improvement easy to distribute, but it also distributes a defect. Making incidents visible becomes part of controlling that reach.

The proposed response is a debugging loop borrowed from friends who worked on growth at Facebook: identify problems, prioritize by frequency and severity, understand a fix, execute it, check improvement and regressions, and keep monitoring production. A change only earns its place after the team examines what it improved and what else it disturbed.

How does production experience become the next test? The loop below makes the feedback path visible. Monitoring feeds new problems into investigation; verification checks a candidate change before the team resumes watching its behavior. The return path matters because a successful fix does not end the possibility of failure.

Monitoring has two separate ambitions: cover more conversations and discover behavior beyond the problems already named. Latency, interruptions, and automatic speech recognition issues are examples of known problems a team can track over time. Emerging patterns may become apparent only when many conversations are considered together.

The methods contribute different kinds of understanding:

• Manual listening: Start with individual calls to build context and intuition about what actually happens. Sharma explicitly recommends keeping this step, even though it cannot scale to all traffic.
• Rubrics and automated scoring: Turn observations into five or ten checks for greetings, closing, validation, core logic, and other known requirements. Evaluation tools can then apply metrics, model-based judging, and deterministic or stochastic scoring to more calls.
• Cross-conversation analysis: Look for recurring behavior across calls that the existing rubric may miss. Increasing scoring coverage does not automatically expand the set of problems the scores can recognize.

Hamming invests heavily in cross-conversation analysis, as do some of the strongest teams it works with. The talk does not describe a discovery algorithm. Its practical distinction is the unit of analysis: a call can receive acceptable scores while a collection of calls reveals a pattern worth investigating.

An isolated, low-impact issue warrants less attention than a systematic, high-impact failure. But systematic annoyances still deserve work: repetition becomes unpleasant at scale and can matter when comparing competing agents. The P0 target is a recurring failure with serious consequences. Sharma’s example is a financial-services agent meant to freeze credit cards that fails to perform the freeze. It defeats the purpose of the workflow even if the conversation sounds fine.

“Please fix my agent” is the easy part of the loop. Attempting a change takes less effort than establishing that it works. The missing appointment returns here as a concrete test case: take the failed conversation and replay it five, ten, twenty, or fifty times. Repeated runs estimate how consistently the changed system passes that case, rather than accepting one favorable conversation. 9:56

The test needs to preserve the original failure’s meaning: an apparent booking must correspond to a scheduled appointment. Then broaden the interaction while keeping that intent. Change wording, conversational patterns, accents, and style; add another intent to the call. The observable improvement sought is that the booking task succeeds across related interactions, without introducing failures elsewhere. Sharma proposes this verification procedure rather than demonstrating a repaired version of his appointment call.

What does varying the call add beyond replaying it? The diagram separates consistency on one recorded failure from coverage of different ways to express the same goal. Both feed verification, but they answer different questions: whether the original case repeatedly passes, and whether the improvement survives ordinary variation.

Some questions require actual recipients. For outbound calls, Sharma considers the first five seconds especially important: vocal quality and opening words shape the human response. His recommendation is live A/B testing for those hypotheses because simulations alone cannot supply the relevant result. Synthetic tests exercise controlled interactions; the live experiment observes how people respond to the choices being compared.

Verification therefore has several jobs: reproduce the failure, test variations around its intent, assess human responses where necessary, and check regressions. Continued production monitoring closes the loop by revealing what the pre-deployment checks did not anticipate.

Until this point, the caller earnestly wants a problem solved. Adversarial callers change the task: they may try to induce disclosure of protected health information or personally identifiable information. Sharma raises increasingly capable automated callers as a threat scenario. The concrete concern is persuasion aimed at an agent or a human who can access sensitive information.

The pressure to ship useful agents increases what an attacker can reach. More data gives an agent more information to reveal; more tools give it more actions to misuse. Fast deployment compounds that exposure. Human-sounding voices also create a separate risk for people who trust a persuasive caller. Capability and naturalness improve legitimate service while giving adversarial interactions more to work with.

Hamming released its red-teaming product in April to test this concern. Sharma reports that the team can break approximately one in five agents across its tests in financial services, healthcare, consumer settings, and elsewhere. Reported outcomes include bypassing verification and using prompt injection to obtain data the testers should not receive. The recording does not define a broken agent or give the sampling and attack procedure, so the fraction describes those tests rather than general attack prevalence. 13:01

The defensive program has complementary parts:

• Pre-deployment testing: Exercise the agent before ordinary callers encounter it. Text-to-text and voice-to-voice testing are both options; Sharma acknowledges tradeoffs without detailing them here.
• Production monitoring: Combine per-call scoring, manual listening, and cross-call analysis. Watch what the agent says and does, and also whether callers are trying to induce prohibited behavior.
• Continuous red teaming: Sharma recommends 24/7 adversarial testing, especially when bad interactions can be costly. This adds deliberate attempts to exploit the agent alongside observation of real traffic.

Watching both sides matters because the same undesirable agent behavior can arise during legitimate use or after deliberate manipulation. Monitoring the caller’s behavior helps a team recognize that difference. Continuous red teaming then asks whether someone intentionally searching for a weakness can find a path that ordinary-call testing misses.

The ending preserves the positive case: voice agents could make service feel more human than clunky interactive voice response trees, chatbots, or waiting on hold. Sharma expects more incidents as deployment grows and attackers exploit vulnerabilities. That is a forecast, grounded in his concern about exposure rather than a measured future outcome. His missing appointment supplies the personal stake: people will act on these systems’ promises.

The closing invitation is to work on this problem and examine deployed agents’ architecture and evaluations. Both deserve attention: architecture determines what the agent can do, while evaluations determine which failures the team can recognize. Better calls remain the goal; the reliability work follows the agent from its first tests through everyday use and deliberate attack.