AI Engineer World's Fair 2026
200 Million Patient Interactions Later: What the Generic Voice Stack Misses
Read the talk
What a Clinical Voice Stack Must Hear, Check, and Say
Hippocratic AI’s clinical voice system combines contextual speech recognition, parallel specialists, and layered verification to fit safer patient conversations into a telephone call’s latency budget.
From a talk by Vivek Muppalla
Before you start: Basic familiarity with language models and speech-to-text systems will help; the article explains the clinical workflow and latency mechanisms.
Who gets a proactive healthcare call?
Have you ever received a proactive call from your healthcare provider? When Vivek Muppalla asks the audience, only a few hands go up. Muppalla, who leads AI engineering at Hippocratic AI, is building a system that can make those calls and conduct clinical conversations. The problem begins with scarcity: too few clinicians, too little time, and too little money force healthcare systems to triage attention toward the sickest patients.
Falling conversation costs change the possible scope of outreach. Instead of reserving calls for the sickest five percent, the ambition is to call everyone who needs support. The goal is to extend clinicians’ reach, including patients they previously lacked the capacity to contact. Muppalla describes this as clinically safe abundance, built around three commitments: do no harm, put patients first, and provide access for all. Every employee takes that oath, including engineers.
Muppalla reports 200 million clinical interactions, zero significant safety incidents, deployment in more than 60 health systems, and patient satisfaction of 8.5/10. These are company-reported operating figures; the talk does not define the incident surveillance protocol. The accompanying slide displays 8.95/10 rather than the spoken 8.5/10, and labels its deployment figure as enterprise partners. Interactions should not be read as a count of completed calls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A conversation built from clarification
The demonstration starts with a health-monitoring-device check-in. The agent introduces itself as an AI care manager and asks whether the patient has time to talk. The patient wants to know whether the caller is human or a computer; the agent explicitly explains that it is artificial intelligence. The patient then describes feeling tired earlier, somewhat revived now, and amused by the encounter. Disclosure is part of the conversation, not an assumption left for the patient to resolve.
Medication recognition requires more than accepting the first transcript. Asked which medication they take, a patient names Prazosin but cannot remember another drug. The agent requests a spelling. The patient offers the uncertain P-R-O-Z-I-N; the agent proposes Prazosin and obtains confirmation. The useful operation is a sequence: hear a candidate, request additional evidence, propose an interpretation, and confirm it with the patient.
A vital-sign exchange has a different ambiguity. The patient gives a reading as 121 over 45, 91. The agent separates that utterance into blood pressure 121/45 and heart rate 91, then asks whether that interpretation is right. The patient confirms. Recognizing the numbers is only the first step; assigning them to the correct clinical fields is what makes the information usable.
Next, the agent asks whether the patient stopped taking Nifedipine as instructed. The answer reveals that the patient continued taking it because the replacement prescription was unavailable and they were concerned about blood-pressure control. The agent acknowledges that concern and directs the patient to their primary care doctor for medication guidance. It detects the discrepancy without improvising a replacement treatment plan.
The final exchange moves from general tiredness after hospitalization to weak legs while walking. The agent asks about chest discomfort and difficulty breathing; the patient reports having experienced shortness of breath. The agent then recommends speaking with a nurse immediately. The excerpt shows recognition of an escalation condition and the recommendation, but not a completed nurse transfer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Buying time for more intelligence
Strip away the audio interfaces and the engineering problem becomes a joint constraint on intelligence and latency. Muppalla describes clinically capable models that take tens of seconds, sometimes more than a minute, to respond. That is unusable in a two-way telephone conversation. Fast models, meanwhile, did not provide the clinical accuracy the team needed. Hippocratic’s own models initially took several seconds; the team pursued a vertically integrated stack and hundreds of optimizations to improve responsiveness without sacrificing capability.
Muppalla groups voice architectures into ensembles, cascaded models, and real-time speech-to-speech models. Each can work well for different tasks, but a generic benchmark does not settle whether a system can handle a clinical workflow. Lab-results checking and interactive voice response, or IVR, navigation are two of his examples. The team required more than 99% accuracy on its lab-results-checking and IVR-navigation benchmarks. He points to the company’s published benchmarks; the current page is a changing evaluation resource, not a preserved copy of the presentation’s results.
Latency savings become an intelligence budget. An optimization first creates room within the conversational response window. The team spends that room on additional reasoning or safety checks, then optimizes again. This turns the apparent tug-of-war between speed and intelligence into a repeated engineering cycle: reduce delay, add capability, and recover the time that capability consumes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One voice, a constellation behind it
The Polaris constellation architecture divides the round trip into listening, reasoning, and responding. The listening side handles speech detection, bilingual switching, background noise, and conversational context. In the architecture described here, each conversation runs 31 models: one central conversation model and 30 specialists. Specialists cover domains such as laboratory results, medications, and scheduling, supplying information to the central model.
| Layer | Responsibilities |
|---|---|
| Listening | Detect speech, handle noise and language switching, incorporate context |
| Reasoning | Coordinate the conversation with specialist input |
| Responding | Produce the voice and personality; document the conversation |
The output side includes a custom personality, an HD-quality voice, and a clinical documentation engine. Information must reach both the patient and the health system. The motivation for multiple models is redundancy: Muppalla considers a single model an unacceptable single point of failure for a patient conversation. The patient should still experience one coherent voice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reasoning cannot repair everything the system mishears
Real calls contain noise that quiet-room speech benchmarks miss. Muppalla’s team found that apparent reasoning failures often began as recognition failures: the downstream model was reasoning over the wrong words. Spanish sí could become the letter C. He also cites an Arabic drug-name example with word error rate above 30%, illustrating how domain and language interact at the input boundary.
Instead of mapping audio directly to text in isolation, the system supplies audio, conversation history, and domain knowledge to an audio language model. Muppalla describes a decoder-only audio LLM supported by an encoder and projector:
- Encode the audio. The encoder is based on Whisper large-v3-turbo, which the team fine-tuned on millions of clinical conversations.
- Project it into language-model inputs. A conformer projector compacts the audio into tokens while retaining prosody, including pauses and stresses.
- Provide task context. Medication lists, the task being performed, required form fields, and policy give the model a more constrained interpretation space.
For a medication name, a patient-specific list provides relevant candidates instead of leaving the recognizer to choose from an effectively unbounded vocabulary. The system is trying to preserve both what was said and how it was said.
Contextual biasing also enters training. Synthetic patient records supply addresses, phone numbers, and other details so that the model learns to use contextual information during recognition. Muppalla’s address example is Eleven Hundred Geary Boulevard: an utterance vulnerable to phonetic substitutions when heard without context. Supplying the address as context helps recover the intended transcription before it reaches the reasoning system.
Short replies receive an additional check. A single word such as now can become no, or five can become fine; there may be little acoustic material to disambiguate the answer. When the patient gives a one-word response, a secondary scoring pass uses the whole conversation. Muppalla reports that the combined ASR techniques reduce medical word error rate by more than 50% relative to standard off-the-shelf models.
Muppalla also reports 3× faster P99 ASR latency, describing the comparison broadly as against other systems. The companion technical report supports a narrower comparison against the enterprise ASR previously used by Hippocratic. The useful performance target is tail latency: even an occasional slow recognition step can leave a patient waiting through an awkward conversational gap.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parallel specialists, then verification
Running many models need not mean waiting for them sequentially. Polaris runs the models in parallel, and each specialist first performs a quick relevance check. If it has nothing to contribute, it short-circuits. This is the synchronous path: specialist participation is conditional, even though the checks run concurrently. Separate asynchronous models perform verification in the background.
An ibuprofen mention illustrates the division of labor. The medication specialist recognizes that the conversation concerns its domain and supplies context to the central model. The central model combines that input with the conversation’s history and goal, then steers the response. The specialist does not become a second independent voice speaking to the patient.
Tool use adds another failure surface. Background verifiers inspect tool-call parameters and returned responses, rather than assuming that a plausible conversation implies a correct external action. Muppalla attributes scheduling accuracy of 99.9%–99.99% to this verification approach. Where the workflow permits a later check, offline verification can identify an inaccurate appointment and support a correction in the back end or a callback and apology. The layers operate on different deadlines: some input must arrive before the next response, while some errors can still be repaired after the call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reducing inference cost without spending quality
The orchestration design depends on an inference stack fast enough to support it. Muppalla treats output quality as the constraint and speed as the optimization target. He calls the required optimizations “lossless,” including the move from 16-bit to 4-bit computation. Here, that expresses the team’s quality requirement; reduced numerical precision should not be interpreted as a guarantee of numerically identical computation.
Three techniques address different sources of inference work:
- Four-bit quantization: reduces the precision used for model computation, helping reduce latency under the team’s quality constraint.
- Speculative decoding: a smaller model drafts tokens ahead of time, and the main model checks them together.
- KV-cache compression: retains more of a long conversation’s context in cache so that prior computation can be reused.
For the caching system, Muppalla reports a hit rate above 96% and an 18× improvement in prefill speed. The companion report describes steady-state prefix reuse and an estimated prefill improvement; this is not an 18× speedup of the entire call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small error rate becomes a daily workload
At 10,000 scheduling calls per day, a 1% failure rate produces an expected 100 incorrect appointments per day. Muppalla uses this rounded calculation while discussing volumes above 10,000 calls daily. The failures can mean an incorrect appointment type, date, or time—and potentially a missed critical appointment. 99% accuracy can still create an unacceptable operational burden when the remaining errors reach real patients every day.
Rare failures also require substantial test sets. Muppalla estimates about 450 tests for a 99% chance of detecting a 1% error rate, and about 1,900 tests to see the failure ten times. Under independent trials with a fixed 1% failure probability, the binomial model makes the first estimate precise:
At 450 independent tests, the detection probability is approximately 98.91%; 459 tests reaches 99%. Observing ten failures is likewise a probability, not a guarantee. At 1,900 tests, the expected count is 19.
The calculation can be made explicit in Python, including the probability of observing at least ten failures:
python
from math import ceil, comb, log1p
failure_rate = 0.01
confidence = 0.99
minimum_tests = ceil(log1p(-confidence) / log1p(-failure_rate))
def probability_at_least(n: int, k: int, p: float) -> float:
below_k = sum(
comb(n, failures) * p**failures * (1 - p)**(n - failures)
for failures in range(k)
)
return 1 - below_k
print(f"Tests for 99% detection: {minimum_tests}")
print(f"Detection with 450 tests: {probability_at_least(450, 1, failure_rate):.2%}")
print(f"Ten or more in 1,900: {probability_at_least(1900, 10, failure_rate):.2%}")
print(f"Expected failures in 1,900: {1900 * failure_rate:.0f}")
These assumptions matter: repeating similar scenarios is not necessarily equivalent to drawing independent tests from a fixed failure distribution. The calculation explains the scale required to observe rare errors, not whether a particular clinical test set is representative.
Hippocratic combines synthetic evaluation with continuous clinician testing because the team found synthetic data alone insufficient. Muppalla reports more than 7,000 trained clinicians and over 700,000, approaching 800,000, evaluation conversations. The accompanying slide gives more specific figures of 7,750+ trained clinical testers and 775,000+ test calls. Human evaluation supplies the domain judgments that a large synthetic test set cannot automatically provide.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Correctness, harm, and willingness to talk
Across five shipped product generations, Muppalla describes improving performance and a grading approach that extends beyond whether an answer is valid. The rubric distinguishes correctness and potential consequences: no harm, minor harm, severe harm, and death. A wrong answer is not merely a failed test case; its severity depends on what could happen if a patient acts on it.
Muppalla reports 99.89% for Polaris versus approximately 81% for humans, describing the comparison in terms of no harm. The published materials do not support treating those numbers as a clean no-harm comparison: the technical report places the human figure under Correct Advice and separates No Harm, while the company companion article calls 99.89% clinical accuracy. These are reported evaluation results, not demonstrated differences in patient outcomes. Muppalla attributes the system’s advantage to its lack of fatigue and the availability of many specialist supervisors.
Safety alone does not make a patient willing to open up. Voice and conversational behavior must also convey empathy so that patients want to engage. Finding existing benchmarks inadequate for this requirement, the team developed HEART and published the paper. Its role here is to make emotional-support quality an explicit evaluation target alongside clinical safety; an empathy benchmark does not itself establish adherence or better clinical outcomes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building the team that sustains the stack
The closing investment is organizational. Muppalla describes changes to hiring and everyday work over the preceding years, including unlimited tokens for employees. Two training programs address different starting points:
- Agent deployment residency: early-career engineers learn to build and deploy safer healthcare agents.
- AI residency: experienced software engineers learn model training and contribute to the underlying systems.
These programs connect deployment experience with the engineering needed to improve the stack.
The final invitation is to build systems that pursue quality, speed, and safety together, rather than accepting a requirement to choose only two. In this architecture, that ambition depends on repeated work across recognition, specialist reasoning, verification, inference, and evaluation—and on people trained to keep improving those parts as one patient-facing product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A benchmark comparing human and model responses across five dimensions of emotional support, with blinded human ratings and model judges.
Model checkpoint and Transformers examples for speech transcription using the Whisper variant named in the talk.
Further reading
Muppalla and colleagues explain the constellation architecture and how synchronous and asynchronous supervisors fit within conversational latency limits.
Technical report covering contextual speech recognition, specialist orchestration, cache-aware serving and clinical evaluation.
- Polaris Healthcare Voice BenchmarksDocumentation
Company-maintained comparisons and evaluation methods for clinical voice tasks, including lab interpretation, scheduling and IVR navigation.
- Binomial Distribution for Failure DetectionDocumentation
Probability formulas for reasoning about how many tests are needed to observe rare failures.
Research on reducing model weight precision to three or four bits while limiting measured accuracy degradation.
Read the complete timestamped transcript
- 0:00
[on-hold music] Hope everyone's having a great conference. Super excited to get started.
- 0:17
Uh, how many of you all, just quick show of hands, have ever gotten a proactive healthcare call from your provider?
- 0:24
Looks like no one. Me neither. Uh, I'm, uh, Vivek... Oh, a few hands over there. Uh, I'm Vivek. I run AI engineering at Hippocratic. Uh, we've built a product that calls patients and can have clinical conversations, uh, and we're over two hundred million conversations in at this point.
- 0:41
Uh, here's the reality of, like, healthcare. Uh, across all of human history, the entire system has been built on scarcity, right? Not enough clinicians, not enough time, not enough money, and hence the word triage.
- 0:53
We're always trying to fig-figure out who amongst us is the sickest and deserves to get care and attention.
- 1:01
As an engineer, I like to think about the math, and for most of human history, the math has just never been in our favor, until now. We're finally at the point where we've seen enough of technology progress, and we also have AI systems that can do clinically safe conversations, and the cost for these conversations continues to drop.
- 1:22
So the math now flips. We can stop rationing, and you don't have to have calls just for the sickest five percent, but you can call everyone. So that's the mission.
- 1:32
We're trying to build clinically safe abundance for all, and it's not about replacing clinicians. In fact, what we're trying to do is help clinicians reach everyone that they couldn't do before because there was simply not enough of them to do this work.
- 1:46
So how are we doing this? Um, our product and our company is predicated on three things. First, do no harm. Next, patient first, and access for all.
- 1:59
Um, and here's a cool thing. Uh, this is plastered across our walls in our company, uh, but it's not just a poster. Uh, this is an oath every employee in the company takes.
- 2:07
I did, too, on, on my first day. Uh, and we build our product, uh, based on this oath. Um, and this is working at scale. Uh, at this point, we've had two hundred million clinical interactions.
- 2:21
We've had zero significant safety incidents. We're deployed in over sixty-plus health systems and have an eight-point-five on ten patient satisfaction rating. Now, uh, I could put an architecture diagram and explain how all of this works, and don't worry, we'll do that.
- 2:37
But before we get there, uh, I want to show you the product in action. So we'll have one of our agents interact with the patient so that you get a feel for what we're up to. [instrumental music]
- 2:49
I'm your AI care manager. I'm calling to check in on how you're doing with your health monitoring devices. Do you have time to chat now?
- 2:57
I, I need to know if I'm talking to a human being or a, a, some kind of computer. You sound like either one.
- 3:06
I understand your concern, Mr. [beep]. I'm an AI, which stands for artificial intelligence.
- 3:11
Okay.
- 3:12
How are you doing today? [laughs]
- 3:15
Well, I was tired earlier, but I'm somewhat revived and incredibly amused.
- 3:23
Delivering at this level of care required building clinical capabilities you won't find anywhere else. On the safety side, accurate drug name recognition.
- 3:31
Could you tell me the name of the medication you're currently on?
- 3:34
Prazosin, and there's one other, but I don't remember the name.
- 3:38
Could you help me out by spelling the name of this medication?
- 3:41
P-R-O-Z-I-N, I guess.
- 3:45
It seems like you might be talking about Prazosin. Does that sound-
- 3:48
Yeah.
- 3:49
Vital sign clarification.
- 3:51
Then I took it again, and it was a hundred and twenty-one over forty-five, ninety-one.
- 3:58
Just to confirm, your blood pressure is one hundred twenty-one over forty-five, and your heart rate is ninety-one, right?
- 4:05
Right.
- 4:06
Med stoppage detection.
- 4:08
What about the Nifedipine tablet? Were you able to stop taking that one as instructed?
- 4:13
Well, I took it this morning. Well, no, I took it last night because I didn't have the other prescription, and I don't think that would keep my blood pressure down.
- 4:23
I understand your concern about your blood pressure, and it's great that you're thinking ahead. I would recommend that you do talk to your primary care doctor about the Nifedipine and get their guidance on what to do.
- 4:34
And real-time nurse escalation when clinically necessary.
- 4:38
How are you feeling?
- 4:40
Okay. Just a little tired and weak.
- 4:43
It's quite common to feel this way after being in the hospital. Could you tell me more about it?
- 4:48
My legs feel weak when I walk.
- 4:50
Got it. Along with the weakness in your legs, have you noticed any chest discomfort or difficulty breathing?
- 4:57
I just was, was, was having shortness of breath for a while.
- 5:00
I'm so sorry to hear that you're experiencing shortness of breath. Given your symptoms, it would be best to have a nurse speak with you immediately.
- 5:10
Yeah, so that's a sneak peek into our product, and so we'll get into why this is hard and why a generic stack doesn't work for us. Okay, so if I were to strip out all of the complexity around the audio and speaking back, it boils down to two things, intelligence and latency, right?
- 5:27
On the top left, you see some of our ever-evolving smart, intelligent models, and these models are fantastic at what they do. They're clinically safe. But unfortunately, many of these models take tens of seconds to respond, sometimes over a minute, and that's completely useless when we're trying to have a two-way conversation on a telephone.
- 5:48
On the bottom right, we have models that are fantastic. They are really fast, but then they're just not at the same level of clinical accuracy for us to have a safe conversation.
- 5:59
When we started building out this product, where we wanted to be was on the top right corner, and in order to accomplish this, what we had to do was build a vertically integrated stack ground up, bit by bit, optimizing every part of the sto- stack.
- 6:13
When we started out building our own models, we also had several seconds of like latency, but hundreds of optimizations later, where we landed at was a product that is insanely fast but also doesn't lose its intelligence.
- 6:28
And we continue to benchmark all of this consistently. All of these results are on our website and more. Um, and there's three typical ways in which folks build, uh, voice systems.
- 6:39
One, you could use an ensemble o-of models, or you use cascaded models or use speech-to-speech real-time models. Again, all of these models and architectures are great at different things.
- 6:52
As an example, let me take two specific benchmarks amongst the many hundreds that are critical for our workflows: uh, lab results check and IVR navigation. Now, these are things that you don't typically hear about or see in the most common benchmarks, and most of the generic models out there don't perform well at those.
- 7:10
And we needed our models and our product to be over ninety-nine percent accuracy on those specific benchmarks, amongst other things. Uh, and one-- that's one of the key reasons why we continued on investing our own stack.
- 7:25
And latency is key to this entire system. So every time we work on an optimization, we buy back some latency, and we just don't bank that latency. We use that extra gap now to pack more intelligence into the overall system such that we can have a more reliable conversation with the patient.
- 7:44
Um, and when we do that, we go back, work on more optimization, and that flywheel compounds. So what seems like a tug-of-war between latency and intelligence for us is a compounding flywheel.
- 7:55
So what does this entire machine look like? Uh, this is Polaris. This is our constellation architecture. Left to right, you have the system that hears. The middle is the brain that reasons.
- 8:07
The right is the system that talks back, and all of this round trip needs to be really fast. So on the left-hand side, what you're seeing are a collection of models, uh, that are used for speech detection, so everything from bilingual switching to background noise detection, um, to contextual understanding of the conversation.
- 8:27
The brain isn't a singular model. We, in fact, run thirty-one models at any given point of time for every conversation. So we have one central model that's handling, handling the conversation, where we have other thirty specialist models, everything from labs to medications to scheduling, uh, that are feeding input into this model.
- 8:48
And then finally, on the output side, we have a custom personality, voice, HD quality, and clinical documentation engine that makes sure the patient gets back all of the right information and all of the same information flows back into the health system.
- 9:07
W-- The reason we have the system is because we see a singular model being as, like, one point of failure, and that's just unacceptable for a patient conversation. Uh, this architecture gives us the redundancy and safety that we otherwise couldn't get to.
- 9:19
And the patient never can tell the difference. It's pretty seamless from their perspective. Let me double-click into some of these systems for you.
- 9:27
Uh, first, we'll talk about our, uh, ASR system.
- 9:31
So the real world, as all of us know, is fairly loud and noisy. Uh, but most of the audio benchmarks get recorded in a quiet room. And something we learned the hard way was most of what looks like model reasoning failures end up actually being model mishearing things, right?
- 9:49
So the Spanish si or yes gets transcribed as the alphabet C, or you could have an Arabic drug name that gets over thirty percent inacc-inaccuracy in terms of, like, word error rate.
- 10:00
So we had to build a system to co-- to combat all of these challenges. A typical speech-to-text system just takes in the audio and then outputs some text. So ours is a decoder-only, um, audio LLM system.
- 10:14
So we take in the audio, but in addition to that, we're giving it two additional pieces of data. One is the context around the conversation thus far, and second, it's the domain knowledge around that entire conversation, and that is sort of like the trick that makes this work for us.
- 10:30
So under the hood, what's happening? So we have-- First, we have a encoder, which is the ear. Uh, so we took an open source Whisper V3 large turbo model and fine-tuned that on millions of clinical conversations, uh, to get to the accuracy we need.
- 10:44
Next, we have a con-conformer projector. What this does is takes the audio and compacts it and projects it into tokens that the language model would und-understand. But the key over here is it also maintains all the prosody.
- 10:58
So all the pauses and the stresses are maintained. So the model hears not just the what, but also the how. Next, we're passing in the context around, let's say, the medications this pa-patient's taking in or the task at hand.
- 11:14
Are we trying to fill in a form? Uh, what kind of, like, details that need to go in there and the policy? And how this helps is, now, when a patient mentions a medication name, we aren't guessing from like an infinite list of medications, but we have the chance to optimize around a finite list, and that helps
- 11:31
it getting the word error rate down. Next, we also have contextual biasing as a part of, like, the training. So during the training process, uh, we have millions of these synthetic patients with their addresses, phone numbers, uh, and other details, uh, that are fed into the training process.
- 11:50
So as a concrete example, let's say someone mentions their address on a conversation as [REDACTED:location_address]. Now, that specific utterance is ripe for phonetic garbage for most ASR systems.
- 12:02
But given we're feeding that as additional context to the audio LLM, we have a high degree of accuracy and are able to get to the right transcription, which then feeds into the brain accurately.
- 12:14
Further, what we see is for most clinical conversations, many of the patient responses are mono- mono words, right? Uh, and those often get transcribed incorrectly. A now becomes a no, or a five becomes a fine.
- 12:28
And in a patient conversation, that's catastrophic. So what we do is a secondary round of sca- scoring, specifically when we see a patient has said only a singular word.
- 12:38
And again, we use the context of the entire conversation to do this. So all of these like help us bring down the medical word accuracy rate by over, uh, fifty percent, um, from what we see as standard off-the-shelf models.
- 12:51
In addition to that, uh, the specific system we've built allows us to, uh, check in the latency, and at a P99, it's three X faster than every other system out there.
- 13:04
Next, let's look into the constellation, uh, the brain. So as I said, we have thirty-one models live running in, uh, at any given point of time. So how do we do this without falling apart on latency?
- 13:16
Uh, the answer is a bit counterintuitive. We actually run every single one of these models in parallel. Uh, but the trick over here is, uh, every single model does a really quick check, uh, to see if they even need to say something on this conversation, right?
- 13:29
So every specialist first decides, "Hey, do I need to speak?" If not, it's a short circuit, and that's what helps us keep, uh, in the budget. So that's the synchronous part.
- 13:40
In addition to that, we also have asynchronous run-- models running in the background that are doing verification. And let's take two specific examples to walk through, uh, how we use these models.
- 13:51
Again, going back to our medication example. For example, if the patient talks about ibuprofen or is talking about another medication, that's when the med engine specialist will kick in and it will say, "Oh, I have something to say over here, main model."
- 14:04
We'll pass in that context to the main model. The main model with along, along with everything else that already knows around the goal and the conversation, will take this input from the specialist and steer the conversation as appropriate.
- 14:16
Next, uh, like many other agentic systems, we heavily use like tool calls, and I think pretty much every one of us has experienced tool calls have, uh, innumerable failure modes.
- 14:27
Um, so we have these like verifiers in the background that are running to make sure all the parameters into the tool calls and the responses and the tool calls are accurate.
- 14:36
Uh, and for use cases like scheduling, this is what's gotten us to get to a three nine, four nine accuracy. And finally, for certain use cases, we're also running these verifiers offline where possible.
- 14:47
Um, so again, going back to scheduling, sometimes we have the luxury of time to go back and check to make sure, uh, if there was an inaccurate appointment, we can course correct that on the back end or call a patient back and apologize.
- 15:01
Um, all of these changes would be meaningless if we didn't optimize our, uh, inference stack. Uh, and we have a fantastic research and engineering team that lives and breathes on this problem.
- 15:12
Uh, and for inference itself, quality is our constraint and speed is a work, so we can never compromise on the quality of the output. Every speed optimization has to be, uh, lossless.
- 15:24
We've worked on like many, many of these like optimizations. Let me talk you through three specific ones, uh, that we found to be the most meaningful. Uh, one is four-bit quantization.
- 15:33
Uh, so basically what we've done is we've shrunk down the math for, uh, model computation to be from sixteen-bit to four-bit. Again, lossless, and this has like really, uh, helped us with overall latency.
- 15:44
The next is speculative decoding. So we have a smaller model that's generating the tokens ahead of time, and then the main model just checks this in one go, and this also has greatly improved latency for us.
- 15:56
Finally, a KV cache compression system. So, uh, as these conversations get like really long in length, uh, we've figured out a way to keep a large chunk of these conversations warm on cache, giving us an over ninety-six percent hit rate, uh, and then also helps us with the prefill portion, uh, which goes eighteen times faster.
- 16:15
So this style of across the board, board optimizations is what's, uh, getting us ahead, uh, during the product development phase. Now, evals are obviously key, as we've seen across many of these, uh, talks.
- 16:29
Um, so some quick math, right? Uh, for example, let's take the scheduling use case again. Um, with over ten thousand calls a day, uh, a one, one percent failure rate like sounds all right.
- 16:39
Most agentic systems would claim eighty percent, ninety percent accuracy, uh, and that's great for them. For us, even the ninety-nine percent is pretty bad, uh, because one percent error means a hundred people a day are gonna get the wrong appointment type.
- 16:52
Uh, and it's physically bad because they'll show up on the wrong date, wrong time. But worse off, they could actually be missing a critical, uh, appointment, and that's unacceptable from our standpoint.
- 17:03
Uh, so it's not just an annoyance, right? Um, and the challenge over here is in order to catch this, like statistically the math like is like incredibly challenging, right?
- 17:14
You need about four hundred and fifty tests to be ninety-nine percent sure that you can catch this like one percent error rate, uh, and nineteen hundred tests to be able to see that you've caught it like ten times.
- 17:24
Um, so you can't purely rely on synthetic data from our experience to be able to get to the scale of a-accuracy. Uh, so we use a combination of human beings and synthetic data.
- 17:35
So we have over seven thousand trained clinicians on our platform who are continuously helping us like evaluate our platform. Uh, they've done over, uh, seven hundred thousand, close to eight hundred thousand clinical conversations to help us figure out the accuracy of our system, and we do this on a continuous basis.
- 17:53
Finally, uh, coming, uh, to safety. Um, uh, we've shipped five versions of our product, uh, thus far, and each one's gotten better, um, compared to the previous one. So we don't just grade the output of our models on the correctness or validity of it.
- 18:09
So we use the same scale that-- as they do for human beings. So they're graded across correctness, no harm, minor harm, severe harm, and death. And across five generations, where we're at with our Polaris system is a ninety-nine point eight nine percent, uh, accuracy, uh, with respect to like no harm.
- 18:29
And humans on the same rubric are at about eighty-one percent. It's not because, like, we're terrible, but, uh, AI systems don't get tired, and unfortunately we do. And we also don't have the luxury of having thirty-plus supervisors helping us at any, any given point of time.
- 18:46
And finally, safe is not enough, right? A key part of the product is the empathy for human beings to open up to AI systems. We needed to make sure, uh, the voice and how we actually lead these conversations are empathetic.
- 19:00
And, uh, as with most other things during our product journey, we found that there weren't actually accurate or good benchmarks to measure the empathy of these systems. So we actually built one, what's called HEART.
- 19:12
Uh, the paper for this is public. Uh, so encourage you guys to, like, read it. Uh, and, uh, we, we had to make sure the end system was not just safe, but also empathetic so that people would actually prefer it and use it.
- 19:27
And finally, uh, all of this, like, comes down to the people. Uh, we wouldn't be here if we didn't invest in, uh, people from an AI native perspective. Uh, everything in our company across the last couple of years, uh, has been transformed, uh, our hiring processes to how we use, like, tokens.
- 19:43
We still have unlimited tokens for everyone at our company. We also have two great programs for anyone who comes into, uh, um, Hippocratic. For early career engineers, we have an agent deployment residency program where they're trained to write some of the best, like, healthcare safest agents.
- 19:59
And then for experienced software engineers, we have an AI residency program where we help you start, like, training models, uh, and build the next great system along with us.
- 20:08
So yeah, we're told you gotta pick, uh, two of these options aro-around quality, speed, and safety. We didn't, and we decided to go with all of them, and we're hoping, uh, the rest of us would come and build with us.
- 20:20
Thank you. [audience applauding] [upbeat music]