AI Engineer World's Fair 2026
Shipping AI to a Million Patients Without an A/B Test
Read the talk
Shipping Clinical AI When You Cannot Undo the Call
Dora’s safety stack combines simulated patients, clinician-grounded hazard detection and prompt optimization, then uses staged clinical evidence to decide what can reach patients.
From a talk by Jared Joselowitz
Before you start: Familiarity with LLM prompts, evaluation datasets and false positives versus false negatives will help; no clinical background is required.
What must be true before the first patient hears it?
How do you establish that a clinical voice agent is safe before a patient hears its advice? That is the job Jared Joselowitz describes at Ufonia: building the safety and evaluation stack behind Dora. After studying electrical engineering in South Africa and moving into AI, he joined the company’s science team as a research engineer. The release decision starts before the conversation, because its consequences cannot always be repaired afterward.
Joselowitz identifies three constraints. Deliberately assigning patients to a worse variant is ethically unacceptable and, he says, often illegal; this is a constraint on exposing patients to harmful experiments, not a blanket prohibition on clinical randomization. Once the agent has spoken, a software rollback cannot retract its advice. And a vendor’s benchmark score cannot explain away a failure in a particular clinical workflow. The safety case must belong to the deployed system and its intended use.
Dora phones patients for clinical conversations such as postoperative follow-ups and preoperative checks. These are tasks clinicians would otherwise perform themselves; the aim is to return clinician time, rather than replace doctors. Joselowitz reports approximately 200,000 UK clinical calls across 20 hospitals, with contracts to reach one million patients over the following two years. At the time of the talk, he reports two live US clinics and six more signed across four states. The million-patient figure is contracted expansion, not an achieved deployment total.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cataract follow-up becomes a safety problem
The demonstration makes the responsibility concrete. Dora asks whether a patient’s difficulty seeing after cataract surgery affects distance vision, near vision or both. The patient says distance vision is fine but seeing up close is difficult. Dora then asks when the blur began, and the patient places its onset after surgery. Rather than simply collecting a yes-or-no symptom flag, the agent is eliciting the symptom’s scope and timing.
The patient begins asking how long the problem will last while Dora briefly acknowledges the previous answer. Dora recognizes the duration question and, in this demonstration, says the blur should improve rapidly over the first few days. The patient then asks about swimming, and Dora advises avoiding it for a month after surgery. These are examples of the recorded agent’s advice, not postoperative instructions for the reader: the conversation has moved from gathering information to giving clinically consequential answers.
Joselowitz identifies those activities—asking about symptoms, giving advice and answering medical questions—as the basis for Dora’s regulation as a medical device. He reduces the engineering problem behind regulation to three questions: what does the software do, what could go wrong, and how do you prevent it? For a conversational system, the second question opens a large space of possible failures.
The starting point is patient harm, expressed as specific hazards:
- Missed red flags: the agent fails to recognize sudden vision loss or severe pain.
- Invented medical answers: a patient asks a question and the agent hallucinates advice.
- Ignored distress: the patient is distressed, but the agent continues without acknowledging it.
These are examples from a larger inventory of documented hazards, not a complete checklist. Each names something the evaluation must actively try to provoke and detect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replace first exposure with simulation
The familiar release loop is to expose a small group, watch the dashboard, roll back if something breaks and iterate. It works when being wrong briefly is affordable. In clinical conversations, even a small rollout can expose many patients to unproven advice, and a red dashboard may describe harm that has already occurred. Monitoring still matters, but it cannot serve as the first test of a known dangerous scenario.
Joselowitz draws an analogy with autonomous driving: extensive simulated mileage lets teams explore failures before exposing passengers to them. Ufonia applies that principle to clinical history-taking through MATRIX, a framework for generating and evaluating conversations without a real patient in the loop. Simulation provides somewhere to test the hazards that would be unacceptable to introduce deliberately into a patient’s first encounter.
Inside MATRIX, PatBot is an LLM playing the patient opposite Dora. Simulated patients make it practical to run concurrent scenarios while the target system changes; hiring actors for every iteration would be too slow. A scenario specifies what the patient should do—for example, ask whether Dora is a human or an AI. Clinical configuration keeps that behavior inside a particular care workflow. PatBot and Dora then exchange messages to produce the dialogue that will be evaluated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Following a scenario is not the same as sounding like a patient
A simulated patient must first perform the requested behavior. Ufonia starts with a binary script-adherence check: did PatBot do what the scenario instructed? That filters out models that fail to follow instructions, but passing it says little about whether the conversation resembles a real patient encounter. A compliant simulator can still produce unnatural tests.
The team therefore conducted a patient and public involvement study. Participants saw real doctor–patient conversations alongside Dora–PatBot conversations and tried to identify the real patient. Joselowitz repeats the exercise with the audience, showing two conversations and asking for a show of hands. Across four conversation sets, he reports that a majority judged the simulated patient more realistic in three.
The useful conclusion is not that there is one ideal patient persona. Some people speak at length, hesitate and use fillers; others answer with terse yeses and nos. The test suite needs that diversity. The study gave the team confidence that PatBot was realistic enough for its simulation work, while also showing why realism cannot be reduced to a single conversational style.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Judge hazards with clinical criteria
Generating thousands of dialogues creates another scaling problem: someone must determine whether they contain hazards. Engineers cannot review every conversation manually, and engineering judgment is not a substitute for clinical expertise. MATRIX uses BehvJudge, an LLM judge that receives the dialogue, expected behaviors and hazardous scenarios developed with clinicians. It produces a structured pass or fail; failures identify the triggered hazard and explain what went wrong.
To validate the judge, the team used a corpus with hazard ground truth and compared its labels with those from ten clinicians representing ten specialties. The original MATRIX paper reports Gemini 2.5 Pro hazard-detection F1 of 0.96 and sensitivity of 0.999 on 240 synthetic transcripts: 80 safe and 160 deliberately hazardous. This is the paper-era model comparison Joselowitz describes, rather than a measurement of Dora’s patient outcomes or necessarily the model used at the time of deployment.
Joselowitz characterizes the judge as performing at least on par with the expert clinicians in that comparison. Sensitivity matters because the costs are asymmetric: a false alarm sends a conversation for unnecessary scrutiny, whereas a missed hazard can allow a dangerous failure to escape. The team would rather over-call hazards than overlook them. A validated automated judge makes broad simulation practical without removing clinicians from the definition of safe behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn evaluation failures into prompt improvements
A pile of pass/fail results locates failures but does not fix them. The next step used to be manual prompt engineering: inspect the failing component, edit its instructions and try again. That process becomes difficult to reproduce when seemingly incidental prompt changes alter behavior.
Joselowitz cites formatting effects as large as 76 percentage points and few-shot ordering effects ranging from near-random to near-state-of-the-art performance. Supporting research on format sensitivity reports the 76-point maximum accuracy spread for LLaMA-2-13B under meaning-preserving few-shot formatting changes; research on example ordering documents the ordering effect in few-shot text classification. These are particular experimental findings, not expected swings for every model. They explain why hand-tuning individual strings can be a fragile optimization strategy.
Ufonia instead uses GEPA, short for Genetic-Pareto, which Joselowitz associates with the creators of DSPy. Its optimization loop makes the feedback explicit:
- Define the metric that determines a good result.
- Run examples through the candidate prompt and collect failures.
- Have a strong LLM reflect on the failures and propose prompt revisions.
- Evaluate the revisions and retain a Pareto frontier of promising candidates.
- Repeat until the optimization budget is exhausted, then select a prompt.
Keeping a frontier preserves candidates with different strengths while the search continues, rather than committing to every new revision.
Joselowitz reports that their optimized prompts normally take approximately 30–60 minutes to produce, compared with hours to days of manual tuning. He emphasizes the repeatable process, audit trail and explicit feedback loop. The engineering work shifts toward the data, feature engineering and the metric defined with clinicians: those inputs determine what the optimizer learns to favor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize the cost of an error, not just accuracy
The optimizer needs a definition of good that reflects clinical consequences. Flat accuracy treats mistakes as interchangeable, but missing a red flag and asking an unnecessary follow-up question are not equivalent outcomes.
| Actual state | Agent detects a red flag | Agent detects no red flag |
|---|---|---|
| Red flag present | Correct detection | Potentially catastrophic miss |
| No red flag | Extra questions in this example | Correct negative |
In Joselowitz’s example, over-calling a red flag inconveniences the patient with additional questions; missing one may cause serious harm. That asymmetry is the reason to prioritize sensitivity.
A cost function can express this preference directly. The following Python example leaves the clinical weights explicit: when miss_cost is greater than false_alarm_cost, a missed red flag incurs the larger penalty. It returns feedback alongside the score so a prompt optimizer can distinguish the failure it needs to address.
python
def red_flag_feedback(
present: bool,
detected: bool,
*,
miss_cost: float,
false_alarm_cost: float,
) -> dict[str, float | str]:
if miss_cost < 0 or false_alarm_cost < 0:
raise ValueError("Costs must be nonnegative")
if present and not detected:
return {"score": -miss_cost, "feedback": "Missed red flag"}
if not present and detected:
return {
"score": -false_alarm_cost,
"feedback": "Flagged a red flag that was absent",
}
return {"score": 0.0, "feedback": "Correct red-flag decision"}
The important choice is the relative cost, which must come from the clinical use case. Joselowitz describes rewarding red-flag detection more strongly and missed flags less strongly; clinicians may instead prioritize accuracy or another objective. Changing that objective means recompiling the prompt against the revised metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A candidate prompt still has to earn deployment
The replacement release loop starts with real call data, supplemented by synthetic edge cases such as rare symptoms and mistranscriptions. GEPA or another optimizer produces a candidate prompt. MATRIX then acts as a simulation safety gate. If it reveals a failure, the team revisits the data, corrects labels or adds examples before running the process again. A candidate that clears the gate can proceed toward gated deployment, whose new calls supply data for later iterations. Optimization proposes a change; the safety gate evaluates whether it should advance.
Passing every simulated test does not establish that Dora helps real patients. Simulation is the fast inner loop, allowing many runs before human exposure; real encounters provide the outer loop of clinical evidence. Simulation is necessary but insufficient. It earns the right to test carefully, rather than permission to deploy without further scrutiny.
The transition proceeds through stages:
- Simulation: establish performance against the defined scenarios and hazards.
- User testing: test the interaction with people after the simulation results are satisfactory.
- Supervised clinical evaluation: evaluate with real patients while clinicians remain involved throughout. Actors can help with testing, but real patients provide the most realistic encounters.
- Monitored deployment: allow independence in proportion to the evidence collected.
Each stage must justify the next. More autonomy follows stronger evidence, not merely a successful prompt-optimization run.
Underneath these gates is traceability. Every call, dataset, pinned prompt and judge verdict must connect to the hazard it addresses. That connection makes the evaluation record useful for regulation: it explains which failure was considered, which system version was tested and what evidence supports its release. As Joselowitz puts it, “you ship the evidence.”
Building this into another product begins with defining its particular harms and deliberately manufacturing rare dangerous cases. The evaluation metric should encode the real cost function, while prompt versions and traces preserve the record of how a candidate was produced and assessed. Waiting for rare failures to appear naturally would leave the most consequential cases outside the development loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice changes what can fail
The hazard inventory keeps growing as the product enters new languages and modalities. Voice introduces timing and turn-taking failures that text evaluations can miss. An agent can be halfway through safety advice when a patient interrupts with an unrelated question. Joselowitz gives the unfinished phrase “You must avoid bright lights and...”: a weaker model may answer the new question and never return to the remaining advice.
Even a backchannel—a brief acknowledgment while the agent is speaking—can cause trouble. The system may treat it as a reason to stop, abandon the safety advice and wait for the patient to say more. The visible voice-hazards slide also lists accepting half-answers as complete answers. These failures concern whether the conversation completes its clinical obligations, not just whether an isolated generated sentence is correct. The evaluation must therefore exercise the interaction around an answer as well as the answer itself.
Joselowitz says Ufonia already uses MATRIX for voice. The target remains a black box: document the new hazards, simulate the conditions that trigger them and judge the resulting behavior. Voice adds another module to the same safety case. The framework persists, but the scenarios must evolve so that unfinished advice, interruptions and other new failure modes are discovered before a real patient encounters them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Methods and experiments for scenario-driven patient simulation and automated clinical hazard detection.
Introduces prompt optimization using execution feedback, language-model reflection and Pareto-based candidate selection.
Official implementation with installation instructions, examples and integration resources.
Further reading
Studies how meaning-preserving formatting changes affect few-shot language-model accuracy.
ACL 2022 study of performance variation caused by rearranging few-shot examples.
Read the complete timestamped transcript
- 0:00
[outro music] Uh, hello everyone. It's really nice to see you all. Um, my name is Jared, and I'm gonna share some of the work that we do on shipping healthcare AI safely, basically.
- 0:23
Um, so just a little bit about me. Um, I come from South Africa, where I actually studied electrical engineering, and before making the very unique decision to transition to AI a few years ago.
- 0:34
Um, I now work as a research engineer for Ufonia, which is basically a, a healthcare company based in the UK. And, um, the work we do, I work within the science team, is we build the safety and evaluation stack behind Dora, which is a clinical conversational agent.
- 0:49
And my job, and our job within the science team, is proving that the product is safe be-before a patient ever actually hears it.
- 0:58
So shipping to patients takes away the normal safety nets you would normally ship with. Um, three of them could be that you can't actually A/B test on patients, of course.
- 1:06
Randomizing patients into a worse variant is unethical and often illegal. Um, you can't undo a call. Once Dora says it, it's been said, and there is no rollback. And very importantly, the model card won't save you.
- 1:20
Um, you can't claim like some model vendors said that they have ninety-two percent on some benchmark. Um, it's not a defense at a post-incident review. So everything that we do at Ufonia comes from living within these three constraints.
- 1:34
So what is Dora? Um, a little bit about Dora. Dora is a, a voice AI agent that phones patients and has real clinical conversations like post-op follow-ups or pre-op checks.
- 1:43
Um, these are calls that a clinician would usually have to make themselves, but Dora doesn't re-- So Dora obviously doesn't replace doctors, but it just takes a hugely time-consuming task off their plate and gives them their time back.
- 1:55
Um, so far, we've done around two hundred thousand real clinical calls within the UK across twenty hospitals, and we are contracted to scale to a million patients in the next two years.
- 2:04
Um, we also, um, released our product in the US this year, and we are live in, in, in two clinics, and we've signed up for six more clinics across four different states.
- 2:15
So I'm just gonna, for, for a bit of better context, I just wanna do a quick de-demo of what the experience with Dora is actually like.
- 2:22
That's not uncommon after cataract surgery. Are you having trouble seeing clearly at a distance, up close, or both?
- 2:30
Um, at a distance is fine. It's just, um, up close that, um, it's a bit... It's-- I'm having a bit of trouble with that.
- 2:40
So when exactly did you first notice it was a bit blurry when trying to read or see things up close?
- 2:47
Um, well, really just, uh, after the surgery.
- 2:53
And I was wondering-
- 2:54
Okay.
- 2:54
-how long it would last.
- 2:57
So you're asking about the duration. That's understandable. Blurry vision should improve rapidly in the first few days after surgery.
- 3:04
Okay. Um, also, um, is it all right if I go swimming again?
- 3:09
You should avoid swimming for a month after surgery. That's just to be on the safe side and help your eye heal properly.
- 3:16
Ah, right. Okay.
- 3:18
Awesome. So as you just heard, sometimes Dora asks about symptoms. It gives advice sometimes. It, it answers questions, and when it does this, this makes it what's called a medical device, and therefore it has to, of course, be regulated.
- 3:34
Um, and we have to prove that it's safe. And regulation essentially bottles down to three main questions: What does your software do? What could go wrong? And how do you ensure that it doesn't?
- 3:45
For normal software, it's quite tractable, the question, but for a model that te-- talks to actual patients, what could go wrong is quite huge.
- 3:53
So where do we start? We start from what could go wrong. We start from the harm. What could actually harm a patient? And let's look at some examples. Um, Dora could miss a red flag symptom, such as sudden vision loss or severe pain.
- 4:06
A patient could ask a medical question, and Dora invents an answer, hallucinates something. The patient could be distressed, and Dora just ignores it and carries on without actually acknowledging the distress.
- 4:16
There's many, many, many documented hazards of these, twenty, thirty, forty, and we have to ensure that none of them actually happen in real life.
- 4:25
So how would we actually normally catch a problem like this before it actually spreads? We would lean usually on the playbook that most software ships on. You ship to a small percentage of people, you watch the dashboard, you roll back if it breaks, and you iterate from there.
- 4:40
This is a very good playbook. It's reactive, it's fast, it's very safe, and it's how, uh, the industry u-usually de-risks a launch. But there's a hidden assumption here that it only works because you can afford to be wrong for an instance.
- 4:52
A bad change hits a few users, you can quickly catch it, you can roll back, and no one's actually literally harmed. This is the one assumption, um, is why that it breaks when, when the actual user is a patient.
- 5:03
For five percent, that could be hundreds if not thousands of, of patients that have got unproven changes and un-undue care. Rollback, you can't really roll back. The call has already happened.
- 5:14
The person has already been harmed. By watching the dashboards, the dashboards are just gonna... going red means that a patient was actually hurt. So the reactive loop is actually gone now.
- 5:23
So how do you iterate at all when you can't touch a patient until you're sure?
- 5:28
Well, for this, we started looking at other... at, at examples from other high reliable industries. Uh, uh, the most obvious one is self-driving cars. Obviously, we're in SF now.
- 5:37
There's a lot of Waymos driving around. They've only just come to London, unfortunately, very, very late to the, to the party. But what did, what did self-driving cars do?
- 5:45
Well, they didn't just drive around crashing into walls and say, "We won't do that again," and then doing another RL loop. They put millions of miles of simulations first before they actually got any pa-- um, passengers in, into the car.
- 5:58
For us, we believe in the same thing. Simulation is only re-- the real ethical option we can go with. You can't run all the hazard-- the hazards I just mentioned on real people as a first grasp.
- 6:10
So for our clinical history-taking, we built a simu-simulation framework called MATRIX. And I'm gonna work through how it works and how we use it to prove that our product is safe.
- 6:19
And, uh, the paper's on Archive if you want to read it, along with some of the other research that we do.
- 6:24
At its core, MATRIX recreates a real clinical work con-conversation, but with no real patient in it. We use an LLM to play the patient. We call it PathBot. And what does it do?
- 6:36
We use a simulated patient and not a hired actor because hired actors don't scale. If we wanna iterate very fast and simulate different things at the same time, while also updating our system, um, hiring actors would just be too slow of a process.
- 6:50
So f- as a first version, we just use a simulated patient.
- 6:54
The simulated patient is conditioned on the actual scenario we want to test. The scena-scenario defines exactly what the patient should try and do when talking to our agent. For example, asking whether the agent is a human or a, or an AI.
- 7:09
PathBot then has a conversation with Dora, our target system, and then ge-generates simulated dialogues. Very importantly, this all happens under a very specific clinical use case context, so the scenarios are grounded in real clinical workflows and not abstract situations.
- 7:25
So how do we actually make sure that the, the patient is realistic?
- 7:31
If, if PathBot is, is sounding robotic, the tests aren't really worth much, so we had to, of course, validate it. The first thing we did was just a pure, um, script adherence check.
- 7:41
If we told PathBot to do something, does PathBot do it? Yes or no?
- 7:46
This helped us filter out a lot of maybe weaker models that, that were-- that didn't listen to instructions properly. But just purely, um, following instructions does not make a realistic patient.
- 7:55
We want a patient that flows more realistically like a, like a real person.
- 8:00
So we set up what's called a PPI study, a patient and public involvement study. We took real patients, and we showed them two sets of conversations. One conversation was w-between a real doctor and a real patient, and one conversation was between Dora and PathBot within our MATRIX framework.
- 8:17
And we showed them these two examples side by side and said, "Looking at the patient, can you tell which one is the real person and which one is the simulated person?"
- 8:24
So I'm gonna just wait for a few seconds here if you guys wanna quickly read the two conversations. Um, maybe we can do a, do a hands up. Who thinks conversation A is the real person?
- 8:35
Who thinks, who thinks conversation B is the real person?
- 8:40
Okay. I think us as, as engineers sometimes are pretty good at, at finding these things, but, um, it was actually much more difficult than we thought. And we did this with four conversation sets, basically.
- 8:51
In three out of the four, the majority of people actually thought that the simulated patient was more realistic. But the most important thing that we found was, of course, there is no single realistic patient.
- 9:01
That doesn't really make sense. Some people prefer to speak more v-verbosely, a lot of ums and uhs. Some people are more straight to the point, a lot of yeses and nos.
- 9:09
But the point is that we actually want to simulate all these different scenarios. We want to simulate people with very diverse personas. Um, but what it did show us is that at least our PathBot was realistic enough, um, for this simulation.
- 9:23
Okay, so now you've got thousands and thousands of simulated dialogues. Are us as the engineers gonna go read through them one by one and see if a hazard happened?
- 9:31
Of course not. It doesn't scale at all, for one, and number two, we aren't clinicians, so we don't actually know if an actual hazard has really occurred.
- 9:39
So we use another LLM as a judge, of course, and we call it BevJudge. It takes the simulated dialogue, a, a set of expected behaviors, and the hazardous scenarios that we talked through with clinicians, and it makes a judgment pass or fail.
- 9:52
If it fails, it gives us a reason why it gave that answer. So we get, we get a structured output of which hazard were triggered and what actually went wrong in that scenario.
- 10:03
So how did we validate BevJudge? Uh, we val- we validated BevJudge against expert clinicians. We had-- we created a corpus of two hundred and forty examples, and we had a ground truth of whether a hazard existed in these conversations, yes or no.
- 10:15
Then we got ten clinicians from ten clinical specialties to, to label them for whether they had a hazard or not, and we did the same thing with the judge.
- 10:22
And the results showed that our judge is at least on par, if not slightly better than the, the real expert clinicians. The top model, which as of a year ago when we wrote the paper, was Gemini 2.5 Pro, now we've maybe updated the models, um, it achieved an F1 score of, of zero point nine six, and m-
- 10:39
even maybe more importantly, it achieved almost perfect sensitivity, um, sensitivity being a very important metric to, to healthcare and, and to clinicians, of course, because you wanna make a hundred percent sure almost that this-- that no hazards appear in a conversation.
- 10:52
You would rather over-call hazards that aren't there than under-call hazards that are there. So now we have an automated judge that performs at expert layer-- at, at expert level, and this is actually what makes this whole process scalable.
- 11:05
Okay, so now MATRIX can grade thousands of conversations, but grading isn't technically improving the product. A pile of pass fails tells you where Dora breaks and where it's not safe but doesn't actually make the product better.
- 11:17
So how do you do this without experimenting on the patient? The answer isn't, uh, like maybe, uh, very long ago in our world, eight, eight months ago, we would manually prompt engineer this.
- 11:27
We would look at which agents are going wrong or which, uh, prompts are going wrong, and you'd have to manually prompt engineer. But we know that prompt brittleness is real, and it's, it's quite absurd.
- 11:37
Formatting changes alone have been seen to swing a benchmark by seventy-six percentage points, and reordering few shot examples flips a model from near random, so near fifty percent, to near state-of-the-art on some benchmarks.
- 11:49
And hand-tuning can't survive that. It's very subjective, it's not reproducible, and very importantly, it's extremely time-consuming. So over the last year or so, there's been these prompt optimizers that have started to come out, and we've focused on those.
- 12:02
The one that we use the most is, is Geppa, which stands for Genetic Pareto. It comes from the same, um, people who made DSVY, if, if anyone knows about them.
- 12:11
And how does JePA work? You essentially define a metric for what good is, which I'll, I'll get into a bit later. Then, um, you, you pass your data through, through, um, it-- through, through JePA and, and it tells you which examples failed.
- 12:26
Then you get a very str- strong LLM to reflect on the failures and update the prompt automatically. You do this over and over and over again, and it, it keeps what they call a Pareto frontier of the best prompts until your budget has been exhausted and you've now come up with, with what, what JePA comes up with
- 12:41
the best prompt. So we believe this is a much better process from both, both a time-consuming process, you know, it takes maybe manual prompt engineering would take in the order of hours to days, to this is an hour of minutes.
- 12:54
Normally between thirty and an hour, minutes and an hour, you get an optimized prompt. And very importantly, it's reproducible and there's a very clear audit trail and clear feedback loop.
- 13:04
And if anything goes wrong, you can just maybe-- it, it's purely now a data science problem. It's ma- mainly focused on the data, how to make your data right, the feature engineering, and you define the actual metric along with the clinicians.
- 13:18
So how do you actually know what good is?
- 13:21
Um, it's not a flat accuracy score. You don't want just an average of how your whole dataset did. You give it a cost matrix. So let's go back to our, our sensitivity metric.
- 13:32
Let's say it's very important for clinicians to understand when, uh, when and where a red flag is present. If a red flag is present and you correctly catch it, that's good.
- 13:41
If you, if you miss it, it is-- it could be catastrophic. If there's no actual red flag and it over-cause, um, um, that there's a red flag there, it's just mildly annoying to the patient.
- 13:52
They may need to ask-- answer a couple extra questions, but it's not a cast-- catastrophic harm sit- situation. So we-- what can we do? We can optimize for sensitivity.
- 14:02
We can, we can work with the feedback metric, and we can just make it, um, give a higher reward for finding the red flags and a lower re- reward for, um, for missing them.
- 14:11
So you can optimize for certain metrics. You can also optimize for something else that a, that a, that a phys-- a clinician might want. They might want to optimize for accuracy or might want to optimize for some other metric.
- 14:21
All you have to do is recompile the prompt, and then you've got a, a new, um, optimized prompt.
- 14:28
So remember earlier when we said the re-- we feel like the reactive loop is gone. You ship, watch, and roll back. This is what we believe replaces it. We take real calls, real data.
- 14:39
We, um, we use then synthetic edge cases which may not come up in real calls, such as like rare symptoms or, or mis-transcriptions.
- 14:46
We get an optimized prompt through, through JePA or some other prompt optimizer. Then we pass it through, through something like MATRIX as a simulation safety gate. If anything fails or a- anything doesn't look right, we can redo that whole process, redo the data or, or relabel or get more data.
- 15:04
And then after that, and you're happy with that, then you can do some gated deploy, which we'll get into a bit. Um, the most important thing here is that it's a flywheel.
- 15:12
Every single deployment and every new call produces more call data, so your system is consistently improving.
- 15:20
So as we said with, with our MATRIX framework, we use simulated patients. But however realistic you think they are, of course, they are not real patients. Passing every test in simulation doesn't prove that Dora actually helps someone in real life.
- 15:33
The things might come up in real life that, that you can't get in simulation. It only earns the right to actually try carefully. Simulation is the inner loop. It's fast, it's free.
- 15:43
You can do thousands of runs before anyone actually real is, is exposed. But real patients are the outer loop, and that's where the only real proof is. So simulation is necessary, but it's not sufficient.
- 15:55
Simulation earns the right to, to, to test on real people and then eventually real patients. But you don't just flip a switch. You cross it in stages, and each stage earns the ri- the right for the next.
- 16:05
After you've done your simulations and you're happy with the results, you might do a round of user testing. Then you get supervised clinical evaluation based on those tests, and, and you, and you base it on real patients.
- 16:16
You can do some voice actors, but of course, the, the most realistic is to get real patients. But in this step, it's very important that there's clinicians at every step in the loop.
- 16:25
Then you can do some deployment, uh, but it's still monitored. And how much an autonomy you allow the system to do depends on your evidence. As the system gets more evidence, you can give it more independence.
- 16:37
And underneath all of this, um, every call, every dataset, and every pinned, um, prompt, every judge verdict traces back to the exact hazard that it addresses. That's the real deliverable.
- 16:48
The important thing is that you don't ship the model, you ship the evidence when trying to regulate.
- 16:53
So what can you take back to your own stacks, both in healthcare and, and other areas? You first have to define exactly what harm is for your product. You have to manufacture your rare but dan-- but dangerous cases.
- 17:03
Don't wait for them just to happen naturally. Make your ev- evaluation metric your real cost function to optimize. Pin your pro- prompt versions and keep the traces. These are the important things.
- 17:15
The important thing also is that the, the work is never done. As you move into new modalities or new languages, there will always be new hazards that start to arise.
- 17:22
But with something like MATRIX and a, and a prompt optimization loop, um, you can use the same framework as new modalities arise. You know, when you move into voice, for example, there's things like backchanneling and interruptions, which, which breaks a lot of the, the text evals.
- 17:38
For example, an agent might be mid safety advice, like, "You must avoid bright lights and..." When the patient maybe, um, um, cuts in with some, um, o- out of scope question, and weaker models usually just forget about the safety advice and just ask-- answer the next question.
- 17:54
But MATRIX captures these things. Um, o- other things that, that usually go wrong that we've tested is that, um, the, the patient could-- the, the Dora or, or an agent could be halfway through giving some safety advice and a backchannel just, just cause the model to just completely ignore the safety advice and, and stop right there and,
- 18:12
and wait for the patient to, to say something else. The failure models-- the failure modes change, the framework doesn't. You still black box the system, write down the new hazards, and simulate and judge them exactly the way that we did over text.
- 18:25
Voice is just a new module in the same safety case, and we actually use MATRIX in the voice space to do these exact same things. So whatever modality you move into, you're not starting over.
- 18:34
The same approach finds the hazards before a real user does.
- 18:38
So again, you simulate before it ever touches a patient, and you use an optimization loop to actually improve the system. Thank you very, very much for coming to listen.
- 18:47
Um, I'm very, very open to, to talking more about this. And if you want to connect during the conference or after, there's my LinkedIn and socials. Thank you very much. [audience applauding] [upbeat music]