# From Ambient Documentation to Clinical Intelligence

Chaitanya Asawa, Abridge

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/u6q-byPWUuo-from-ambient-documentation-clinical-intelligence) · [Video](https://www.youtube.com/watch?v=u6q-byPWUuo)

## From Ambient Documentation to Clinical Intelligence

Abridge’s path from writing visit notes to contextual clinical assistance exposes the engineering demands of physician-calibrated evaluation, specialized models, and live order matching.

Basic familiarity with language models and software workflows is helpful; no clinical training is assumed.

### Finishing the visit without carrying the notes home

[0:53](https://www.youtube.com/watch?v=u6q-byPWUuo&t=53s)

A clinician in Abridge’s opening testimonial reports finishing a 22-patient day by 4:30 p.m., with notes complete. Another describes the tool as indispensable to their practice. Others describe a change during the encounter itself: they can listen and look at the patient without continually deciding what to remember or enter into the computer. The benefit extends beyond the clinic to getting home afterward. These are individual accounts of the experience, rather than a controlled productivity measurement.

### From promising technology to clinical adoption

[2:08](https://www.youtube.com/watch?v=u6q-byPWUuo&t=128s)

Healthcare can still run on fax machines while presenting some of the hardest problems in applied AI. That combination frames Chaitanya Asawa’s move into the field: the opportunity is both to improve an established workflow and to build clinical intelligence around it.

Asawa began in research engineering at Vicarious, where the pursuit of artificial general intelligence involved neuroscience-inspired methods, probabilistic graphical models, and robotics. He then joined Glean to work on a problem he had experienced firsthand: workplace information scattered across systems, even though that context was essential to decisions. Asawa reports spending about six and a half years at Glean as it grew from ten people to more than 1,100.

His interest in healthcare came from a different contrast. He remembers seeing AI for skin-cancer detection on a Nature cover, then receiving a bill after a minor hospital visit and struggling to understand what he had paid for. Advanced technical capabilities existed alongside basic problems of cost and access. A seminar he organized with physicians, researchers, and entrepreneurs revealed how little of that promising technology was reaching clinics at the time.

When he returned to the field years later, the deployment picture had changed. Asawa reports that [Abridge](https://www.abridge.com/press-release/patient-centered-clinician-intelligence-platform-keynote) reached 300 US health systems in two to three years, naming Kaiser, Mayo, Johns Hopkins, and Sutter among its customers. Getting inside those systems created opportunities beyond documentation. He now leads engineering for clinical decision support and agentic experiences, where existing clinical context can inform assistance throughout a visit.

![Journey slide with three columns labeled Vicarious, Glean, and Abridge, describing AI and robotics, enterprise assistance, and healthcare AI.](https://ai.engineer/api/talk-essay-media/17aaf98b73069533c8fb0b219be076b4fafcae3f7fbb575e106d2ee7c2b8e9c4)

A career journey through Vicarious, Glean, and Abridge.

### Why the clinical note was the entry point

[5:25](https://www.youtube.com/watch?v=u6q-byPWUuo&t=325s)

The economic pressure is broader than documentation. In many industries, productivity improvements reduce the cost of producing goods. Asawa contrasts that pattern with rising healthcare administrative costs and a reliance on additional labor that cannot keep pace. He connects this partly to **Baumol’s cost disease** and partly to technology’s limited effect on healthcare productivity. Thin hospital margins, closures, patient debt, and clinician burnout make the consequences tangible; some doctors even discourage their children from entering the profession.

Clinical documentation offers a specific place to intervene. After each patient visit, a clinician must produce a note. A common structure is the **SOAP note**—subjective, objective, assessment, and plan—with details and formatting that vary by specialty. It records the patient’s complaint, the clinician’s assessment, and what should happen next. Asawa estimates that clinicians often spend about two hours a day writing notes, including after-work “pajama time.”

**A clinical note is both an administrative record and an input to future care.** It supports billing, but it also gives the next clinician context about the patient’s history. That context may need to travel across health systems. An error therefore has consequences beyond a poorly written document: it can affect financial records and the longitudinal medical record used in subsequent decisions.

Abridge started with this longstanding pain point because the technology had become capable of generating high-quality notes personalized to the clinician. Asawa describes documentation as an effective entry into a technology-reticent industry: clinicians already cared deeply about getting these records right. He credits the product with improving provider satisfaction, freeing capacity to see patients, and producing better records for billing and audits.

### Building around the doctor–patient conversation

[8:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=508s)

Asawa credits the documentation product alone with the adoption described earlier. The next step follows from where its information originates: the doctor–patient conversation. The note is one output, but billing, clinical-trial matching, and clinical decision support also depend on what happens in that encounter. Much of the administrative machinery surrounding care exists to carry information from the conversation into another workflow. **Capturing the conversation creates a foundation for assisting those downstream workflows.**

![Nested coral rectangles connect conversations to clinical notes, coding and claims, care management, clinical trials, decision support, and clinical outcomes.](https://ai.engineer/api/talk-essay-media/a9ce64735daedd6d694347ff0f691d9a320890d1f23aba0fa7258f54dd90257e)

From clinical notes to broader work built around the conversation.

### From trial eligibility to preparing tomorrow’s charts

[9:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=568s)

The recorded demonstration begins with a clinician invoking “Hey, Abridge” and asking whether Nathan is eligible for any clinical trials. The assistant says he may qualify for the Abridge HF study: he remains symptomatic despite maximal therapy, and most screening criteria are already met. But eligibility is incomplete. An updated echocardiogram is needed to confirm his ejection fraction. The clinician requests that echo, and the demo responds, “Confirmatory echo ordered.” The next request moves beyond the current patient: prepare tomorrow’s charts.

The broader workflow covers both sides of the encounter. Before the visit, Abridge suggests discussion topics, including clinical and billing concerns. Afterward, it generates the patient visit summary and clinical note and prepares pended orders. The demo’s order confirmation should be understood in that review workflow: orders are prepared for clinician review and signature, rather than autonomously executed.

Three sources of context make this assistance possible:

- **The electronic health record (EHR):** prior laboratory results, notes, and other patient history.
- **The live conversation:** what the patient is experiencing now, which Asawa likens to the place where healthcare debugging happens.
- **Medical literature and clinical guidelines:** sources against which patient-specific reasoning can be grounded.

The EHR supplies history, the conversation supplies the present encounter, and medical evidence supplies a basis for interpreting them together.

### Making evaluation part of the operating system

[11:05](https://www.youtube.com/watch?v=u6q-byPWUuo&t=665s)

The engineering objectives are familiar—**quality, latency, and cost**—but clinical assistance makes all three demanding. An incorrect workplace answer might be recoverable with limited consequences; an incorrect clinical answer can cause harm and destroy trust. During a live visit, an answer must also arrive while the information is useful, at an appropriate point in the conversation. Finally, inference must remain affordable at deployment scale. Asawa maps these objectives to the company’s aim to save lives, save time, and save money.

Abridge treats evaluation as an operating system for development, not a test performed just before launch. The sequence spans the product lifecycle:

1. **Build internal benchmarks before developing the product.** Trial matching, note generation, decision support, and coding each need a robust reference set.
2. **Evaluate offline against cases encountered in practice.** This establishes a starting point without assuming the benchmark captures every real-world condition.
3. **Release to a trusted clinician alpha group, then beta.** Expand exposure gradually, with A/B testing at scale where appropriate.
4. **Continue monitoring after full rollout.** A few successful prototype cases do not establish reliable behavior in clinical practice.

The staged release matters because offline evaluation cannot perfectly represent what happens when a product meets real encounters.

Clinicians embedded throughout the company calibrate **LLM judges** to encode the behavior they want from the product. This gives engineers who are not clinicians a feedback loop for improving it. The judges act as a reusable expression of domain expertise, while online signals add evidence from actual use: edits to generated notes, thumbs up or down, star ratings, and free-form feedback. Together, these signals support iteration throughout development and deployment.

### Answering a question whose context is unstated

[14:50](https://www.youtube.com/watch?v=u6q-byPWUuo&t=890s)

Consider the provider’s question: “Does this patient meet the criteria for febrile neutropenia?” The request leaves much of its context unstated. A contextual decision-support system must retrieve previous laboratory values from the EHR, incorporate the live conversation, and use clinical guidelines and medical journals as reasoning sources. Answering the question requires assembling patient-specific evidence, not simply recalling a definition.

![White question-and-answer panel asking whether a patient meets febrile neutropenia criteria and needs hospital admission, with paragraphs of clinical reasoning below.](https://ai.engineer/api/talk-essay-media/1f2c30cd8cf5ff74d43c81bdd26ac4c7eae3bf83ed79eb8724a31d59dbf63709)

A clinical question about febrile neutropenia and hospital admission, followed by a detailed response.

Evaluating that answer introduces a second hard problem. Sudoku has a useful asymmetry: finding a solution can be difficult, but checking a completed grid is straightforward. Clinical reasoning has a much smaller **generator–verifier gap**. A verifier capable of reliably determining the correct answer may need much of the same ability as the generator. Simply asking another language model whether the answer is right does not provide a sufficiently independent reference.

Abridge therefore uses multiple judges, each examining a different aspect of the response:

| Judge | Evaluation focus |
| --- | --- |
| Clinical quality | Whether the response contains the right clinical content |
| Boundary and adversarial | Behavior under boundary-testing and adversarial inputs |
| Clinical safety | Safety of the response |
| Tone and style | How the answer is expressed to the clinician |

These are complementary signals. Each captures part of a quality problem that is difficult to measure with a single verdict.

### Human references without one golden answer

[16:49](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1009s)

For the clinical-quality judge, the independent reference comes from physicians. A single golden response would be too restrictive: many differently worded answers can be clinically valid. Instead, Abridge starts with real clinical cases and asks physicians to specify the elements a good answer should contain. The reference is a **rubric**, rather than a paragraph the model must reproduce.

Asawa describes a four-clinician process:

1. Two physicians independently create rubrics for the case.
2. A separate physician adjudicates their work and combines it into a final rubric.
3. A fourth clinician performs quality assurance on that rubric.

An LLM judge then compares the agent’s response with the required elements using semantic matching. This preserves room for valid variation in wording while giving developers concrete feedback about what the response includes or misses.

### Training for smaller clinical workflows

[17:55](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1075s)

Rubric-based evaluation provides a target for improving agent architecture, models, and search ranking. Cost and latency then determine how that quality can be delivered during a live conversation. Asawa reports an annualized run rate of 100 million medical conversations. That is a measure of reported operating scale, not a count of completed encounters or a disclosed training-corpus size.

Clinical-note generation illustrates the opportunity for specialization. A note contains distinct sections, including history of present illness, past medical history, and assessment and plan. Instead of asking one foundation model to generate everything, Abridge decomposes the task into smaller workflows and post-trains smaller models, sometimes at the level of an individual section. A narrower task can require less general capability, allowing smaller models to serve it at lower cost and latency.

Asawa connects this strategy to a data flywheel from clinical conversations. He describes Abridge’s data advantage as unusually large, qualifying the comparison with what the company knows about other datasets. The strategic question is whether the team has distinctive data and sufficient focus to make model training worthwhile.

There are two different reasons to train:

- **Quality is already sufficient:** preserve that quality while reducing cost and latency.
- **Quality still needs improvement:** use distinctive data and a tightly defined clinical task to pursue quality gains faster than general frontier-model progress.

The second is a potential advantage, not a demonstrated guarantee of outperforming frontier models. The key is the combination of relevant data and concentrated effort on a problem that a general model developer may not prioritize.

### Spend inference when the conversation calls for it

[19:49](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1189s)

In-visit orders expose another way to control inference cost. Clinicians mention medication and non-medication orders during the encounter. Abridge listens, queues those orders in the background, and leaves them for the clinician to sign off in the EHR. The goal is to remove the work of manually preparing orders while retaining the clinician’s approval step.

Invoking expensive order processing every few seconds would waste computation. Instead, the system looks for conversational events that justify heavier processing. Detecting that an order was mentioned is only the first task: the system must also match that mention to an approved, relevant order in the health system’s catalog. Cheap, fast gates determine when to hand the work to larger models.

An illustrative TypeScript interface makes the separation explicit. The gates decide whether a conversation event deserves more work; the matcher resolves it against the approved catalog; the output remains pending signature. The gate and matching implementations are separate dependencies because they perform different jobs.

```typescript
type ConversationEvent = {
  text: string;
  encounterId: string;
};

type CatalogOrder = {
  id: string;
  name: string;
};

type PendingOrder = {
  encounterId: string;
  catalogOrderId: string;
  status: "pending-clinician-signature";
};

type OrderPipeline = {
  gates: Array<(event: ConversationEvent) => Promise<boolean>>;
  match: (
    event: ConversationEvent,
    approvedCatalog: readonly CatalogOrder[],
  ) => Promise<CatalogOrder | null>;
};

async function proposeOrder(
  event: ConversationEvent,
  approvedCatalog: readonly CatalogOrder[],
  pipeline: OrderPipeline,
): Promise<PendingOrder | null> {
  for (const gate of pipeline.gates) {
    if (!(await gate(event))) return null;
  }

  const match = await pipeline.match(event, approvedCatalog);
  if (!match || !approvedCatalog.some(order => order.id === match.id)) {
    return null;
  }

  return {
    encounterId: event.encounterId,
    catalogOrderId: match.id,
    status: "pending-clinician-signature",
  };
}
```

This structure separates the decision to spend inference from the harder task of resolving an order, and separates both from clinical authorization.

The closing invitation is to take healthcare seriously as an AI engineering domain. Asawa returns to his own earlier skepticism about whether promising technology could actually work inside clinical practice. He presents Abridge’s adoption as evidence that it can. The remaining work demands frontier capabilities under unusually high stakes: reliable clinical reasoning, evaluation grounded in physician judgment, and assistance that arrives affordably at the moment a clinician can use it.

## Resources

- [Abridge's patient-centered clinical intelligence platform](https://www.abridge.com/press-release/patient-centered-clinician-intelligence-platform-keynote) — Company announcement describing pre-visit, in-visit and post-visit workflows, clinician review, and reported deployment scale.
- [From Ambient AI to Clinician-Directed Intelligence](https://www.abridge.com/blog/ambient-ai-prompt-editing) — Chaitanya Asawa and Matt Troup explain prompt editing, clinician control and contextual decision support.
- [How Abridge evaluates clinical decision support](https://tech.abridge.com/blog/how-we-evaluate-clinical-decision-support) — A technical account of physician-built rubrics, safety and adversarial evaluations, staged deployment and ongoing feedback.
- [Innovation at the Speed of Trust](https://www.abridge.com/blog/clinical-ai-specialty-models) — How specialty documentation models move through clinician review, automated evaluation, staged rollout and online monitoring.
- [Allocation of Physician Time in Ambulatory Practice](https://adfm.org/media/1476/ann-2016-time-study.pdf) — An observational study of physician work across four specialties, including after-hours EHR and clerical work recorded in diaries.
- [AI-Native Healthcare — Janie Lee and Chai Asawa](https://www.latent.space/p/abridge) — A longer interview and transcript exploring Abridge's models, evaluations, infrastructure and clinical workflows.

## Slides

### Logo screen

[2:06](https://www.youtube.com/watch?v=u6q-byPWUuo&t=126s) [A white geometric mark centered on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad634ff4a23db27fd8cd5580c9950e56ef3d1953faffde4b34a270ace7a42802.webp)

Interface text:
2

### Healthcare and frontier AI

The talk follows Abridge from clinical documentation to clinical intelligence and explores frontier AI problems in a high-stakes setting.

[2:13](https://www.youtube.com/watch?v=u6q-byPWUuo&t=133s) [A statement says healthcare needs frontier AI and tests it at the highest stakes.](https://assets.ai.engineer/sites/aiecode2025/sha256/41aa3905ee27d3f284b8c3c3c9ed71e0d3a4537addf6382a0631f286ae40afcd.webp)

Healthcare is a domain that both needs frontier AI and puts it to test at the highest stakes

Interface text:
3

### Journey

Chaitanya Asawa moved from AI and robotics at Vicarious to enterprise search at Glean. At Abridge, he leads engineering for clinical decision support and agentic experiences.

[2:56](https://www.youtube.com/watch?v=u6q-byPWUuo&t=176s) [Three cards outline work at Vicarious, Glean, and Abridge.](https://assets.ai.engineer/sites/aiecode2025/sha256/98834bf2c74009c940b75ca0197eb2c06acf63759029378cf178f328b79f6a86.webp)

Journey

vicarious
Vicarious
AI & Robotics
• Foundation in real-time, multi-modal AI systems
• Intelligent automation

glean
Glean
Enterprise Assistant
• Founding Engineer
• Context-aware search infrastructure

ABRIDGE
Abridge
Healthcare AI
• Lead Clinical Decision Support & agentic experiences
• High-stakes AI work

Chaitanya Asawa
Career Journey
July 2026

Interface text:
4

### Healthcare Productivity: The Last Frontier

Healthcare spending rises while labor productivity ends near its starting level. The second chart shows administrative growth outpacing physician growth from 1975 to 2010.

[5:24](https://www.youtube.com/watch?v=u6q-byPWUuo&t=324s) [Two charts show healthcare expenditure and labor productivity, and growth in administrative staff and physicians.](https://assets.ai.engineer/sites/aiecode2025/sha256/077cb4caf65dd67609f09ac8a187f2126637aff45a6b47437b89def3e5484478.webp)

State of Healthcare
Healthcare Productivity: The Last Frontier

TWO-DECADE PRODUCTIVITY PARADOX
Healthcare Labor Productivity²
National Healthcare Expenditure¹ ($bn)
Y-axis ticks, left: 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140
Y-axis ticks, right: $0, $1,000, $2,000, $3,000, $4,000, $5,000, $6,000
Timeline: 1995, 2000, 2005, 2010, 2015, 2020, 2023

SUSTAINED SHIFT TOWARD ADMINISTRATIVE INTENSITY
Admin. Growth³
Physician Growth³
1982 Prospective Payment System (DRGs)
1995 HIPAA
2009 HITECH Act
Y-axis ticks: 0%, 500%, 1,000%, 1,500%, 2,000%, 2,500%, 3,000%, 3,500%
Timeline: 1975, 1980, 1985, 1990, 1995, 2000, 2005, 2010

1 Centers for Medicare & Medicaid Services: National Healthcare Expenditure Data;
2 U.S. Bureau of Labor Statistics: Labor Productivity, Output, and Hours Worked for Private Community Hospitals;
3 Physicians for a National Health Program (Derived from Bureau of Labor Statistics, National Center for Health Statistics and Census Bureau's Current Population Survey Data)

Interface text: 5

### Clinical recording on a phone

A phone displays a recording interface as the speaker describes clinical documentation: clinicians often spend around two hours a day writing notes, sometimes after work.

[6:35](https://www.youtube.com/watch?v=u6q-byPWUuo&t=395s) [Gloved hands hold a phone displaying a recording interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/22996659adabfbc3e495852050f414a1439951e68c1566e65ce0478f79e66766.webp)

Phone screen: [illegible]

Interface text:
7

### From Conversation to Clinical Outcomes

Starting with clinical notes, the diagram maps higher-value, higher-effort uses of conversations through coding, care management, trial matching and decision support to clinical outcomes.

[8:34](https://www.youtube.com/watch?v=u6q-byPWUuo&t=514s) [A value-versus-effort diagram maps clinical conversations to notes, claims, care management, trials, decision support, and outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/35054d132ae253416fed32383a2592b6c061ffbfcf1f831fd13a437cd1401f4b.webp)

Conversation to Clinical Outcome
Follow through from every medical conversation

In-Conversation Clinical Decision Support
Improve quality of patient care

Conversation to Clinical Trial
Match patients to trials

Conversation to Care Management
Coordinate care across clinician+patient teams

Conversation to Code / Claim
Automate codes, costs, and claims to drive quality in clinical documentation

Conversation to Clinical Note
Draft clinical notes (including pulling prior context) to reduce documentation burden

Y-axis: Value
X-axis: Time/Effort

### Agent KPIs

Healthcare raises the stakes for agent quality: errors can harm trust and patients. Latency matters when an agent must act during a conversation, while cost matters at scale.

[11:25](https://www.youtube.com/watch?v=u6q-byPWUuo&t=685s) [Agent KPIs: quality, latency and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/a88f4499f0d6dd19110d0203975486cef84efc05722b4eef97531d94cf9a6afc.webp)

Agent KPIs

• Quality
• Latency
• Cost

### Evals as the operating system

Abridge moves from internal benchmarks and backtesting to staged A/B testing and continual monitoring. Clinician-calibrated LLM judges, user feedback and edit logs provide quality signals.

[12:47](https://www.youtube.com/watch?v=u6q-byPWUuo&t=767s) [Evaluation runs from internal benchmarks and backtesting through A/B testing and continual monitoring, with clinician review, calibrated judges, user feedback and edit logs as quality signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/1da63da82c0b2ae0ae713b782e18d8cb7bcb21b6162cb2d3c4ab209480878fe7.webp)

Evals as the operating system

EVALUATION ACROSS THE SYSTEM LIFECYCLE
Internal Benchmarks
Backtesting
A/B Testing
Continual Monitoring

Pre-Deployment
Staged Rollout
Post-Deployment

QUALITY SIGNALS
→ Clinician review
→ Expert-calibrated LLM judges
→ User feedback (star ratings, tthumbs up/down, free text)
→ Edit logs

ABRIDGE
CONFIDENTIAL & PROPRIETARY

### Clinical decision support example

The example weighs fever, neutropenia and admission. The speaker describes using EHR labs, the live conversation, guidelines and journals to answer an underspecified question.

[14:49](https://www.youtube.com/watch?v=u6q-byPWUuo&t=889s) [A clinical assistant answers a question about febrile neutropenia, admission and antibiotics; the response continues beyond the visible area.](https://assets.ai.engineer/sites/aiecode2025/sha256/871c05f451391bcfd7d37c511797c8918e36c3859374a783a0dcbb828b8a5d26.webp)

Does this patient meet criteria for febrile neutropenia? Any indication for hospital admission? Abx?

Yes—she meets a practical febrile neutropenia syndrome definition, because she had a single fever of 38.4°C and neutropenia with ANC 0.8 ×10³/µL while on palbociclib for stage IV ER+/PR+/HER2− breast cancer with bone metastases. Strict IDSA-style “severe neutropenia” is usually ANC <500/µL or expected to fall below 500/µL within 48 hours; her ANC of ~800/µL does not meet that threshold unless a further decline is anticipated, but many oncology pathways treat fever with ANC <1000/µL as febrile neutropenia requiring prompt evaluation and empiric antibiotics. 1 2

Admission is not automatic if she is clinically stable and low risk, but risk-stratify formally. She appears potentially low risk based on available data: outpatient presentation, solid tumor, HR 78, RR 18, SpO₂ 96%, normal renal function with creatinine 0.8 mg/dL, and fatigued but in no acute distress. ASCO/IDSA recommend using clinical judgment plus a validated tool such as MASCC or CISNE; MASCC ≥21 supports outpatient management, whereas MASCC <21, CISNE ≥3, clinical instability, significant comorbidity, pneumonia/complex infection, renal/hepatic dysfunction, or anticipated prolonged/profound neutropenia favor admission. 3 4 Given metastatic cancer and current fever on therapy, admission would be reasonable if there is unreliable follow-up, inability to tolerate PO, concern for evolving infection, worsening counts, positive cultures, or clinician concern for progression/clinical deterioration.

Antibiotics are indicated now after cultures—do not wait for imaging or final

### Clinical Decision Support Evaluation

Evaluation combines a rubric-based clinical quality judge with adversarial, safety, and tone judges. Safety positives are checked with human annotation after major changes; the tone judge uses pairwise comparisons.

[16:20](https://www.youtube.com/watch?v=u6q-byPWUuo&t=980s) [Evaluation uses clinical quality, boundary and adversarial, clinical safety, and tone and style judges.](https://assets.ai.engineer/sites/aiecode2025/sha256/a31e7125896cde0f61401f1a7059d9cdaf2088f9ef710f232cd456b2abe9a0e0.webp)

Clinical Decision Support Evaluation

• Clinical quality judge
  ○ Rubric-based
• Boundary / adversarial judge
  ○ Medico-legal edge cases
  ○ Non-discrimination
  ○ Ethical expectations
• Clinical safety judge
  ○ LLM judge run on same dataset as clinical quality judge
  ○ Check positives w/ human annotation after any major change
• Tone and style judge
  ○ LLM judge + regex criteria
  ○ Runs as a pairwise comparison judge
• ... and more

### Clinical Quality Judge

Two physicians independently review 1,025 de-identified cases and draft rubrics and responses. A third physician resolves discrepancies to produce the final rubric and golden response.

[16:48](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1008s) [Diagram of independent physician reviews followed by adjudication to produce a final rubric and response.](https://assets.ai.engineer/sites/aiecode2025/sha256/f213db48297e65c03f66657cfcbd8134f51f52488770db0d43abb95bcc9cfedd.webp)

Clinical Quality Judge

1 INITIAL INDEPENDENT REVIEW
1025 DE-IDENTIFIED CASES & CDS QUESTIONS
Physician 1
Physician 2
CASE
CASE
PHYSICIAN A DRAFT RUBRIC
PHYSICIAN B DRAFT RUBRIC

Physician A
Review cases and create initial golden response & rubric.

Physician B
Review cases and create initial golden response & rubric.

2 ADJUDICATION
ADJUDICATION PHYSICIAN
ADJUDICATION PHYSICIAN

Adjudication Physician
Reviews initial outputs and resolves discrepancies to finalize rubric & response.

3 FINAL GOLDEN RESPONSE & RUBRIC

Final Output
The official, high-quality clinical rubric and golden response for medical evaluation.

PROCESS & SCALE OVERVIEW
Dataset Scale: 1025 De-identified Cases (CDS Questions)
Initial Reviews: 2 Independent Physicians
Reviewer Output: Initial Rubrics & Golden Responses
Final Adjudicator: 1 Additional Physician
Final Outcome: Adjudicated Rubric & Golden Response

### Example Clinical Rubric

This hormone-therapy example lists eight elements of a good response. An LLM judge compares the agent’s responses with those elements to assess performance as the system changes.

[17:32](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1052s) [A question about hormone therapy followed by eight rubric elements covering side effects, risks, interactions, and references.](https://assets.ai.engineer/sites/aiecode2025/sha256/8d7e52adf07d7e700dc4b0e77ba2410a6cfcdc62b05d9dff51098cc00e905a3f.webp)

Q: What are the potential interactions or side effects of continuing Estrace and Prometrium in managing hot flashes for this patient?

Rubric (8 elements):
- Common side effects of oral estradiol + micronized progesterone (breast tenderness, fluid retention, sedation)
- Increased age-related VTE and stroke risk with oral estrogen in a 73-y/o
- Progesterone required for endometrial protection (intact uterus)
- Any postmenopausal bleeding requires evaluation
- Increased breast cancer risk with prolonged combined E+P therapy
- Recommend continued annual mammography on prolonged combined therapy
- CYP3A4-mediated drug interactions affecting estradiol metabolism
- Provide a reference for interactions/side effects of oral estradiol + progesterone

### Cost and Latency at Scale

Processing live conversations makes cost and latency difficult at a run rate of 100 million medical conversations per year.

[17:59](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1079s) [Cost and latency considerations for live processing at a run rate of 100 million medical conversations per year.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8b763a85fd70ab94cac74301c49b45b6a7c24dd58b75aa8c04b4dd1fb90eca7.webp)

Cost / Latency

• Doing this live in conversation
• ...for run rate of 100 million medical conversations/year

### Clinical Note Generation

To manage cost and latency, the speaker breaks clinical note generation into smaller workflows and post-trains smaller models for individual sections.

[18:18](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1098s) [Cost and latency slide with a screenshot of a clinical note interface showing history and other note sections.](https://assets.ai.engineer/sites/aiecode2025/sha256/cee1f08f535a6d527419ed879e91d18d0164a52ae5860fc9a2232fdf07d05f8f.webp)

Cost / Latency

History of Present Illness

The patient presents to discuss a proposed two-way communication and documentation tool for CNAs and family caregivers.

They discuss product concepts to reduce cognitive overhead of caregiving documentation and improve transparency between CNAs and family caregivers. They consider an always-on ambient recording approach using vision or sound with minimal prompting, but note potential trust concerns and that it is not appropriate for an MVP.

They favor an MVP centered on a phone app with a basic web portal, with possible small hardware that syncs with the phone. They outline a two-role system with separate CNA and family caregiver logins and largely similar views, including a profile header, a central add function allowing typed entry or recorded and dictated entry, and a timeline of caregiving actions. They note documentation requirements may vary by state and describe a need for structured, template-based note-taking, potentially organized into tabs for structured templates versus other data, with a calendar or day switcher to review events across dates.

They propose bidirectional messaging between CNA and family caregiver, including the ability to comment on specific timeline items similar to document-style comments. They identify concern that increased messaging could add burden for CNAs and suggest mitigation via availability controls, including limiting messaging to working hours, showing online or away status, and clarifying which staff member is on shift when multiple caregivers cover 12-hour rotations.

Past Medical History
Click to add content to this section

Social History

Interface text:
WORKLIST
Label your note
DICTATE
CLINICAL NOTE
ADD PATIENT SUMMARY (PVS)
REDRAFT
THEY/THEM
SHE/HER
HE/HIM
THEY/THEM
COPY ALL
MARK AS DONE

### Smaller Models for Healthcare Workflows

The speaker breaks clinical notes into specific workflows, even individual sections, and post-trains smaller models for them to reduce cost and latency.

[18:39](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1119s) [Healthcare comprises many specific workflows, motivating post-training of smaller models.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea0bf263a562c90f838be181a91c31e782e61eab02289673661ca490e7349335.webp)

Cost / Latency

• Insight: Healthcare is many specific workflows.
• => Post-train smaller models

### Right to Win in Training Models

When quality is maxed out, training can reduce cost and latency. When it is not, the speaker argues that distinctive data and focus may enable quality gains against frontier models.

[19:17](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1157s) [Training models can reduce cost and latency when quality is maxed out; otherwise, data and focus may enable quality gains.](https://assets.ai.engineer/sites/aiecode2025/sha256/58727525d8c2d01f0af96202b8b563b7548068db17f93138d837d9914895e535.webp)

Right to win in training models

• Quality is maxxed out => cost/latency reduction
• Quality isn’t maxxed out => do we have the data/focus to have the right to win?

### Healthcare and frontier AI

Asawa closes by saying healthcare needs frontier AI and tests it at the highest stakes. He says Abridge has shown that healthcare technology can work at scale.

[20:45](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1245s) [Healthcare needs frontier AI and tests it at the highest stakes.](https://assets.ai.engineer/sites/aiecode2025/sha256/3128f83027a69efa16b54da8bf84421977e0b9a15aa34206ed42e3e5cc36aa20.webp)

Healthcare is a domain that both needs frontier AI and puts it to test at the highest stakes

### Contact

Chaitanya Asawa closes by inviting attendees to connect on Twitter or LinkedIn.

[21:09](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1269s) [Presenter contact slide with social and professional profile details.](https://assets.ai.engineer/sites/aiecode2025/sha256/107dbf0ddcafddfdc7fffe5dadc893f1f6bb3738ffb17472c3d7926eef954144.webp)

Contact

Chaitanya Asawa

Twitter: c_asawa
Linkedin: https://www.linkedin.com/in/casawa

## Full transcript

[0:00](https://www.youtube.com/watch?v=u6q-byPWUuo&t=0s) [upbeat music] Thank you so much for everyone being here. We're gonna get started in a second.

[0:16](https://www.youtube.com/watch?v=u6q-byPWUuo&t=16s) Um, but before we get started, I am curious, how many of you currently work in the healthcare industry in some shape or form? Oh, that's amazing to hear. Uh, how many of you are clinicians by training?

[0:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=28s) Okay, a couple. How many people in the room are engineers?

[0:31](https://www.youtube.com/watch?v=u6q-byPWUuo&t=31s) Okay, awesome. Um, and then how many people have heard of Abridge before?

[0:37](https://www.youtube.com/watch?v=u6q-byPWUuo&t=37s) Okay, awesome. Uh, well, I'm gonna let you hear actually from our users to start off on a little about Abridge. [upbeat music]

[0:53](https://www.youtube.com/watch?v=u6q-byPWUuo&t=53s) Full day of twenty two patients, out by four thirty PM, notes done. That's nice.

[1:01](https://www.youtube.com/watch?v=u6q-byPWUuo&t=61s) When I think about Abridge, I think the thing that comes to mind is it's really a cornerstone of how I practice medicine today. Um, there's just no way, um, I would do a clinic or see a patient without using.

[1:14](https://www.youtube.com/watch?v=u6q-byPWUuo&t=74s) I can be present throughout my clinical encounters. I don't have to think about, um, "Oh, wait, did I get that? Do I need to write that down?" Because I know Abridge has my back and has everything ready for me.

[1:25](https://www.youtube.com/watch?v=u6q-byPWUuo&t=85s) Abridge makes me feel free, 'cause I can really look at a patient, really listen, and not have to be thinking about, "What do I need to put in the computer?"

[1:34](https://www.youtube.com/watch?v=u6q-byPWUuo&t=94s) Full day of twenty two patients, out by four thirty PM, notes done.

[1:37](https://www.youtube.com/watch?v=u6q-byPWUuo&t=97s) I get to go home and protect my family.

[1:38](https://www.youtube.com/watch?v=u6q-byPWUuo&t=98s) I don't have to think about, "Oh, wait, did I get that? Do I need to write that down?"

[1:40](https://www.youtube.com/watch?v=u6q-byPWUuo&t=100s) I can really look at a patient, really listen, and not have to be thinking about, "What do I need to put in the computer?" [upbeat music]

[2:08](https://www.youtube.com/watch?v=u6q-byPWUuo&t=128s) Our marketing team produces really good videos, and so they always hype me up. Um, but the goal of this talk for me, and I know that we have a lot of engineers in the room, my g- my goal is to talk about healthcare as a domain.

[2:21](https://www.youtube.com/watch?v=u6q-byPWUuo&t=141s) At least I felt in the past that there was a lot of stigma around maybe the technical problems weren't as interesting in healthcare. And it is true in some ways, there's some parts of healthcare that might not be as tech forward.

[2:31](https://www.youtube.com/watch?v=u6q-byPWUuo&t=151s) A lot of things run on fax machines, for example. Um, but I wanna give exposure throughout this talk of two things. One, Abridge's journey from clinical documentation to clinical intelligence and what that looks like.

[2:45](https://www.youtube.com/watch?v=u6q-byPWUuo&t=165s) Um, and then two, I wanna expose you to some of the technical problems we work on, um, and that have to be th- that are truly frontier AI produ- uh, problems that have the highest stakes.

[2:57](https://www.youtube.com/watch?v=u6q-byPWUuo&t=177s) A little about me. My name is Chaitanya. You can call me Chai. Um, my career has always been in AI companies and startups. I first started as, uh, in research engineering at a company called Vicarious, uh, which its goal was actually to develop AGI, but they took very different methods.

[3:11](https://www.youtube.com/watch?v=u6q-byPWUuo&t=191s) They wanted methods inspired by neuroscience and probabilistic graphical mo- uh, models, um, and they concretely worked on robotics. Uh, then I started working at this company called Glean, 'cause I faced this problem in my workplace itself.

[3:23](https://www.youtube.com/watch?v=u6q-byPWUuo&t=203s) How, uh, like, information scattered all over the place, context is everywhere, and it's so key to decision-making. And Glean was building basically the ChatGPT for your workplace. I was there about six and a half years as one of their earliest engineers, as we went from ten people to over eleven hundred people and work with some of the

[3:39](https://www.youtube.com/watch?v=u6q-byPWUuo&t=219s) largest companies all over the world. Um, but that j- journey was amazing. I love that product. I love that company. I love the people there. Um, but I've actually always really been interested in healthcare.

[3:50](https://www.youtube.com/watch?v=u6q-byPWUuo&t=230s) I remember a decade ago, cover of Nature magazine was AI to detect skin cancer. I was like, "Wow, as someone interested in AI?" That was amazing. At the same time, I went to the hospital for something, and I remember seeing, oh, coming back, it was a really minor thing, but I remember coming back and looking at the

[4:05](https://www.youtube.com/watch?v=u6q-byPWUuo&t=245s) bill, and I was like, "I'm not really sure what exactly I paid for." Um, and so there's these known problems in healthcare, and I'll talk a little about some of them, access to care and cost, and then we had these amazing solutions, so I was like, "Why don't we bridge these things together?"

[4:18](https://www.youtube.com/watch?v=u6q-byPWUuo&t=258s) Um, and remember, this is about a decade ago. Um, and so I actually started this seminar where I invited speakers who were physicians, researchers, entrepreneurs to, to, uh, to talk about the space.

[4:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=268s) And what I learned was while there was really, really cool technology, very little of it made its way into the clinic at that time. And so again, my, my journey went a different way.

[4:38](https://www.youtube.com/watch?v=u6q-byPWUuo&t=278s) But as I, as I peeked my head out ten years, uh, ten years later then, actually our technology has gotten better than ever, as everyone knows, and the AI wave as it's taken over the whole world has also influenced healthcare.

[4:50](https://www.youtube.com/watch?v=u6q-byPWUuo&t=290s) And as you, as you saw towards the end of that video, Abridge, in the matter of two to three years, got its way into three hundred of the largest health systems, uh, in the United States, Kaiser, Mayo, Johns Hopkins, Sutter, and so forth, and maybe, maybe you've visited some of these hospital systems.

[5:06](https://www.youtube.com/watch?v=u6q-byPWUuo&t=306s) And once you're inside the hospital systems, you realize there's so, so much more you can do, and I'll talk about that journey that we've had. I specifically work on, um, lead our engineering teams for clinical decision support and our agentic experiences that the technology has now enabled and how we can bring that to healthcare.

[5:25](https://www.youtube.com/watch?v=u6q-byPWUuo&t=325s) But first, maybe, maybe some of the problems that inspire us as a company at Abridge. One, one of the things that we've noticed, uh, or many people, econo- economists have noticed over the past few decades is actually in many other industries, you actually see the cost of a good go down, and that's because the productivity has increased.

[5:43](https://www.youtube.com/watch?v=u6q-byPWUuo&t=343s) But in healthcare, we actually see administrative costs have only gone up over the past, uh, few decades, um, and productivity hasn't necessarily increased. Um, and a lot of our problems in healthcare we solve with labor, but even that we cannot actually keep up.

[5:59](https://www.youtube.com/watch?v=u6q-byPWUuo&t=359s) So there's this bit of this, like, productivity pa- uh, paradox you might have heard of, like Baumol's cost disease, and it's part of-- partially because technology, I think, hasn't fully touched healthcare as much as it's touched other industries to increase that productivity.

[6:12](https://www.youtube.com/watch?v=u6q-byPWUuo&t=372s) A few other problems, you know, hospitals are shutting down, margins are actually razor thin for many health systems. Of course, some patients have massive, uh, medical, uh, debt. And then we actually-- and the problem that's-- another problem that's very near and dear to our heart is that we hear all the time that doctors are burnt out, and

[6:30](https://www.youtube.com/watch?v=u6q-byPWUuo&t=390s) they actually often don't recommend it as a profession to, uh, to their children.

[6:36](https://www.youtube.com/watch?v=u6q-byPWUuo&t=396s) So what we started as, as a company was working on clinical documentation. So the idea here, if you're not familiar with it, is at, at the end of every patient visit, the, the clinician must create a note.

[6:48](https://www.youtube.com/watch?v=u6q-byPWUuo&t=408s) Uh, our typical format is a SOAP note that has a couple different formats, like what's the chief complaint of the patient and a few other sections, and then what's the assessment plan?

[6:57](https://www.youtube.com/watch?v=u6q-byPWUuo&t=417s) What do we do with this patient? You have to do-- write this after every single visit, and there's some different variations on this de-depending on specialty. Um, typically clinicians end up often doing-- it takes like two hours a day to write, just write these notes, and you often do it what's known as pajama time after work itself.

[7:16](https://www.youtube.com/watch?v=u6q-byPWUuo&t=436s) Uh, and that's a common source of clinician burnout, spending all this time outside of work, and it's not the most fun part of the job. However, these documents are actually extremely high stakes because these clinical notes are often used as a basis of billing, but also, uh, which is of course, financial things are high stakes, but also

[7:33](https://www.youtube.com/watch?v=u6q-byPWUuo&t=453s) have clinical impact. And the reason for this is because these prior no- these notes are used for the next clinician, or as you switch health systems, they use-- they provide context to the clinician of the, uh, patient's longitudinal medical record.

[7:47](https://www.youtube.com/watch?v=u6q-byPWUuo&t=467s) So it's actually really high stakes to get this right. Um, we started here because we-- it's a known pro-- uh, pain point that's existed for many, many years. But finally, the technology's caught up to do really, really high quality medical notes that's actually personalized to the clinician.

[8:03](https://www.youtube.com/watch?v=u6q-byPWUuo&t=483s) This was an amazing wedge into healthcare industry, which has typically been technology reticent because it re-led to... They actually care a lot about getting these notes high quality and right.

[8:13](https://www.youtube.com/watch?v=u6q-byPWUuo&t=493s) It led to higher doctor satisf-- uh, sas- provider satisfaction and such, they could actually see more patients. Um, and it can actually help create a higher-quality record, um, that helps prevent, uh, as it relates to billing, auditing and other reasons.

[8:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=508s) So we started there. Just this product alone scaled to three hundred ho-hospital systems. But I wanna show you a little about where we're going next. And, um, and the core thesis of the company is that every area in healthcare is, everything is around the conversation.

[8:42](https://www.youtube.com/watch?v=u6q-byPWUuo&t=522s) So we started over here with the co-co-conversation to clinical note. Everything else is downstream of that, whether you-- it relates to billing, whether it relates to things like cl-clinical trial matching or whether it relates to clinical decision support.

[8:57](https://www.youtube.com/watch?v=u6q-byPWUuo&t=537s) It's all about the conversation, that sacred doctor and patient conversation, and we've just built all this administrative machinery around that. But how can we bring it back to that conversation and actually automate some of that, uh, administrative machinery?

[9:13](https://www.youtube.com/watch?v=u6q-byPWUuo&t=553s) So to give you a tactical example of what this looks like, um, and I'll play this video. [coughing]

[9:26](https://www.youtube.com/watch?v=u6q-byPWUuo&t=566s) Of where, where we're going from here.

[9:28](https://www.youtube.com/watch?v=u6q-byPWUuo&t=568s) We've been building a solution that allows the physician to interact with Abridge directly by using their voice. Hey, Abridge, [beep] is Nathan eligible for any clinical trials?

[9:41](https://www.youtube.com/watch?v=u6q-byPWUuo&t=581s) He may be eligible for the Abridge HF study. He remains symptomatic despite maximal therapy, and most screening criteria are already met. But an updated echocardiogram is needed to confirm his ejection fraction and complete eligibility assessment.

[9:53](https://www.youtube.com/watch?v=u6q-byPWUuo&t=593s) All right. Please order that echo for him.

[9:56](https://www.youtube.com/watch?v=u6q-byPWUuo&t=596s) Done. Confirmatory echo ordered.

[9:58](https://www.youtube.com/watch?v=u6q-byPWUuo&t=598s) And then when I'm done for the day, I can just ask Abridge, "Hey, Abridge, [beep] can you prepare my charts for tomorrow?"

[10:06](https://www.youtube.com/watch?v=u6q-byPWUuo&t=606s) And Abridge is working for me.

[10:10](https://www.youtube.com/watch?v=u6q-byPWUuo&t=610s) Pause it right there. One of the things that you'll, you'll notice is that we are thinking about how to, uh, revolutionize the entire visit, uh, for a clinician. From pre-visit, how, um, earlier in the-- we have suggested discussion topics.

[10:23](https://www.youtube.com/watch?v=u6q-byPWUuo&t=623s) Here's things that you can talk about, uh, with your patient, whether they're clinical or more, uh, billing related. Um, we have, after the visit, we actually create everything for you.

[10:32](https://www.youtube.com/watch?v=u6q-byPWUuo&t=632s) The patient visit summary, the actual clinical note, and we actually pend orders, as you might have seen. We're able to use-- W-we are able to do this all by reading all this context.

[10:42](https://www.youtube.com/watch?v=u6q-byPWUuo&t=642s) We have access to all of the EHR context, so we know everything about the patient, the, the prior labs, the prior notes. We have access to the live conversation between the doctor and the patient.

[10:52](https://www.youtube.com/watch?v=u6q-byPWUuo&t=652s) That's where the quote unquote debugging happens in healthcare, where you learn about what the patient is facing now. Um, and then we have access to world's medical literature that we can ground and clinical guidelines that we can ground all of our work in.

[11:05](https://www.youtube.com/watch?v=u6q-byPWUuo&t=665s) So, do, do, do. I wanna, I wanna s-switch now, given the context of where we're going as a, as a product, I wanna switch into some of the key technical and engineering problems we face and inspire you on some of the, what I think are f-very much frontier AI challenges.

[11:27](https://www.youtube.com/watch?v=u6q-byPWUuo&t=687s) So if you've ever worked on a agentic product before, the-- regardless of vertical, um, the three KPIs that tend to matter are quality and latency and cost. In healthcare, I feel that we're actually playing on hard mode for all of these three KPIs.

[11:42](https://www.youtube.com/watch?v=u6q-byPWUuo&t=702s) This is a high stakes scenario, especially when you're doing something like clinical decision support. You have to be right because the downside is extremely high when you're wrong. When I used to work at Glean, you know, while I love that product, I could be wrong and it would have been fine.

[11:55](https://www.youtube.com/watch?v=u6q-byPWUuo&t=715s) Maybe we answered a question incorrectly. But in healthcare, if we answer something incorrectly, there's actually consequences, and we entirely lose our trust. So quality needs to be absolutely high, and I'll talk a little about how we keep that bar high.

[12:07](https://www.youtube.com/watch?v=u6q-byPWUuo&t=727s) And then latency and cost also really matter for us when you're live in the conversation. You can't, uh, with latency, you can't act on information too late, and you have to act on the-- also at the right time for it to be useful.

[12:20](https://www.youtube.com/watch?v=u6q-byPWUuo&t=740s) And then finally, cost at the scale we're, we're doing this at.

[12:25](https://www.youtube.com/watch?v=u6q-byPWUuo&t=745s) Um, and, and, and as an interesting aside, it actually relates to, uh, our-- We have a motto inside the company that our goal is to save lives, save time, save money for, uh, for the hospital system and for the healthcare industry as a whole.

[12:36](https://www.youtube.com/watch?v=u6q-byPWUuo&t=756s) And it actually-- I think it's funny that it really maps to the three K-KPIs that you care about in any agentic product.

[12:44](https://www.youtube.com/watch?v=u6q-byPWUuo&t=764s) So talking a little about quality, how do we keep that bar high? For us, we really treat evals as the l-operating system, the life's blood of the, of the company.

[12:54](https://www.youtube.com/watch?v=u6q-byPWUuo&t=774s) This starts from internal benchmarks and offline evaluation. Before we develop any product, you know, whether we're talking about clinical trial matching, clinical note, clinical decision support, coding, we start with a robust set of internal benchmarks.

[13:08](https://www.youtube.com/watch?v=u6q-byPWUuo&t=788s) This is pre-deployment, and then we test that against, you know, things that we've actually seen in the wild. Then we al-- have a staged rollout. We know that we need to, uh, make contact with realit-reality.

[13:20](https://www.youtube.com/watch?v=u6q-byPWUuo&t=800s) Not everything offline will perfectly represent what happens in practice, and so we slowly roll it out. Uh, maybe it starts with the alpha s- group of clinicians that we trust and under-- they understand the stakes.

[13:31](https://www.youtube.com/watch?v=u6q-byPWUuo&t=811s) We roll out to beta. Maybe there's A/B testing at scale. And then even after it's fully rolled out, you always need continual, uh, monitoring. Again, the stakes are really high, and you cannot get away with just being, like, a prototype that you just ship out there and be like, "Yeah, I mean, I tested it on a few

[13:45](https://www.youtube.com/watch?v=u6q-byPWUuo&t=825s) cases, and it works." How we do this is we always have expert-calibrated LLM judges. So we have clinicians embedded throughout the entire company. The clinicians are d-domain experts, but not all of us are clinicians.

[13:58](https://www.youtube.com/watch?v=u6q-byPWUuo&t=838s) I'm not a clinician. So how can we, the rest of the company, still move fast is by encoding that clinician judgment into LLM judges. You know, I think a really great evaluation system has a property that it reflects the behaviors that you want in your product.

[14:14](https://www.youtube.com/watch?v=u6q-byPWUuo&t=854s) At the end of the day, we are making a product for clinicians, and so who best other than our clinicians to actually create our judges that represent what they want?

[14:21](https://www.youtube.com/watch?v=u6q-byPWUuo&t=861s) And those judges, once you have that, create a feedback loop so that anyone, whether you're a clinician or not, can actually, uh, hill climb and learn from that. We also have a lot of online signals, whether how you're editing the clinical note and your typical thumbs up, thumbs down, uh, and star ratings and other free-form text.

[14:40](https://www.youtube.com/watch?v=u6q-byPWUuo&t=880s) So this is a general framework we use for our, for all our products. I wanna deep dive into the product that I work on, which is clinical decision support.

[14:50](https://www.youtube.com/watch?v=u6q-byPWUuo&t=890s) So to give you an example of what clinical decision support looks like, and specifically, we're building something novel, which is contextual clinical decision support. Maybe a provider asks a question like, "Hey, does this patient meet the criteria for febrile neutropenia?"

[15:04](https://www.youtube.com/watch?v=u6q-byPWUuo&t=904s) Um, and so what we have to do here is actually a lot of context is underspecified in this question. We're-- So the first thing we're gonna do is we're actually gonna pull from the EHR data previous, uh, previous lab values.

[15:17](https://www.youtube.com/watch?v=u6q-byPWUuo&t=917s) Then using that context, we're gonna, uh, use the, uh, c-- uh, we're also gonna use the live conversation, and we're gonna use clinical guidelines and medical journals to com-- use as the reasoning sources for combining all this context to actually answer the provider's question.

[15:34](https://www.youtube.com/watch?v=u6q-byPWUuo&t=934s) Now-- But I wanna focus on evaluation. Again, the stakes are really high here. We, we really can't get this wrong. So how do you tell whether or not an answer is correct?

[15:42](https://www.youtube.com/watch?v=u6q-byPWUuo&t=942s) And sometimes I, I f- uh-- And this is a case where the generator and the verifier gap is really small. What I mean by this is, in some problems in AI, such as like Sudoku, it's really, really hard to generate a solution to Sudoku, but it's extremely easy to verify it, uh, once you do have the solution.

[16:00](https://www.youtube.com/watch?v=u6q-byPWUuo&t=960s) And that makes, uh, that makes it much easier to hill climb against, uh, and build evaluation for. But in a case like this, the generator and verifier gap is really small.

[16:08](https://www.youtube.com/watch?v=u6q-byPWUuo&t=968s) If I had a really, really good generator, uh, verifier, then that would just be my generator itself. So how do I create a reference that isn't just a language model itself and ground itself so I have trust?

[16:21](https://www.youtube.com/watch?v=u6q-byPWUuo&t=981s) So what we do is we tackle this by having many, many different signals. Uh, we have a clinical quality judge, which I'm gonna dive deep into, and then we tackle from, uh, we have many signals from a boundary and adversarial judge.

[16:33](https://www.youtube.com/watch?v=u6q-byPWUuo&t=993s) We have a clinical safety judge. And we also have judges that re-represent product as-aspects like tone and style, and that matters a lot as well for AI products. So all of these are different signals that try to get a piece of this, like, really, really hard-to-measure problem and guarantee it in the way that we want the product

[16:49](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1009s) to be. So diving into the clinical quality judge. So again, I said the v-- generator–verifier gap is really small here. So what we need is we actually need human references to tell are, are we generating the right thing?

[16:59](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1019s) But you can't just create a human golden response because there is a lot of variability in the potential responses. So what we did is we took a lot of real clinical cases.

[17:09](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1029s) We had independent physicians create a rubric. So this rubric said elements of what we wanted in the response. So it's not, "Here's the exact response," because again, there's many infinite possible responses, but a re-- a good rubric elements that what a good r-response would look like.

[17:22](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1042s) And then we had a separate physician that actually adjudicated it, brought these two independent rubrics together, created a final rubric, and we actually had a fourth clinician do QA on these rubrics.

[17:32](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1052s) Once you have these rubrics, and here, here's a sample rubric, what it looks like. Here's actually a question. And then, I mean, there, there's more context in the case itself.

[17:38](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1058s) You have a rubric of what are the elements that a response should look like. Now, we can actually have an LLM judge that compares our agent's responses to these rubric elements and does some s-- uh, basic semantic match to tell, hey, is our model performing well as we continue to hill climb, whether it's our agent architecture, our

[17:55](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1075s) models, or search ranking algorithms. I wanna talk a little now about cost and latency, two other really, really hard problems for us. So we do this, as we said in that intro video, we do this on the-- live in the conversation, and we do on the run rate of a hundred million medical conversations a year.

[18:13](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1093s) How do we do this in a way that doesn't really break the bank for us? So one, one place that this problem comes up, uh, or is, is actually in generating the clinical note.

[18:22](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1102s) So when you're generating the clinical note, there's many different sections to it. There's a history of p- uh, present illness, past medical history, and there's the assessment and plan.

[18:30](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1110s) So one of the core insights for us is, rather than, say, using a foundation model to generate all of this, is we can actually break, decompose this problem into simpler, smaller workflows.

[18:41](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1121s) Healthcare is actually many specific workflows. You don't need, you know, Fable Five, uh, to actually solve all of your, uh, clinical notes. We, we don't need frontier-level intelligence for every problem, so we actually post-train a lot of smaller models for different problems, such as different-- Actually, even to the granularity of different sections in the clinical note.

[19:00](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1140s) And that lets us use much smaller mo- uh, models because it's a m- more specific problem and at, uh, at much cheaper cost and latency. And we have this data flywheel that we have this unique dataset of a hundred million medical conversations a year.

[19:14](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1154s) And as far as we know, no one else has such a h- uh, large dataset. So our key insight is having a right to tr- win in training models.

[19:22](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1162s) There are problems where the quality's already maxed out, and so you should train models then to reduce cost and latency. But there are other problems where the quality isn't maxed out, and people say, "Oh, the frontier model would just steamroll you."

[19:33](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1173s) Our key insight is we can actually potentially beat the rate of change on the frontier model if we have the right, uh, to win by having the right data that they may not have and the focus on a problem that they may not be focusing on, and that lets us still maximize quality.

[19:49](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1189s) Another, uh, problem that I'll quickly touch on is in-visit orders. So, uh, do- doctors really aren't big fans of pending orders, but often they'll mention orders during the visit itself, medication or non-medication orders.

[20:01](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1201s) So what we have is while we're listening in the visit, as the, uh, clinician says, uh, order, we actually queue it up in the background, uh, le- and let them, uh, actually sign it off in the EHR.

[20:11](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1211s) But you can imagine if we did this in a very naive way, like every few seconds are just listening for orders, that would really break the bank. Um, and so a lot of our tricks are like, how do we find the right events in the conversation to actually trigger heavier models that will actually do the order matching?

[20:26](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1226s) Because you need to match the order, not just is the order said, but does it match and reference the system's orders that are approved by the system and are relevant to the conversation.

[20:34](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1234s) So we have a number of different gates that are cheaper and faster that let us trigger actually larger models and hand off to them for actually doing the end-to-end work.

[20:46](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1246s) Um, but the last message, this was of course a very quick talk, but the message I wanna leave you with is healthcare is a domain that needs frontier AI and actually puts it to the test at, uh, higher stakes.

[20:56](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1256s) In the past, I as an engineer myself was wary about working in healthcare. Does, uh, does, like, does healthcare technology actually work? Well, Abridge has proven this at scale for sure, and I was-- hopefully I gave you a taste of some of the frontier problems that we work on.

[21:10](https://www.youtube.com/watch?v=u6q-byPWUuo&t=1270s) So thank you so much. My name is Chaitanya again. You can, uh... And feel free to connect with me at Twitter or LinkedIn. Thank you. [audience applauding] [upbeat music]
