Make your LLM app a Domain Expert: How to Build an LLM-Native Expert System

What does an LLM need to understand before it can make a useful decision in a specialized industry? Christopher Lovejoy approaches that question as a doctor turned AI engineer: he describes eight years training and working in medicine, followed by seven years building AI systems that incorporate medical expertise. His work included tech-enabled home care at Cera Care, whose contemporary company announcement reported approximately $500 million in annualised revenues—the milestone he calls ARR in the talk.

At the time of the presentation, Lovejoy worked at Anterior, a New York-based, clinician-led company building clinical reasoning tools for health insurance and healthcare administration. Lovejoy says its insurer customers covered about 50 million US lives. That is the customers' coverage population, not a count of patients evaluated by the AI.

The system for incorporating domain insights matters more than the sophistication of the models and pipelines. This is Anterior's bet about vertical AI: capable reasoning still needs the context of a particular industry, customer and workflow. The surrounding system must capture that context and make it usable quickly. In Lovejoy's schematic, a domain-expert product manager sits at the center; in healthcare, that means a PM with clinical expertise.

The last mile problem becomes concrete in a case processed by Anterior's AI, Florence. A patient presents with right knee pain, and their doctor recommends knee arthroscopy. To assess whether the treatment is appropriate, Florence must answer several questions. One asks whether there is documentation of unsuccessful conservative therapy for at least six weeks. This is the criterion in the example, not a universal coverage rule.

Each part of that apparently straightforward question hides a separate judgment:

• Conservative therapy: Relative to surgery, physiotherapy, weight loss and other non-invasive interventions may be conservative options. Medication is less straightforward: in one context it is conservative treatment; in another it is the more aggressive option.
• Unsuccessful: Symptoms may improve significantly without disappearing. Does success require complete resolution, or is partial improvement sufficient? If partial improvement counts, somebody must define the threshold.
• Documentation for at least six weeks: A note saying physical therapy began eight weeks ago establishes a start date. It does not explicitly establish that treatment continued or was completed. The workflow must define how much the reviewer may infer from silence in the record.

The difficulty is not merely recognizing medical terms. It is applying the customer's interpretation of those terms to incomplete evidence.

The engineering challenge is to translate discoveries like these into pipeline context, then iterate. Stronger models improve the starting point, but they do not supply every customer-specific interpretation. Lovejoy reports that substantial pipeline work reached a plateau around 95% on Anterior's care-request approval task, and that iteration through the domain-feedback system brought performance to roughly 99%. The companion slides clarify the measure: F1 rose from 95.73% to 99.24% after eight weeks with a specific customer. The talk calls this accuracy; the slides label it F1, and neither supplies a sample size or evaluation protocol.

Lovejoy also points to a recent KLAS Points of Light recognition. The practical lesson he draws is that models already reason well enough to establish a strong baseline, while the remaining work requires more context. Anterior calls its surrounding system the Adaptive Domain Intelligence Engine: it converts customer-specific domain insights into performance improvements through two connected activities, measurement and improvement.

Begin by defining the few outcomes customers care about most. In medical necessity review, Anterior's customers want to minimize false approvals: approving care that was not needed can expose a patient to unnecessary treatment and leave the insurer paying for it. Domain experts and customers define these metrics together, translating a broad business goal into an outcome that can guide engineering work.

These examples force a useful choice: if only one or two metrics could improve, which would matter most to the user? Alongside that choice, build a failure-mode ontology—a structured account of the ways the application can fail.

For medical necessity review, Anterior uses three broad categories: medical record extraction, clinical reasoning and rules interpretation. Each contains more specific subtypes, refined as the team learns from cases. Domain experts need to lead this process. Someone examining traces in isolation may identify technical patterns while missing the workflow context that makes an output wrong.

Capture correctness and failure type in the same review session. Anterior's internal dashboard places the patient's record and the applicable guidelines on the right, with the AI's decision and reasoning on the left. A clinician marks the output correct or incorrect; an incorrect output also receives a failure-mode label. Keeping these judgments together lets the reviewer explain the error while the evidence and decision are still in view.

Those paired labels connect engineering problems to customer consequences. Plot false-approval count on the horizontal axis and failure modes on the vertical axis. The resulting ranking tells the PM which kinds of errors contribute most to the outcome the team wants to reduce. Prioritization now has a concrete basis: address the failure modes responsible for the most false approvals first.

The same labels also produce datasets for improvement. Lovejoy argues that cases drawn directly from production better represent the application's incoming data than synthetic cases. His example handoff is a set of 100 cases from the preceding week that share a prioritized failure mode. An engineer can work against that set and repeatedly measure whether the relevant behavior improves.

A small TypeScript representation makes the selection explicit. Each reviewed production case retains its identifier, correctness judgment and failure category, so a targeted dataset can be selected without reconstructing the review:

type FailureMode =
  | "medical_record_extraction"
  | "clinical_reasoning"
  | "rules_interpretation";

type ReviewedCase = {
  caseId: string;
  reviewedAt: string;
} & (
  | { correct: true; failureMode: null }
  | { correct: false; failureMode: FailureMode }
);

function selectFailureCases(
  reviews: readonly ReviewedCase[],
  mode: FailureMode,
  startInclusive: Date,
  endExclusive: Date,
): ReviewedCase[] {
  return reviews.filter((review) => {
    const reviewedAt = Date.parse(review.reviewedAt);
    return (
      !review.correct &&
      review.failureMode === mode &&
      reviewedAt >= startInclusive.getTime() &&
      reviewedAt < endExclusive.getTime()
    );
  });
}

This selects the case references for an evaluation set; the evaluation still needs the associated inputs and expert judgments.

Track each failure dataset across pipeline versions, with performance score on the vertical axis. Because these datasets were selected for failures, their initial scores are low by construction. A release focused on one category should produce an improvement in its series; subsequent releases may improve other categories. Keeping all the series visible also reveals regressions that a single aggregate score could hide.

Experts can participate in improvement as well as measurement. Give a nontechnical reviewer a way to suggest changes to the pipeline or add domain knowledge that the pipeline can use. Evaluate each suggestion against the targeted failure sets and broader evaluation sets before deciding whether it should go live. The proposed knowledge enters an experimental path first; its usefulness is a question the evaluations must answer.

Anterior extends the same review dashboard with a domain-knowledge addition button. The clinician already has the case in view, has judged correctness and has identified the failure mode. Now they can propose the missing context. In one example, the model concludes that there is no suspicion of a condition even though the patient has that condition. A clinician can supply the relevant medical interpretation of suspicion. In another, the reasoning depends on a scoring system that the model cannot access; the reviewer can propose adding that system to the available knowledge.

Release may be gated automatically by evaluations or include a human approval step. Lovejoy describes a possible same-day cycle: a production case arrives, a clinician identifies the missing knowledge, the team evaluates the addition, and the change goes live if the results support it. The speed comes from preserving context through the process. One expert review supplies performance measurements, failure-mode labels and suggested improvements together.

An audience question makes the term domain expert more precise: how much expertise is enough? Match the reviewer to the workflow and the quality being assessed. For clinical reasoning, Lovejoy would ideally choose an experienced doctor with expertise in the relevant specialty. Simpler tasks may be appropriate for a more junior clinician. The essential qualification is experience doing the actual workflow—whether as a nurse, doctor or another relevant practitioner.

The next question concerns bespoke versus off-the-shelf tooling. Anterior's tooling is bespoke. Lovejoy favors building it when review outputs are consequential and feed several parts of the platform, because the review interface must integrate closely with the rest of the system. The rationale is the role of those outputs in measurement and improvement, rather than customization for its own sake.

Finally, reviewers can be internal experts or customers. Anterior typically begins by hiring people in-house to generate the initial review data needed for iteration. Customers may also want to validate the AI themselves. In that situation, the same kind of review interface can become a customer-facing product, although Lovejoy presents customer-operated validation as a possibility rather than the default arrangement.

The complete workflow puts the domain-expert PM between production evidence and engineering action:

1. Review production decisions. Experts assess the AI's outputs and generate metrics and failure labels.
2. Prioritize a failure mode. The PM uses its contribution to customer-valued outcomes to decide what deserves attention.
3. Set a measurable target. Assign an engineer a particular failure dataset and a performance threshold. Lovejoy's illustrative target is to raise performance on that dataset from 0% or 10% to 50%; it is not an overall application accuracy target.
4. Experiment against the dataset. The engineer changes prompts, changes models or tries fine-tuning, using repeated evaluations to measure progress.
5. Return the change and its impact. Once the target is met, the engineer gives the PM the proposed changes and evaluation results.
6. Decide whether to release. The PM weighs those results alongside effects elsewhere in the product before approving production deployment.

The targeted dataset makes the engineering assignment concrete, but meeting its threshold does not settle the broader release decision.

This is the operational meaning of a self-improving domain-native application: live customer cases supply the context, expert reviews turn that context into measurements and proposed changes, and evaluations guide iteration under a domain-expert PM. More powerful models remain useful, but the ongoing work is to give the product a more nuanced understanding of the workflow in which its decisions will be used.