Healthcare work and the intended improvement
A clinical workflow comprises the tasks, decisions, and handoffs involved in care. Clinicians provide professional care, administrative staff coordinate access, and technical operators maintain supporting services. Their interconnected work determines whether information reaches someone able to act.
Intended use specifies a system's purpose, users, setting, inputs, and permitted role. Its patient population is the group for whom that use is intended. These boundaries precede architecture: collecting symptoms for review and independently directing care are different purposes. The IMDRF definitions connect medical purpose to software evaluation and oversight.
A proposed follow-up workflow
ExampleInformation collection leaves clinical and coordination responsibilities outstanding.
Read the diagram as text
- Patient.
- Automated collection.
- Clinician assessment.
- Care coordination.
- Technical operators.
- Patient → Automated collection: Data: reported symptoms.
- Automated collection → Clinician assessment: Data: collected information.
- Clinician assessment → Care coordination: Control: authorized follow-up.
- Technical operators → Automated collection: Control: service operation.
- A bounded care example — A prospective study evaluated telephone symptom collection about three weeks after routine cataract surgery. An ophthalmologist supervised the automated calls in real time. Non-English speakers and people with hearing or cognitive difficulties were excluded. Those conditions define the evidence: supervised follow-up in that population does not establish unsupervised performance for everyone.
- The proposed change — Specify what happens when information is wrong, arrives late, or never arrives. A missed warning and an invented answer require different tests from an awkward conversation. Scenario design should start with possible patient harm within the actual workflow.
| Application | Output and authority | Candidate benefit and baseline |
|---|---|---|
| Documentation | Draft note for professional review. | Less documentation work; compare with existing notes and templates. |
| Operations | Referral information and routing assistance; staff retain responsibility for handoffs. | More completed care connections; compare with structured intake and clearer responsibilities. |
| Clinical support | Patient-specific information informs a clinician's decision. | Better decisions or outcomes; compare with existing rules, reminders, and reference access. |
A learned prediction estimates a target using patterns fitted from examples. A forecast of incoming referrals can inform staffing, but it does not allocate appointments. Machine Learning Fundamentals explains how learned estimates differ from authored rules.
A generated answer is newly composed text. Fluency does not establish factual correctness. When generation uses retrieved records or references, the application must check whether they support the answer; Retrieval-Augmented Generation develops that evidence-to-answer relationship. Predictive scores instead need a defined outcome and evaluation against observed results.
What a clinical record can establish
An electronic health record, or EHR, stores patient information across visits and care settings. An encounter is a particular care interaction. Notes describe it; orders request actions; results record findings. A longitudinal record brings these entries together over time, without guaranteeing completeness.
Provenance identifies where information came from, who created or changed it, and which version is being used. It helps distinguish a current observation from a later transcription or revision.
Event time is not availability
ExampleEarlier specimen collection does not make a result usable.
Result content cannot enter this decision.
Read the diagram as text
- Patient P.
- Result R. Specimen collected 09:00; initially pending.
- 09:30: unavailable.
- 10:15: issued, not ingested.
- 10:20: application access.
- Patient P → Result R: Patient association.
- Result R → 09:30: unavailable: Decision-time status.
- Result R → 10:15: issued, not ingested: Later status.
- Result R → 10:20: application access: Later status.
- Decision at 09:30. Result content cannot enter this decision. Active: Patient P, Result R, 09:30: unavailable. New: Patient P, Result R, 09:30: unavailable.
- Issuance at 10:15. Provider availability precedes application arrival. Active: Patient P, Result R, 09:30: unavailable, 10:15: issued, not ingested. New: 10:15: issued, not ingested.
- Ingestion at 10:20. New availability cannot justify earlier use. Active: Patient P, Result R, 09:30: unavailable, 10:15: issued, not ingested, 10:20: application access. New: 10:20: application access.
| Recorded information | What it supports | What it does not establish |
|---|---|---|
| Medication order | A prescription or administration instruction. | That medication was dispensed or administered. An administration record and a patient's report of taking medication are different evidence. |
| Observation without a number | A result may use another datatype or appear in components. | A normal finding. Inspect the observation's meaning, result, interpretation, and any absence explanation. |
| Two measurement values | Comparison after checking measurement identity and compatible coded units. | Equality of meaning from bare numbers. Preserve less-than or greater-than bounds; matching units alone does not establish comparable specimens or methods. |
| Matching display labels | A possible terminology correspondence to investigate. | Equivalent coded meaning. Retain terminology system, version, and code; consult definitions or mappings when they differ. |
Unavailable data have different explanations. Fast Healthcare Interoperability Resources (FHIR), a healthcare data-exchange standard, distinguishes unknown, not asked, not performed, withheld, and error states. None means healthy. If no absence reason is recorded, the application cannot infer which explanation applies.
Changed values can reflect a changed patient or a changed measurement process. Observer technique, examination conditions, and timing all matter. Repeated measurements can be consistently wrong; repeatability does not establish validity. Longitudinal comparisons therefore need acquisition context as well as values.
- Ordering-dependent information — A laboratory measurement becomes observable only after someone orders the test. Testing frequency can encode clinical concern. This is informative missingness: availability itself carries information about the care process, which may change across institutions.
- Documentation-dependent information — Absent smoking-history codes cannot establish that a patient does not smoke. Information may be omitted, buried in narrative notes, or held at another institution.
- Billing-dependent information — Diagnosis coding also responds to reimbursement and documentation practices. More recorded diagnoses can reflect more intensive recording rather than an equivalent increase in illness. The label must match the clinical concept being predicted.
Temporal leakage occurs when evaluation supplies information unavailable at the intended decision time. Specimen collection, result issuance, and application ingestion are separate events. Reconstruct accessible versions, not merely the completed historical record.
These distinctions make data quality task-dependent. A field adequate for billing may be unsuitable as a disease-onset label; missing documentation may invalidate a completeness check. Data Quality and Curation explains the broader methods for assessing fitness, provenance, and coverage.
Patient context, permissions, and record updates
FHIR represents information as resources connected through references. Profiles constrain resources for particular uses. This shared structure supports integration, but compatible syntax does not establish correct patient identity, clinical meaning, or complete information.
SMART on FHIR supports application launch and delegated access. Patient context identifies whose record is selected; encounter context identifies the care interaction. Scopes specify permitted operations. Requested scopes can differ from granted scopes, and launch context does not independently prove correct identity.
Approval does not bypass checks
ExampleReviewed work can still fail authorization or concurrency checks.
Read the diagram as text
- Selected patient context.
- Context and access checks.
- Read permitted records.
- Draft and professional review.
- Write checks. Recheck context, permission, and base version.
- Saved.
- Rejected.
- Reconcile and re-review.
- Selected patient context → Context and access checks: Data: context.
- Context and access checks → Read permitted records: Control: valid.
- Context and access checks → Rejected: Control: invalid.
- Read permitted records → Draft and professional review: Data: records.
- Draft and professional review → Write checks: Data: reviewed change.
- Write checks → Saved: Control: checks pass.
- Write checks → Rejected: Control: access fails.
- Write checks → Reconcile and re-review: Control: version conflicts.
- Read only suitable context — Confirm the selected patient and encounter, granted access, required information, and freshness before generating patient-specific work. A launch capability and permission to read are separate checks.
- Separate approval from permission — A clinician's approval does not create server authorization. Permission to create a resource does not necessarily include reading it; execution must satisfy the granted scopes.
- Distinguish the action — Saving a draft, professionally confirming a note, and authorizing an order have different consequences. In one described order-assistance workflow, conversational mentions are matched to supported orders and queued for clinician sign-off in the EHR.
- Version-aware update — For reviewed version v7, send the update precondition
If-Match: W/"7". If the server is at v8, it returns HTTP 412; reread and review.
Conditional creation creates on zero matches, reuses one match, and fails on multiple matches. Server support must be verified; this prevents some duplicate records, not every duplicate clinical action.
Define the purpose and handling of recordings, transcripts, drafts, and retained evidence, including vendor access and reuse. Patients need understandable information about the workflow. Privacy and Data Governance covers access, retention, and accountability; WHO guidance places stakeholder participation and cybersecurity within health-AI governance.
Documentation fidelity and professional review
Ambient documentation captures a care conversation to draft notes. Attestation is professional confirmation of the record. NHS England's professional guidance calls for meaningful review, correction before inclusion, and respect for patient objections. Its responsibilities are jurisdiction-specific, rather than a universal recording rule.
Source-to-draft comparison reveals errors that fluent prose can conceal. The following constructed examples isolate changes in meaning; they are not measured error frequencies. Speech recognition deserves separate inspection because incorrect transcription can make subsequent generation faithfully reproduce the wrong information.
| Source information | Altered draft | Required correction |
|---|---|---|
| Patient reports no dizziness today. | Dizziness today. | Restore negation; a symptom was denied. |
| Companion describes their own headache. | Patient has headache. | Restore speaker attribution. |
| Medication prescribed; use unconfirmed. | Patient takes medication. | Preserve the difference between an order and reported use. |
| Clinician discusses a possible diagnosis. | Diagnosis confirmed. | Retain uncertainty; the source does not support confirmation. |
| Earlier note lists symptoms; later entry revises them. | Summary repeats only the earlier entry. | Reconcile versions and retain temporal context. |
Source attribution links a statement to its origin. Abridge's linked-evidence interface connects note text to transcript passages and audio playback. This helps locate an error, but a transcript link cannot validate transcription or establish that a supported statement is clinically correct.
A signed note becomes context for subsequent clinicians and can support billing. An error therefore survives beyond its original encounter and may enter later summaries. Evaluation should inspect omissions and downstream readability alongside total drafting, review, and correction effort.
- A measured but bounded benefit — A randomized ambient-documentation study reported reduced note-writing time and practitioner exhaustion-related outcomes. Its documentation-only system used both private EHR interfaces and FHIR APIs. Early-adopter participation and an open-label design limit transfer; those findings do not establish better diagnosis or patient health.
Referral processing, capacity, and access
A referral requests evaluation or care from another clinician or service. Closed-loop referral communication connects relevant information with the responsible recipient; transmitting a request alone does not complete that responsibility.
Electronic referral experience shows that digitization can shift work to referring clinicians and complicate patient scheduling. Intake speed and end-to-end access can diverge.
Referral work has several exits
ExampleFaster intake leaves capacity and follow-up responsibilities intact.
Read the diagram as text
- Intake checks.
- Referrer: complete and resubmit.
- Specialist review.
- Advice to referring clinician.
- Referrer: workup and resubmit.
- Staff coordinate appointment.
- Attendance recorded.
- Staff follow up missed visit.
- Intake checks → Referrer: complete and resubmit: Information incomplete.
- Intake checks → Specialist review: Information sufficient.
- Specialist review → Advice to referring clinician: Advice sufficient.
- Specialist review → Referrer: workup and resubmit: Further workup needed.
- Specialist review → Staff coordinate appointment: Visit needed.
- Staff coordinate appointment → Attendance recorded: Patient attends.
- Staff coordinate appointment → Staff follow up missed visit: Patient misses visit.
- A proposed AI contribution — Extract the referral question and supporting information, flag omissions, and prepare routing suggestions. Keep responsibility for unresolved communication visible, including patients needing different languages or communication channels.
- Prediction versus capacity — Assume a forecast estimates 30 referrals tomorrow and the schedule has 20 slots. The forecast can motivate staffing or capacity changes; it cannot create ten appointments. Choosing among patients also requires urgency rules and availability. Forecast accuracy and allocation quality are separate evaluations.
- End-to-end measures — Track completed handoffs, unresolved requests, time to service, patient effort, and total staff work. Inspect access across communication needs instead of counting processed forms alone.
The same distinction applies to payment workflows. A provider delivers care; a payer finances covered services. Prior authorization requests payer permission before specified care. An appeal after denial requires patient evidence, care guidelines, and coverage policies. Generating a letter does not resolve missing evidence or changing policy.
Linking earlier authorization records to later denials can reveal recurring omissions worth checking before submission. That is a proposed prevention mechanism, distinct from merely producing appeals faster; it needs evidence that the upstream intervention actually reduces avoidable failures.
Clinical information and consequential action
Clinical decision support, or CDS, provides information at useful points in care to assist professional decisions. An alert's usefulness depends on its recipient, timing, supporting evidence, and available response.
A clinician's question may omit information already present in the record or encounter. A described contextual assistant combines historical laboratory results, the live conversation, and clinical references. Completeness, reference currency, and applicability still require checking; Retrieval-Augmented Generation explains the general grounding problem.
Information becomes action through assessment
ExampleA prediction supplies information, not treatment authority.
Read the diagram as text
- Patient information.
- Estimate or sourced answer.
- Clinician assessment.
- Owned clarification or review.
- Authorized response.
- Observed outcome.
- Patient information → Estimate or sourced answer: Data: evidence.
- Estimate or sourced answer → Clinician assessment: Data: advisory output.
- Clinician assessment → Owned clarification or review: Control: evidence insufficient.
- Clinician assessment → Authorized response: Control: response justified.
- Authorized response → Observed outcome: May affect.
- Make the output actionable — Present the supporting information, applicable limits, and decision expected from the recipient. An explanation that leaves the required response unclear does not adequately support the human-AI team.
- Preserve an abstention path — Abstention means withholding an automated prediction or decision. It reduces automated coverage and transfers work elsewhere; the receiving workflow must remain usable. A more restrictive confidence threshold reduces errors only when its score actually distinguishes errors.
- Enforce authority outside generation — Code can enforce identity checks and permitted routing before conversational generation. Deterministic enforcement does not make emergency recognition infallible: a fixed route can still receive an incorrect classification.
Risk is not treatment benefit. Assume one group has adverse-outcome risks of 0.8 both without and with an intervention; another has risks of 0.3 and 0.1. The higher-risk group has no reduction in this example, while the other has a 0.2 reduction. Machine Learning Fundamentals develops prediction versus causal effect.
Conversational intervention also depends on context. One mental-health demonstration distinguished immediate danger requiring external support from past distress where supportive conversation could continue. These examples illustrate differentiated routing, not validated triage rules; past tense alone cannot establish present safety.
Accountable review and completed escalation
Meaningful oversight requires evidence, expertise, time, authority, and an alternative workflow. Responsibility must identify what someone can inspect, change, and stop. The interface should communicate limits and support the actual decision, rather than merely expose a technical explanation.
Expertise should match the task. A clinician experienced in the relevant specialty or documentation workflow may detect failures that a broad professional credential does not prepare someone to recognize. Existing domain experts can be empowered; a new hire is not always necessary.
| Proposed responsibility | Required capability | Unavailable-owner response |
|---|---|---|
| Documentation reviewer | Inspect source material, correct the draft, and decide record inclusion. | Leave work visibly unconfirmed; use the established documentation process. |
| Operational exception owner | Resolve missing information and coordinate the next handoff. | Keep the request assigned and unresolved; transfer responsibility explicitly. |
| Clinical decision owner | Assess evidence and authorize, change, or reject a consequential response. | Use an agreed escalation route and locally defined response window. |
| Incident owner | Contain unsafe behavior and coordinate clinical and technical recovery. | Maintain a named alternate and an independent communication channel. |
Automation bias is undue reliance on automated advice. In a controlled dermatology experiment, participants assessed cases before and after AI advice containing deliberately incorrect recommendations. Some initially correct decisions became wrong. Such experiments test resistance to bad advice, rather than treating approval frequency as oversight quality.
Alert fatigue describes reduced responsiveness associated with excessive alerts. A retrospective clinical study associated repeated reminders and more reminders per encounter with lower acceptance. It did not establish that overrides were inappropriate or that reducing overrides would improve safety.
- Test the reviewer — Use controlled cases with known errors to assess useful-advice acceptance, incorrect-advice rejection, and harmful reversals of initially correct judgments. Repeat testing in the intended workflow before assuming laboratory behavior transfers.
- Test the handoff — Measure whether an escalation reaches the responsible person and results in the required response. Delivery, acknowledgment, and resolution are separate observations; nominal human involvement is insufficient.
- Provide a correction route — A patient or staff member reporting an error needs an accountable reviewer who can inspect evidence and authorize correction. Record the disposition and communicate the result; a trace alone supplies no remedy.
Clinical claims and their evidence
Clinical validation assesses whether a system fulfills its specified clinical purpose in the intended population and setting. IMDRF distinguishes a clinically meaningful association, reliable production of the intended output, and clinical validation. Clinical utility concerns usefulness when the output enters care; technical correctness alone cannot establish it.
A reference standard is the best available basis for establishing the condition being evaluated. It must be distinguished from a convenient comparator. Agreement with one reviewer is not automatically diagnostic accuracy; reviewers and records can be uncertain. Preserve disagreements and define how they are adjudicated.
| Intended claim | Comparison and population | Evidence needed |
|---|---|---|
| Less documentation work | Existing documentation practice among intended users. | Total task and correction effort, accepted record quality, and consequential omissions; generation speed alone is insufficient. |
| Better access | Existing referral workflow for the intended patients. | Completed downstream work and unresolved exceptions, with population coverage and exclusions reported. |
| Better patient outcomes | A defined care alternative in the target setting. | An endpoint—an outcome measured by the study—that captures patient-relevant benefit and harm. |
- Retrospective evaluation — Existing records support evaluation only after fixing the intended decision time, available predictors, outcome definition, and prediction horizon. Later documentation must not leak into earlier decisions.
- Patient separation — Patient-disjoint tests assess performance on unseen people. Repeated encounters from one person can otherwise cross dataset boundaries.
- Time and site separation — Later-period tests probe changing practice; external validation tests another setting. These dimensions can overlap, and their names do not prove independence: returning patients may appear in later periods or multiple institutions.
Shortcut learning can reward the wrong evidence. A pneumonia-imaging study showed that radiographs carried hospital-location signals. When hospitals differed in disease prevalence, recognizing location helped predict the label without exclusively detecting pathology. Strong pooled internal performance therefore did not establish transport to another hospital.
Target mismatch is different: the system may predict its label accurately while serving the wrong objective. A healthcare allocation study found that spending-based scores understated need among Black patients relative to White patients at the same score. Unequal utilization made spending an inadequate stand-in for health need.
- Validate substituted outcomes — A surrogate endpoint substitutes another measure for direct patient benefit. Correlation with health outcomes is insufficient: intervention-induced changes in the surrogate must predict benefit in the relevant context. Improving a convenient biomarker or process measure can leave patients no better off.
- Preserve the evaluation boundary — Evals covers success criteria and graders; Data Quality and Curation covers collection and selection. A held-out split cannot repair a population that omits the intended users or operating conditions.
Errors, prevalence, and response burden
Prevalence is the proportion with the target condition or event. Sensitivity measures detection among true cases; specificity measures non-alerts among non-cases. Positive predictive value, or PPV, measures true cases among alerts. Their denominators answer different operational questions.
Assume two populations of 1,000 people, with sensitivity and specificity both fixed at 90%. Only prevalence changes.
| Count or ratio | 10% prevalence | 1% prevalence |
|---|---|---|
| True cases | 100 | 10 |
| Detected / missed | 90 / 10 | 9 / 1 |
| False alerts / true negatives | 90 / 810 | 99 / 891 |
| All alerts | 180 | 108 |
| PPV | 50% | 8.3% |
Lower prevalence raises the false-alert share from 50% to about 91.7%, while total alerts fall. Alert usefulness and absolute review volume move differently.
Calibration means agreement between estimated risks and observed event frequencies. If patients assigned 20% risk experience the event about 10% of the time, the estimates overstate risk there even if ranking is useful. New disease incidence or practice patterns can invalidate calibration. Evals explains its general assessment.
- Sampling uncertainty — Report counts and denominators alongside performance estimates. More observations generally reduce sampling uncertainty, but do not remove bias. Sparse subgroup samples can leave important deterioration difficult to distinguish from noise.
- Confidence intervals — A 95% confidence procedure covers the fixed population parameter in approximately 95% of repeated samples under its assumptions. It does not assign a 95% probability to an individual patient's outcome.
- Thresholds and consequences — Thresholds translate scores into actions. Missing a genuine warning and triggering an unnecessary assessment can have unequal consequences. Clinical experts must define those costs for the workflow; an extra assessment is not harmless in every setting.
- Coverage and response capacity — Evaluate errors among accepted predictions together with the fraction accepted. Abstention transfers demand to reviewers; measure the resulting handoffs. For conversational interventions, also score category and timing, since inappropriate interruption can obstruct useful support.
Evaluation during real clinical use
Prospective evaluation collects evidence as new cases arrive. In silent deployment, outputs are withheld from care decisions. Supervised live evaluation exposes users to outputs and can reveal reliance, disruption, and safety problems that silent performance cannot establish.
Random assignment uses an unpredictable chance process to allocate study conditions. Allocation concealment prevents recruiters knowing the next assignment before enrollment. These reduce selection bias; chance imbalance, unequal follow-up, and measurement bias can remain. Merely comparing contemporaneous AI users and nonusers does not randomize their differences.
Visibility and assignment change the evidence
ExampleSilent performance cannot reveal reliance on visible outputs.
Read the diagram as text
- Eligible new cases.
- Outputs withheld.
- Live-data performance evidence.
- Random assignment.
- AI-supported care.
- Usual care.
- Compare prespecified outcomes.
- Eligible new cases → Outputs withheld: Silent design.
- Outputs withheld → Live-data performance evidence: Evaluate withheld outputs.
- Eligible new cases → Random assignment: Randomized design.
- Random assignment → AI-supported care: AI arm.
- Random assignment → Usual care: Comparator arm.
- AI-supported care → Compare prespecified outcomes: Observed outcomes.
- Usual care → Compare prespecified outcomes: Observed outcomes.
| Design concern | Interpretation |
|---|---|
| Usual-care comparator | Describe the actual alternative, staffing, resources, exclusions, and delivery conditions. A pragmatic trial aims to inform ordinary practice, but exceptional support can limit applicability. |
| Observational comparison | Specify the interventions and comparison that would answer the causal question. Target-trial reasoning clarifies design; it cannot remove unmeasured confounding or repair inadequate data. |
| Measured outcomes | The ambient-documentation trial supports practitioner workload findings. It did not collect patient-reported comfort and disclosure outcomes, and its note-quality assessment was not comprehensive independent clinical adjudication. |
Before-and-after improvement can reflect changed staffing, patient mix, or selective adoption. Comparative evidence must address these alternatives. Supervised pilots can reveal implementation failures without establishing causal patient benefit; randomized studies require a suitable comparator and outcome assessment rather than randomization alone.
DECIDE-AI addresses early clinical evaluation; CONSORT-AI addresses AI trial reporting. Their descriptions of users, inputs, interfaces, decisions, versions, and errors make evidence interpretable. Reporting completeness does not itself demonstrate effectiveness.
Deployment scope and operating limits
Deployment authorizes a particular use under particular conditions. A successful evaluation does not automatically extend to different patients, sites, hours, or decision authority. The initial contract should name supported tasks and users, excluded uses, required training, and accountable clinical and technical owners.
Local acceptance testing connects that contract to operational evidence. Test wrong patient context, stale information, service outages, concurrent updates, duplicate requests, and reviewer overload. A passing model benchmark cannot substitute for these integration and human-performance checks.
| Proposed acceptance condition | Evidence and owner | Limit on use |
|---|---|---|
| Relevant population and task | Clinical owner reviews representative evaluation and exclusions. | No unsupported expansion of population or purpose. |
| Usable information | Integration owner verifies required resources, meaning, and freshness. | Incomplete or unsuitable context follows an exception path. |
| Effective professional review | Clinical owner observes detection and correction of seeded errors. | Restrict use when reviewers cannot perform the required assessment. |
| Continuity and coverage | Operations owner tests the alternative workflow and communication. | Use only where interruptions and outstanding work can be handled. |
Define suspension authority and conditions before release. Unavailable reviewers, unresolved critical incidents, or unmet operating requirements need an explicit response, rather than an assumption that someone will notice and intervene.
Institutional permission, technical performance, local usability, and demonstrated benefit answer different questions. Applicable external requirements depend on intended medical purpose and jurisdiction and require qualified review. Calling software an assistant, model, or app does not settle its status.
Monitoring a changing care process
Distribution shift changes the patients, data, or operating setting relative to evaluation. New ordering practices, record templates, or staffing can alter recorded signals without an equivalent biological change. Monitoring must connect technical observations to the care process; Observability covers the general instrumentation.
Delayed outcome labels remain unavailable until follow-up matures. Monitor delivery failures and incidents immediately, but compute outcome rates on appropriately followed cohorts. Pending or missing follow-up remains unknown; it is neither a negative outcome nor evidence of successful prediction.
The response changes the label
ExampleAn avoided event can follow a useful warning.
Read the diagram as text
- Patient condition.
- Recorded inputs.
- Risk prediction.
- Clinical assessment.
- Intervention.
- Observed outcome.
- Patient condition → Recorded inputs: Shapes.
- Patient condition → Observed outcome: Affects.
- Recorded inputs → Risk prediction: Model input.
- Risk prediction → Clinical assessment: Informs.
- Clinical assessment → Intervention: May authorize.
- Intervention → Observed outcome: May change.
| Layer | Useful signal | Interpretation limit |
|---|---|---|
| Inputs | Required information and absence reasons. | Missing documentation does not establish a negative finding. |
| Delivery | Outputs received and actions recorded. | A generated completion claim is not the resulting environment state. |
| Output quality | Errors by category and evaluator version. | Changed judge behavior can resemble changed application quality. |
| Human response | Edits, overrides, unattended work, completed escalations. | Approval alone does not establish correctness. |
| Outcomes | Mature follow-up, subgroup counts, and uncertainty. | Unavailable labels cannot certify continued clinical performance. |
An intervention can change the outcome used to judge a prediction. If an alert prompts effective treatment and the event never occurs, that observation alone cannot establish a false alert.
Reviewer feedback creates a different problem. If an interface encourages rubber-stamping, model-influenced approvals can become misleading evaluation or training labels. Independent assessment can expose disagreement, but disagreement still needs adjudication rather than automatic treatment as truth.
- Preserve investigation context — Retain necessary source and resource versions, creation and revision actors, and timestamps. Use access and retention limits appropriate to the data.
- Reassess absolute risks — Changes in incidence and case mix can break calibration despite useful ranking. Inspect subgroup estimates and their support before retaining an action threshold.
- Track application changes — Model, prompt, and reference changes belong in the monitored configuration. Stable test cases can also miss newly encountered ambiguous or combined requests; production failures should refresh coverage.
Continuity, correction, and revalidation
Rollback changes future behavior; it cannot retract an utterance already heard. An incorrect note or missed escalation can require clinical follow-up even after the software is restored. Containment, record correction, and care recovery therefore need separate completion criteria.
A proposed incident procedure identifies affected versions and work, notifies responsible owners, contains unsafe output, and verifies recovery. Restoration must address the cause and demonstrate readiness; a successful restart alone is insufficient.
Restoration preserves outstanding obligations
ExampleService status changes do not erase affected work.
Identify affected work.
Read the diagram as text
- Service S.
- Affected work W.
- Initially operating.
- Suspended.
- Under revalidation.
- Resumed within limits.
- Follow-up outstanding.
- Work reconciled.
- Service S → Initially operating: Initial status.
- Service S → Suspended: Containment status.
- Service S → Under revalidation: Recovery status.
- Service S → Resumed within limits: Verified status.
- Affected work W → Follow-up outstanding: Initial disposition.
- Affected work W → Work reconciled: Verified disposition.
- Detect. Identify affected work. Active: Service S, Affected work W, Initially operating, Follow-up outstanding. New: Service S, Affected work W, Initially operating, Follow-up outstanding.
- Contain. Suspend unsafe output; preserve follow-up. Active: Service S, Affected work W, Initially operating, Follow-up outstanding, Suspended. New: Suspended.
- Revalidate. Test recovery while obligations remain. Active: Service S, Affected work W, Initially operating, Follow-up outstanding, Suspended, Under revalidation. New: Under revalidation.
- Resume. Record verified limits and reconciliation. Active: Service S, Affected work W, Initially operating, Follow-up outstanding, Suspended, Under revalidation, Resumed within limits, Work reconciled. New: Resumed within limits, Work reconciled.
- Continue care — Use a trained alternative workflow with independent communication. After restoration, reconcile interim documentation and orders, interrupted work, and the backlog. Tested restoration matters more than the mere existence of backups.
- Preserve record history — Follow local amendment procedures. Historical CMS guidance illustrates corrections that retain distinguishable original content, authorship, and dates, rather than silently replacing the record.
- Locate downstream work — Use source and version relationships to identify affected notes and derivatives. Missing provenance leaves the affected set uncertain.
Convert clinically judged failures into repeatable regression cases. One mental-health team records the conversation, expected behavior, category, and intervention turn, then replays that boundary in continuous integration. This preserves an expert's expected behavior across prompt, model, and guardrail changes; it does not replace broader clinical evaluation.
Revalidation reassesses the changed system for its intended use. Model updates, prompt changes, terminology mappings, reference replacements, and broader authority can affect different parts of the evidence. Recheck the affected integration, human interaction, and outcome assumptions before resuming or expanding use.
| Operating decision | Proposed criterion |
|---|---|
| Resume | Verified recovery and supported operating conditions. |
| Narrow | Evidence supports only a smaller scope. |
| Replace or retire | Benefit, oversight, or recovery requirements remain unmet. |
Open questions
Transporting performance across institutions remains difficult because acquisition practices can carry predictive shortcuts. Progress would include external evaluations that separate patient differences from recording differences and preserve clinically useful performance after those distinctions are tested.
Sustainable oversight requires useful intervention without overwhelming reviewers. Repetition can reduce acceptance, but overrides are not inherently mistakes. Progress would demonstrate appropriate correction and completed responses under realistic alert burden, rather than simply lowering override rates.
Monitoring intervention-altered outcomes remains unresolved: successful prevention can resemble a false prediction. Progress requires evidence separating prediction error from effects of the response, because the untreated outcome remains unobserved.
Local clinical customization can outgrow centralized review and engineering capacity. Learning local conventions is difficult when professional preference and safety requirements overlap. Progress would preserve shared safety tests while showing that specialty-specific changes improve the intended workflow without degrading other supported uses.
