AI System Design: From Idea to Production

If an AI coding tool can build an application, why spend time designing it first? For a low-stakes experiment, where you can inspect the output and easily tell whether it is right, generating code and trying it can work well. The difficulty starts when other people depend on the result and mistakes have consequences. Apoorva Joshi points to talks from Anthropic and OpenAI to emphasize that enthusiasm for AI coding does not remove the need to define what the software should do.

The specification must define the product requirements, system design, and evaluation criteria. Those three pieces give a coding assistant a target and give its human collaborators a way to judge the result. Writing code faster is useful only if the application implements the right behavior.

The design process has four phases, with each supplying decisions needed by the next:

1. Product requirements: identify the users, the problem, and the constraints.
2. System design: choose the data, architecture, and patterns that satisfy those requirements.
3. Evaluation and monitoring: establish whether the system works before launch and continues working afterward.
4. Optimization: improve accuracy, cost, latency, and reliability before shipping and as production exposes gaps.

Consider an internal health insurance claims review system. A medical reviewer must cross-reference clinical guidelines, insurance coverage policies, and patient history to determine whether a treatment, procedure, or medication is covered. Publicly funded healthcare systems also distinguish between covered and uncovered care, so the underlying review problem is not limited to private insurance. The goal is to use AI to simplify the medical reviewer's work. In the fictitious MDB Health example, the scope is the internal review system, not the external claim-submission process.

Start with a business problem that names the users, describes their current situation, and quantifies their pain. Do not prescribe an agent or a multi-agent architecture yet. In Joshi's hypothetical MDB Health example, medical reviewers average two days per claim review, described as four times the non-urgent industry standard and twelve times the urgent standard. These comparisons are assumptions in the example, not established industry benchmarks. The consequence is concrete: delays in review can postpone patient care, particularly time-sensitive treatment. This is a focused problem with a measurable baseline, while leaving the solution open.

Next, collect the constraints that determine what the system is allowed to do. Regulatory requirements, restrictions on data movement, and vendor procurement approvals can rule out an otherwise attractive architecture. MDB Health's example constraints are:

• Data boundary: patient data must remain in the approved cloud environment.
• Model availability: only models available in that cloud may be used.
• Complex cases: a senior physician must review them.
• Denials: a human reviewer must review every proposed denial.

These requirements establish limits on automation before any model or framework is selected.

Performance requirements belong in the same early conversation. How quickly must the system respond? What is the monthly LLM inference budget? What uptime SLA must it meet? A system with a strict response deadline or spending ceiling has a different design space from one with room for slower, more expensive processing.

AI's role can then be described along three independent dimensions:

The human-review requirements are part of the product contract, not optional exceptions to an otherwise autonomous system.

Choose one or two success metrics tied directly to the business problem. Joshi's proposed target is to reduce average urgent-claim review time from two days to one hour within ninety days of launch. It is specific, measurable, relevant, and time-bound; its proposed basis for achievability is faster information gathering and faster initial recommendations. This is a launch objective, not a reported outcome.

With the requirements established, the first design question is data access. What information does the application need, where does it reside, and can the system consume its raw format? Clinical guidelines describe appropriate care for a condition. MDB Health's coverage policies describe what the insurer will pay for. Patient claims history supplies previous conditions, procedures, and the dates on which they occurred. Assume the guidelines and policies are PDFs in Confluence, while processed claims are stored in MongoDB.

The ingestion pipeline must keep pace with source changes. Otherwise, even a well-behaved model can make recommendations using stale evidence.

Joshi connects the one-hour urgent-review objective to a possible hourly refresh of claim history. The design dependency matters more than the particular schedule: a source's update frequency and the application's freshness needs determine when its processing pipeline must run.

Long guideline and policy documents need preparation for retrieval. Chunk the documents, embed their chunks, and extract metadata such as procedure names and publication dates. Patient history is already structured in MongoDB, so its preparation is different: remove personally identifiable information before passing it to the LLM.

Retrieval should also differ by source. Semantic similarity is useful for finding relevant passages, but medical diagnosis and procedure codes may require exact matching that vector search alone can miss.

Metadata filters constrain the candidate documents; keyword search can capture terminology and codes that need lexical matches. Patient history instead requires selecting the correct patient's records, not finding records about similar patients.

Start with the simplest design, evaluate it, and add complexity in response to observed failures. Defaulting to an agent—or letting a coding assistant select the architecture without this analysis—can produce an over-engineered system. Mapping the actual claim-processing sequence makes the necessary responsibilities visible.

The claim moves through the following steps:

1. Receive the claim request and the physician's clinical notes.
2. Retrieve relevant clinical guidelines, coverage policies, and patient claims history.
3. Supply the LLM with the retrieved evidence, the claim, and instructions for generating a recommendation.
4. Generate an approval or denial recommendation.
5. Send complex cases to a senior physician and proposed denials to a human medical reviewer.
6. Record the final decision and its reasoning in MongoDB.

The distinction between a recommendation and a final decision is essential. A proposed denial enters human review; it does not become a finalized denial merely because the model generated it.

Several AI design patterns can participate in this workflow. Retrieval-augmented generation, or RAG, supplies external knowledge alongside the model's pretrained knowledge. In Joshi's terminology, agents use one or more LLMs and tools to complete tasks autonomously, while agentic systems cover a broader range of semi-autonomous designs. These are useful architectural conventions for discussing who controls execution.

Two patterns make that control boundary especially clear:

• Control flow: code predetermines the sequence of steps, while an LLM performs particular tasks within it.
• LLM-as-a-router: the model categorizes an incoming request and sends it to an appropriate downstream workflow. Its autonomy is limited to that routing decision.

The router diagram makes the distinction visible: one input reaches an LLM router, branches to one of three workflows, and then reaches the end.

Human-in-the-loop adds required human intervention somewhere in an LLM or agent process. Fine-tuning addresses a different kind of need: it is worth exploring when failures concern model behavior rather than missing data or incorrect orchestration, or when a domain-specific task requires better performance. Changing model behavior is not a substitute for retrieving the right policy or enforcing an escalation rule.

For claims review, the selected baseline combines three patterns. RAG supplies the guidelines and coverage policies. A controlled workflow produces a recommendation and applies predefined escalation rules. Human review handles the required cases. An additional LLM routing call could classify whether a case is complex if that classification is not already well-defined, but Joshi keeps the baseline as control flow rather than adding that call by default.

The escalation portion can be expressed as a small Python function. It preserves the model's recommendation while keeping human review pending; a complex proposed denial satisfies both review requirements.

from typing import Literal, TypedDict


class ReviewState(TypedDict):
    recommendation: Literal["approve", "deny"]
    required_reviews: list[str]
    status: Literal["pending_human_review", "ready_to_finalize"]


def route_recommendation(
    recommendation: Literal["approve", "deny"],
    *,
    is_complex: bool,
) -> ReviewState:
    required_reviews: list[str] = []
    if is_complex:
        required_reviews.append("senior_physician")
    if recommendation == "deny":
        required_reviews.append("human_medical_reviewer")

    return {
        "recommendation": recommendation,
        "required_reviews": required_reviews,
        "status": (
            "pending_human_review"
            if required_reviews
            else "ready_to_finalize"
        ),
    }


pending_claim = route_recommendation("deny", is_complex=True)

This function represents the routing boundary, not completed reviews or a database write. The surrounding workflow still has to obtain the required reviews before recording a final decision.

Architecture alone does not define the product experience. Specify the input, output, location, and trigger. Here, the input is a claim request form; the eventual output is an approval or denial with an explanation. A standalone application, an existing website, and a Slack bot would create different interactions. For this example, the application would likely live inside MDB Health's main website and begin processing when a claim is submitted.

In the required cases, the human reviews the system's assessment and makes the final decision. To support that work, the assessment should cite the clinical guidelines and coverage policies behind it. Medical reviewers can then override the model's verdict and record why. They should also be able to flag irrelevant citations, invented references, or real policies applied to the wrong claim. That feedback distinguishes a disagreeable recommendation from a specific failure in the evidence supporting it.

Only after defining the architecture and interaction should the team settle the tooling. A coding assistant's preferred stack may violate the approved-cloud requirements. The remaining choices include the generation model, data-processing tools, embedding model, vector database, and orchestration framework. Existing data-processing capabilities may eliminate work the team would otherwise build itself. Joshi leaves these choices open because the organizational constraints in the example are hypothetical.

Evaluation establishes whether the system works before shipping; monitoring checks its behavior afterward. Both require explicit definitions of acceptable behavior. Because an LLM can generate unexpected, incorrect, or harmful output, guardrails must state which inputs and outputs the application will accept.

For an input guardrail, a request to write a poem is irrelevant to claims review and should be rejected as an invalid request. That is different from denying insurance coverage. For an output guardrail, an approval or denial without supporting citations is invalid. These checks define boundaries, but passing them does not yet establish that the recommendation is good.

Evaluate several dimensions rather than collapsing quality into one score:

Excessive input rejection should trigger investigation. Citation presence needs a separate faithfulness check: a response can include references without grounding its verdict in the retrieved guidelines and policies. Processing time keeps evaluation connected to the original business problem, while cost per recommendation captures an operational constraint. Token usage, token cost, and conversation turns are other possible system-health measures.

After launch, continue tracking the metrics used in offline evaluation so regressions remain visible. Production also exposes signals that a prelaunch test cannot fully supply: how people interact with the recommendations and how much work the system actually saves them.

Track how often medical reviewers override the AI verdict. An increase above a chosen threshold is a reason to investigate the recommendations and their supporting evidence. Also track how long a human takes to review a recommendation. Long review times may indicate that responses are verbose or confusing, even when the model's underlying assessment is useful. These measures connect model behavior to the experience of the people the product is supposed to help.

Good prototype accuracy does not settle production readiness. Cost, latency, and reliability can require further iterations of implementation, evaluation, and testing. Treat them as requirements that must hold alongside accuracy, not work to consider only after the product is deployed.

For accuracy, focus on the information reaching the LLM's context window:

• Prompt engineering: improve the instructions that guide the recommendation.
• Re-ranking: order retrieved material by relevance so the context surfaces the most useful evidence.
• Memory: persisted patient history in MongoDB already retains information across requests; consider whether other information is useful across sessions.

These changes address different parts of the same path: instructions, retrieved evidence, and retained context.

For cost and latency, Joshi proposes semantic caching to expedite work using similar past claims, and batch processing as an alternative to processing claims individually. These are candidates to evaluate against the system's requirements; neither is presented as an implemented improvement in the example. Similarity-based reuse must still respect the patient's applicable evidence, and batching must fit the urgent-review time budget.

Reliability work must also account for API failures. The concrete technique Joshi emphasizes is structured output containing both the decision and its citations. This gives downstream code an explicit response contract. Requiring those fields does not establish that the decision is correct or that the citations support it, so the faithfulness evaluation remains necessary. Refusals and incomplete responses also need handling rather than being treated as valid decisions.

The hard work begins before asking AI to generate code: deciding what the product must accomplish and what behavior is permissible. A latency budget, a cost ceiling, and a regulatory requirement each constrain the architecture downstream. For MDB Health, those decisions lead to evidence retrieval, a controlled recommendation workflow, and mandatory human review in specified cases.

Build the simplest system that meets those needs, evaluate it, and use the failures to decide what changes next. Adding autonomy or orchestration before knowing what is failing makes the system harder to understand without establishing that it works better. Evaluation therefore belongs in the initial design, not at the end of implementation.

Joshi closes by pointing to a Gen AI Cookbook for further examples of retrieval techniques, agentic design patterns, evaluation, and optimization—the implementation exercises that follow once the product's requirements and evidence needs are clear.