AI System Design: From Idea to Production

Hi, everyone. I'm Apoorva, and I'm a data scientist turned developer advocate currently at MongoDB. I spent the first years of my career building machine learning applications for various cybersecurity use cases, and I now use that applied machine learning knowledge to help AI builders successfully build AI applications with, uh, MongoDB and Voyage AI.

In this talk, you'll learn how to think through building AI systems end-to-end from idea to production. We'll take a real-world use case and walk through all the steps of designing it, and hopefully in the end you'll walk away with a repeatable framework that you can apply to, uh, any AI system you design.

Now, you might be thinking, in the age of AI, do we even need to think about what we build? Just write code it and ship it, right? But that's actually where I want to start.

Now, here's the problem with that. Write coding works great when you're building for fun, the stakes are low, and you can easily eyeball whether the output of what you're building is right.

But the moment you're building something real, something other people depend on, something with real consequences, "Just ship it," is actually kind of dangerous. Uh, and it's not just me saying it.

Folks from Anthropic and OpenAI who are so bullish on AI coding are saying the same thing. These are actually quotes from their talks over the past few months.

Specs are the new code. The art is in defining the product requirements, the system design, and evaluation criteria so you can be confident that your AI coding buddies are building the right thing.

And the rest of the talk is just about that. How do we do this well?

It's useful to think about it as a framework. Four phases. You start with product requirements. What are you actually building, uh, for whom, and what are the constraints? Then system design, the data, the architecture, the patterns that actually help you meet these requirements.

Then comes evaluation and monitoring. How do you know, uh, what's being built actually works before and also after you ship? And finally, how do you optimize for cost? Uh, not just for accuracy, but also for cost, latency, and reliability before you ship, uh, and/or as you find gaps in production.

Instead of talking abstractly through a framework, I thought, why not, uh, apply it to a real-world use case so you can see how exactly each decision flows from, uh, one stage of the ne- framework to the next?

So let's take the example of a health insurance claims review system.

So adjudicating health insurance claims has historically been an extremely manual process where, uh, human medical reviewers have to, uh, cross-reference clinical guidelines, insurance coverage policies, uh, patient history and stuff to decide whether or not, uh, a medical treatment, procedure, or medication is covered by a patient's health insurance policy.

Now, if you live somewhere that has free healthcare, uh, even there, there's things that are covered, uh, and things that are not by your healthcare system, so you're still subject to, uh, some version of this process.

Now, our goal with building the system is to see if we can improve or simplify the experience for, uh, medical reviewers with the help of AI.

So let's see what the product requirements stage of the framework for this application might look like.

So first thing to do is quantify the business problem. The business problem should focus on something specific, uh, clearly state who the users of the application are, uh, the current state of things, and quantify the user's pain point.

It should not prescribe what the system is going to be, whether it's going to be an agent, a multi-agent system, something else. Uh, that'll come later.

So here's what the business problem for our claims review system could look like. Medical reviewers at MDB Health spend an average of two days processing claim review requests, which is four times the industry standard for non-urgent cases and twelve times the industry standard for urgent ones.

Delays at this scale postpone patient care, uh, particularly for time-sensitive treatment. So as you can see, this business problem is user-specific. It says this is meant for medical reviewers.

Uh, it states the current state, which is, like, this manual process that leads to them spending two days processing, uh, claim reviews. It's measurable because it states some baselines.

Uh, it's solution agnostic. It doesn't really tell you, uh, what the system should look like. Uh, and it's very focused on a specific problem.

Next, you want to gather any business constraints for the application, so any regulatory, uh, compliance requirements, any constraints on data leaving your organization, procurement constraints, meaning are certain vendors, uh, not approved for use within the organization.

Uh, all of these would be good to know before you even start designing the system. So for our application, say these are some of the business constraints. Patient data has to stay within the approved cloud environment.

Um, only models available on the approved cloud can be used. Um, there's some use cases where human review is required, so any complex use cases require a senior physician's review.

Um, what else? Any denial decisions must be, uh, reviewed by a human reviewer. So there's cases which you can't fully automate with AI. So yeah, all of these are useful design inputs, so you want to have these sooner rather than later.

Also good to know of any performance requirements. Uh, do we need sub-millisecond responses, or do we have some leeway there? Uh, how much can we spend per month on LLM inference?

Any uptime SLAs to be aware of? Next, if you're building an AI product, identifying the role of AI, uh, in your product is helpful. And I like to think about the role of AI along three dimensions.

First, is AI critical to your product or complementary? In this case, complementary, since, um, claims are already happening without... claim reviews are already happening without AI, but they are just slower.

Next, will this be a reactive or proactive system? Meaning, is it triggered by a user or an event, or does it proactively do things? In this case, it's going to be a reactive system since, uh, it's triggered by a claims application actually being submitted.

And finally, what's the level of autonomy, uh, for AI in the application? So in our case, the business constraints prescribe human review in certain cases, so our system can be semi-autonomous at max.

And finally, as part of defining what you're solving for, you also want to come up with one to two, uh, success metrics aligned with the business problem so you can capture what success looks like.

A good success metric is specific, measurable, achievable, relevant, and time-bound. And here's an example of a success metric for our claims review application. So it says, it says reduce the average processing time for urgent claim review requests from two days to one hour, uh, within ninety days of launch.

So it's super specific. It's measurable because it's saying, uh,

the processing time needs to be one hour. Uh, it seems achievable because, um, there'll be AI in the loop, so hopefully any information gathering, uh, getting an initial recommenda-re-recommendation will be faster.

Uh, it's very relevant and rooted in the business problem from before, and it's time-bound because it gives you a, a window for when we need to start seeing success.

Okay, so at this point we've figured out, uh, the what. Let's look at the how.

The first thing you want to think about here is your data strategy. What does... Uh, what data does your application need? Do you even have access to it? Where does it reside?

Uh, is it ready for consumption in its raw form, and so on.

So for our claims review application, these are the data sources that we might need, where they reside, and what their raw access formats might look like. So we need, uh, clinical guidelines probably, uh, which describe what care is appropriate for a given condition.

Uh, MDB Health coverage policies that state what MDB Health will and will not pay for.

And finally, the patient's claims history to know any, uh, past conditions, uh, procedures that were performed, when they were performed, and so on.

Let's assume that the clinical guidelines and coverage policies are stored as PDFs in something like Confluence, uh, and claims are stored in MongoDB as and when they are processed.

The next thing to think about is data update frequency. How often do the data sources update? Now, if you're doing any sort of data processing on these raw data sources, and you typically are, these pipelines are going to have to, uh, run at the right cadence to keep up.

Otherwise, our system is going to work off of stale information, and that's not good, especially in such a high-stakes use case.

So let's work this out for our claims review system. Clinical guide, these probably update annually. Uh, internal coverage policies may be quarterly, and the patient's claim history needs to be updated whenever a new claim is processed.

And in our success metrics, if you recall, uh, we've said we want to aim for a one-hour timeframe for adjudicating urgent claims. So these will probably get updated hourly.

Next, can the data be consumed directly by our application, or does it need any processing? For clinical guidelines, which are typically long documents, you probably want to chunk them, uh, embed the chunks, and also extract metadata such as procedure names, publication dates to make it suitable for retrieval using something like vector search or hybrid search.

Same goes for, uh, coverage policies. But for patient claims history, which is already well formatted and stored in MongoDB, we might just need to remove, uh, personally identifiable information, uh, before passing it to an LLM.

Next, we want to think about what retrieval techniques would be best suited for each of the data sources. And you kind of already start thinking about this in the data processing step, but let's quickly formalize this.

Now, clinical guidelines and coverage policies, um, are well suited for vector search, but they can contain, uh, medical terminology such as diagnosis codes, procedure codes, uh, that might not be picked up by vector search.

So, um, but these will be picked up by metadata filters or something like keyword search. So for these, you'd either want to, uh, use vector search with metadata pre-filtering or hybrid search.

For getting the patient's claims history, you can do an exact match on the patient name or ID, whatever the patient identifier is in the system.

Next, you wanna think about what the system architecture looks like. Now, it can be tempting to jump straight to building an agent, there's so much hype around them, uh, or let a coding agent decide what the system architecture should look like, but you risk ending up with an over-engineered system by doing this.

So what you instead want to do is start with the simplest design, uh, evaluate it, uh, find gaps during the evaluation, and iterate from there.

To decide what the system should look like, let's first map out how an insurance claim would flow through the system, because this will help us, uh, inform the architecture.

So first, the claim request and any clinical notes from the physician are received by our system. Uh, then the system needs to retrieve any clinical guidelines and coverage policies are, that are relevant to the current claim, uh, and also retrieve the patient's claims history.

Then all of this information, along with, um, the claim itself, uh, and any system prompts on how the LLM should generate... go about generating the recommendations, all of these are passed to the LLM, um, and it uses those to generate, uh, a recommendation of whether the claim is...

claim should be approved or rejected. Now, if it's a complex case, our, uh, system, or the LLM at this point, should, uh, send its initial recommendation, um, to a senior physician.

And if the LLM's recommendation is to deny the claim, then it should also be escalated to a human medical reviewer.

And finally, the final decision should be logged to MongoDB, which is where we are storing our claim documents, uh, along with a reasoning for why the claim was approved or rejected.

Now, let's talk about some of the most common AI design patterns that we see, uh, in AI systems today, and see which of those might apply to our system.

So the first one is RAG, uh, short for Retrieval-augmented generation. You all probably know it at this point. Um, basically in these systems, you simply augment an LLM's pre-trained knowledge with external knowledge sources.

So information from external knowledge sources. Then there's AI agents, where you give, uh, a single L- LLM or multiple LLMs in the case of multi-agent systems, full autonomy to complete tasks end-to-end with the help of a set of tools.

And then there's agentic systems, which cover, uh, a broad spectrum of semi-autonomous AI systems.

A common architecture pattern for, uh, these types of systems are control flows, where LLMs perform certain tasks within the workflow, but the sequence of steps in the workflow itself is predetermined by design or by code.

Then there's LLM-as-a-router, where LLMs have limited autonomy, in that their job is simply to, uh, categorize incoming requests and route them to different downstream workflows.

Then you have human-in-the-loop, where an LLM or agent can take certain actions, but it needs human intervention, uh, somewhere in the process.

And finally, there's fine-tuning. So if you're observing that the LLM's failures are behavioral rather than data or orchestration issues, uh, or if you need superior performance on a domain-specific task, then fine-tuning is, uh, a good technique to explore.

I don't have the time to go too deep into these here, but if you'd like to learn more about these design patterns, uh, I'm going to leave this resource here for you to check out later.

So we've thought through what the workflow of the system should look like. We've spoken about a few different AI design patterns. Uh, now let's apply those and decide what design patterns, um, we'd need to implement for our system.

So since we need to retrieve clinical guidelines, coverage policies, and stuff to inform the decision, this is... this definitely looks like a RAG component, so we need some form of RAG.

The workflow is pretty structured in that the LLM first provides its recommendation on whether or not a claim should be approved or rejected. And in very specific and well-defined cases, uh, it needs to escalate claims to human reviewers.

So this sounds like a controlled workflow. You could also argue that this looks a little bit like LLM-as-a-router, especially if you use, um, an additional LLM call to decide what a complex case looks like, uh, unless this is already well-defined.

So yeah, this could be an LLM-as-a-router, but, uh, for now, let's just go with control flow.

And finally, there's also an element of human-in-the-loop, right? Since human reviewers are very much a part of the workflow.

And finally, in the design phase, you also want to think about the end user experience and how you'll collect feedback from your users to improve the system. Here's some questions to ask yourself in this stage to help think through, uh, what the UX and feedback mechanism should look like.

So first, what does the system take as input? In our case, it's going to be a claim request form of some kind.

Then what does the system produce as output? So in our case, it's going to be, uh, an approval or denial verdict, along with an explanation for why, uh, a claim was approved or rejected.

Where does the system live? Does it have its standalone app? Is it embedded in an existing website? Is it a Slack bot? Uh, and so on. Uh, so for

Our application, it's very likely going to be embedded within MDB Health's main website, let's say. Uh, what triggers the system? So in our case, very clearly a claim submission.

What's the human's role? So in our case, they are going to review the system's assessment and make the final decision in some cases.

Um, how does the system explain itself? In our case, um, it explains itself by providing citations, so quoting, uh, the clinical guidelines and coverage policies that, uh, inform the system's assessment.

And finally, how users give feedback. So remember, for our system, um, the consumers are medical reviewers. Um, so they can provide feedback by overriding the verdict of the LLM and also log a reason for why.

Uh, they can also... We can maybe also have them flag any irrelevant citations. So if the LLM is hallucinating, um, clinical guidelines and policies or, uh, yeah, quoting the wrong ones for a particular claim, then we can have them flag those as well.

All right. So at this point, we've decided what the architecture of our system should look like, uh, but we also need to decide what our stack should look like.

Once again, it might be tempting to have a coding agent recommend, uh, what tools to use, but they might not necessarily apply to your system, or you might not be able to use them based on the constraints that we identified during, uh, the product requirement stage.

So we had constraints around, uh, models, uh, need to be supported by our cloud provider and things like that.

Really going to get into this too much since our constraints are, uh, well, hypothetical. Uh, but I just wanted to mention that this is the next step and the different tooling decisions to be made, uh, such as what model to use.

Uh, there's a, a bunch of data processing tools out there that have some nice out-of-the-box data processing capabilities.

If retrieval is part of your system, then, uh, and you're using things like vector search, then you might need to decide what embedding models to use, what vector database to use.

Um, then there's orchestration frameworks and so on. So I'm just gonna leave this here for reference later on.

Next, let's talk about evaluation and monitoring. How do you know that what you've built works before and also after you ship? So evaluation is before, uh, monitoring is after you ship, and you need both of them.

But before we get into evaluation metrics, let's talk about guardrails because these weren't really th- really a thing pre-LLMs.

The reason they've become a topic of conversation is because unlike traditional software, LLM-based systems are going to... are probabilistic systems and can produce outputs that are unexpected, incorrect, or even harmful.

And guardrails are an attempt to mitigate these risks and ensure that your system behaves within acceptable boundaries. And what those boundaries are, are something you need to define. So for inputs to your system, the goal is usually to detect invalid, irrelevant, or harmful inputs.

Uh, so for a claims review system, an input like, "Write me a poem," is irrelevant, and our system should reject this claim.

Similarly, for outputs, our goal is to detect invalid, incorrect, or hallucinated, uh, or harmful outputs. So for our claims application, we'll consider, uh, a response to be invalid if the system doesn't provide citations for, uh, what informed an approval or denial.

Now, guardrail compliance is not the only thing you want to measure. You want to measure the quality of the responses, um, create and measure domain-specific accuracy metrics, and also general system health.

So for input guardrail compliance, uh, for our system, we can calculate a claim rejection rate. How often is our, uh, system rejecting claims because they don't obey, uh, our input guardrails?

Now, if it's rejecting too many times, then that's a call for investigation. But if you didn't measure this in the first place, then you wouldn't have anything to investigate.

So, uh, there we go. That's why you need metrics. For output guardrail compliance, we can have something like a missing citation rate. How many times is the LLM producing responses without citations?

For response quality, we can measure faithfulness, which is whether or not the claim's approval or rejection is actually rooted in the retrieved, uh, information, like clinical guidelines and policies.

Then you want to define at least one domain or application-specific metric. So in our case, something like claim processing time might be a good one because that's really, uh, our north star.

And system level metrics are usually, uh, around overall system health. So, uh, what's your system's average token cost, token usage, uh, average number of turns in a conversation? Uh, so here, just to, um, not have another latency metric, uh, I put down something like cost per recommendation.

So once you've evaluated your system and shipped your product in front of users, then comes monitoring the system for regressions in real time. Now, in production, you still want to track the metrics that you used for, uh, offline evaluation, so the metrics that we [audio cuts out] about.

But you might also be able to track additional metrics that can act as, uh, implicit indicators of your product's success or, like, how, um, How happy users are with your product and overall system health.

For example, for the claims review app, you can track how often a human reviewer overrides the AI verdict. Uh, now, you want this rate to be low, uh, because that means the system is not doing its job.

So if you ever start seeing is, seeing it increasing about, above a certain threshold, then you need to go and investigate. Or you can also track something like how long does it take for a human to review the AI recommendation?

If it takes too long, which means, um, this could mean that the responses are verbose, confusing. So yeah, these would be, uh, some good indicators to track, uh, once you ship your product to production.

Now, once you have a working prototype that you've evaluated, accuracy looks good, it's time to move to production, right? Well, what about cost, latency, and reliability? Uh, these constraints become absolutely non-negotiable as you're moving your product to production.

So you might need to do a few more iterations, evaluations, and testing, uh, between accuracy looks good and moving to production.

So let's talk about some optimization techniques for each of these. So optimizing for accuracy. In LLM-based applications, it really comes down to optimizing the information that, uh, ends up in the LLM's context window.

And here are some techniques for you to refer to later, but let's quickly talk about which of these, uh, might apply to our claims review application. So prompt engineering, of course.

Uh, re-ranking is another good one because, uh, retrieval is a big component of it, so, uh, a re-ranker will make sure that the information being retrieved, um, is surfacing information in the right order of relevance, which is important when working with LLMs.

Uh, we are also persisting patient history, uh, to MongoDB, so our application already has some form of memory. But you could also think about other types of information that might be, uh, useful to persist across sessions.

Here are some techniques for cost and latency optimization. For our use case, semantic caching could be useful to expedite decisions based on, uh, similar claims from the past. Uh, batch processing perhaps to process claims in batches instead of one at a time.

And finally, optimizing for reliability. Most of these would be useful for our use case since they're mainly solving for API failures. But, uh, I want to specifically call out structured outputs, um, which would be a good one to make sure that our system is always producing a structured output which contains not just the decision, but also the

citations. This brings me to the end of my talk, but I wanted to leave you with a few takeaways.

Think deeply about your product's requirements before having AI generate any code. The product spec is the hard part now, uh, it's not the code anymore.

Your latency budget, your cost ceiling, any regulatory requirements, all of these shape every architectural decision downstream. So, uh, good to identify, uh, business and performance constraints before starting to design your application.

Design the simplest system that meets your needs. Evaluate it, and then iterate from there. The most common mistake I see is over-engineering the solution before knowing what's actually failing or not even evaluating what's actually failing.

And that brings me to evaluation. Uh, build evaluation in from the start. Uh, you can't improve what you can't measure.

Link to our Gen AI Cookbook that has a ton of examples of, uh, different retrieval technique, agentic design patterns, and, uh, some of the evaluation and op-optimization techniques that, uh, I was talking about.

And yeah, thank you so much for taking the time to listen to my talk. I'll see you at the conference. [mouse clicking]