AI Engineer World's Fair 2025
Data is Your Differentiator: Building Secure and Tailored AI Systems
Read the talk
Data Is Your Differentiator: Building Secure and Tailored AI Systems
A useful generative AI application needs more than a model: it needs application-specific data, controlled retrieval, observable behavior, and a repeatable path from evaluation to updates.
From a talk by Mani Khanuja
Before you start: Familiarity with language-model prompts and basic API calls will help; retrieval-augmented generation and its main components are explained as they appear.
Dancing coconuts and a deeper foundation
Dancing coconuts open Mani Khanuja’s presentation, courtesy of the Amazon Nova models. They are a playful example of what generative AI can produce—and a memory aid that will return when the application is ready to scale. First comes the less visible work that makes an AI application useful to a business.
The more ambitious the application, the deeper its foundation must be. That foundation is data: the material that represents the company, its brand, and its organization. The iceberg slide makes the supporting work visible, placing mindset, people, process, and technology beneath the business value above the surface.
Data preparation is familiar from earlier machine learning systems, but generative AI changes the requirements according to the application and its business purpose. Transformation, parsing, and loading still matter. So do the interactions between data, technology, and people. If useful information remains trapped in organizational silos, an application that could use it cannot deliver the intended experience. Getting the foundation right therefore includes how information becomes available, not just how it is formatted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The application determines the data requirements
Consider a travel agent. It needs a customer profile to personalize the conversation; learning the relevant details too late is like remembering someone’s name after they have left the room. That personalization also creates a responsibility to prevent disclosure of personally identifiable information, or PII. The agent then needs company information: whether a customer qualifies for a ticket refund depends on travel policies and the relevant airline’s rules. A fluent answer without those inputs cannot reliably resolve the request.
An employee chatbot has a different boundary. It needs company knowledge, but each employee must receive only information they are authorized to access. It may appear in Slack or a custom application, with information drawn from several sources. A marketing application adds another purpose: producing content that fits the brand. Khanuja introduces that third case without developing a separate implementation.
| Application | Required data | Governing concern |
|---|---|---|
| Travel agent | Customer profiles and travel policies | Personalization without PII disclosure |
| Employee chatbot | Company knowledge from multiple sources | Preserve employee access permissions |
| Marketing application | Brand-relevant information | Keep generated content on brand |
There is no single data requirement for generative AI. The intended interaction determines which information the application needs and which boundaries it must preserve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Data enters through the prompt, context, and model
Following the travel agent into a model interaction reveals several distinct uses of data. First, the system prompt supplies instructions and the user query supplies the immediate request. Both are data. A prompt catalog or set of templates can provide different instructions for different business requirements, with the agent selecting the appropriate template.
Next comes context, which is no longer necessarily static text bundled with the application. It can arrive from changing data services and sources. Finally, there is the model itself: an out-of-the-box model may be sufficient, while a fine-tuned model requires training data that represents the company. Data is involved before meaningful output appears, whether it supplies instructions, current knowledge, or model customization. Responsible generation adds another requirement, addressed later through guardrails.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From source material to a retrieval corpus
Amazon Bedrock supplies several parts of this application architecture. Alongside model choice, customization, and evaluation, Bedrock Data Automation handles data processing. Bedrock Knowledge Bases provides managed retrieval-augmented generation, or RAG, and Bedrock Guardrails adds configurable safeguards. The security and privacy requirements apply across agentic RAG, summarization, classification, fraud detection, and contextual chatbots. To examine the pieces more closely, Khanuja narrows the example to a contextual chatbot.
Its source material may include text, videos, and images. Understanding an image here means more than describing a portrait: a financial document may contain charts and line graphs whose business meaning must survive processing. Khanuja describes Data Automation as offering a single API for this processing. In the launch-era workflow, that means a unified asynchronous invocation, InvokeDataAutomationAsync, with configuration and S3 result handling around it—not an entire pipeline without setup.
Knowledge Bases natively integrates Data Automation, then gives the application choices about how to prepare the resulting material for retrieval:
- Choose a chunking strategy. Use hierarchical or semantic chunking, or supply custom chunking logic when the documents require it. Complex tables are one reason the choice deserves attention.
- Choose an embedding model. Convert the chunks into vector representations used for similarity search.
- Choose a vector store. Store those embeddings for retrieval.
- Maintain ingestion and updates. Knowledge Bases manages ingestion and incremental processing. The application still needs an appropriate synchronization trigger; the companion chatbot implementation uses an S3-triggered Lambda to call
StartIngestionJob.
The result is a maintained retrieval corpus, rather than a one-time upload that can be assumed to remain current.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve first, or retrieve and generate
Once ingestion is configured, the application needs a way to retrieve relevant information. The Retrieve API returns similar content, leaving the application to augment a prompt and invoke a model. Search is also configurable: hybrid search combines semantic similarity with text matching. In the current vector-search configuration, HYBRID requires a supported store and a filterable text field; it is not available for every storage configuration.
For a refund question, a Python retrieval call can expose the policy passages before any answer is generated. This function accepts the configured knowledge base ID and lets the caller choose the search mode:
python
import boto3
def retrieve_policy(knowledge_base_id, question, search_type="SEMANTIC"):
client = boto3.client("bedrock-agent-runtime")
response = client.retrieve(
knowledgeBaseId=knowledge_base_id,
retrievalQuery={"text": question},
retrievalConfiguration={
"vectorSearchConfiguration": {
"overrideSearchType": search_type,
}
},
)
return response["retrievalResults"]
The returned retrieval results are evidence for the next step, not a completed refund decision. The application can inspect that evidence before adding it to the model’s prompt.
Retrieval quality can be improved through reranking, post-processing, and query decomposition. For an integrated path, RetrieveAndGenerate retrieves context and generates the response, with parameters for controls such as reranking and query decomposition. Decomposition is useful when a complex question needs to be broken into smaller retrieval requests.
| API | Returns | Application responsibility |
|---|---|---|
Retrieve | Retrieved content | Inspect context, augment the prompt, invoke the model |
RetrieveAndGenerate | A generated response grounded in retrieval | Configure the managed retrieval and generation path |
The choice is about where to retain control. A retrieval-only call leaves an explicit boundary for application logic; the combined API packages more of the workflow behind query and generation controls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add safeguards without losing the access boundary
A working RAG path still needs responsible response generation. For the contextual chatbot, Khanuja highlights preventing PII disclosure and filtering unwanted words. Bedrock Guardrails lets teams define policies, specify examples of information that should not be shared, and apply grounding checks to help reduce hallucinations. Guardrail interventions can also reveal patterns in how users interact with the application, provided the relevant telemetry is collected.
Data Automation, Knowledge Bases, and Guardrails now form an assembled chatbot, with native integration between the retrieval-and-generation path and its safeguards. One consequential boundary remains: current AWS documentation states that guardrails cover input and generated responses, not retrieved references. Guardrails do not replace access control or source-data protection. Sensitive retrieved material needs its own controls, especially if the interface exposes references directly.
With documents ingested, the application can search and return material in image form as well as text. That brings the presentation back to the dancing coconuts. Getting responses is the starting point for the operational practices that follow, many of which apply beyond RAG. The first is chunking: the way information is divided affects generated-answer accuracy, but it is only one step in improving the application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize accuracy, cost, and latency
Optimization offers many possible interventions: better parsing, reranking, hybrid search, query reformulation, and decomposition. Choosing among them requires knowing what is wrong. Evaluation identifies the weakness; optimization addresses it. The objective also extends beyond answer accuracy to the three dimensions Khanuja emphasizes: performance, latency, and cost.
Semantic caching targets repeated work. Users do not always express the same question in identical words, so an exact-string cache can miss a reusable answer. A semantic cache looks for a sufficiently similar question that has already been answered. When a cached result can be reused, the application avoids another foundation-model invocation and the time needed to synthesize a fresh response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Observe the application, then evaluate the right stage
To diagnose failures, capture the chain from user query to retrieval hits to model response. Those records let a team investigate a complaint and determine whether the problem began with the request, the retrieved evidence, or the generated answer. Khanuja’s advice is explicit: “Please don't go into production or even do a pilot without observability.”
Telemetry requires deliberate configuration. Bedrock model invocation logging is disabled by default and requires a destination such as CloudWatch Logs or S3. Enabling it does not establish complete coverage of retrieval events or guardrail interventions; those need to be accounted for in the application’s instrumentation. Full request and response logs can themselves contain sensitive data, so the privacy requirements also apply to this new dataset. Once collected, it becomes material for evaluation.
For RAG, start with context relevance. If search returns irrelevant material, placing it in the prompt spends generation budget on a poor input. Evaluate the search results first; only then use suitable evidence to augment the prompt and ask the model for an answer. This separates a retrieval problem from a generation problem instead of treating every bad answer as a model failure.
Evaluation must fit the task. A summarization application needs summarization-specific metrics, rather than automatically inheriting the criteria for retrieval. Khanuja does not prescribe particular summarization metrics here; the broader practice is to choose evaluation according to the behavior the application is meant to deliver.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Update, test again, and scale
Never go blind: evaluation findings must lead to action. Sometimes the data needs refreshing, because stale data produces stale answers. Sometimes the retrieval or optimization strategy needs to change. The evaluation result should determine which part of the application gets updated.
An update is not the end of the loop. A defined test suite should automatically rerun evaluations after changes. That makes testing a repeatable part of maintenance, rather than a manual exercise performed only before the first release. Khanuja presents this as a way to reduce production failures and improve application quality, without supplying a measured improvement.
Repeated testing leads to the final coconut callback: crack the coconut, then scale to coconuts. The complete COCONUTS memory aid connects chunking strategy, optimization, caching, observability, never going blind, updating, testing again, and scaling. Production is the point at which this working and tested application can bear fruit—not the point at which the first generated answer appears.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Introduction to the Nova model family, including image and video generation with example workflows.
A launch-era walkthrough of multimodal extraction, blueprints, asynchronous processing and Knowledge Bases integration.
A walkthrough of adding guardrail policies to a Knowledge Bases retrieval-and-generation workflow.
Further reading
Mani Khanuja's coauthored guide to semantic and hierarchical chunking, custom processing and query decomposition.
A coauthored historical tutorial covering ingestion, synchronization, retrieval and a chatbot interface using 2024 model choices.
Instructions for enabling supported model invocation logs and configuring CloudWatch Logs or S3 destinations.
Updates since the talk
Configuration guidance for hybrid search, metadata filtering, reranking, guardrails and query decomposition.
Read the complete timestamped transcript
- 0:00
[on-hold electronic music] So today we are going to talk about data as your differentiator or maybe about coconuts.
- 0:22
Let's see. Right. So that's why I was like, I'm confident that I'll, uh, wake you up a little bit. And by the way, these coconuts dance also. [laughs] Courtesy of Amazon Nova models.
- 0:33
Okay. Enough about coconuts, and, um, we'll connect the dots and revisit our coconuts because they're dancing, and I love dancing coconuts, right? Remind us of the beach and the, you know, vacation and everything.
- 0:48
But let's come back from vacation now. Serious work to do. Okay. So we have been talking about generative AI, and, um, we all know that generative AI adds a lot of business value if done right.
- 1:04
Right? That's a, that's a big if. And, um, it can take your business to the highest level, as you can see in this image, right? But however, when you are building something that big that can change how your users interact with your applications, then think about the foundation, right?
- 1:24
The foundation has to be even deeper. And what is that foundation? That's your data. Because your data is representing your company, your brand, your organization. So you have to get the data together.
- 1:40
And we all know this since the very beginning of machine learning world, and you would say, "Mani, what is new that you are, that you are, that you are, you know, presenting over here?"
- 1:50
So the new thing is that the data requirements for building generative AI applications are different. Generative AI is different, and so there are special treatment that you need to give to the data based on your, uh, application and the business requirements.
- 2:07
So that is exactly what we are going to talk about today. And data is not just about, okay, I have to transform my data, I have to load my data, I have to parse it, I have to load it, you know, all that stuff.
- 2:19
That is still there and very important. But the most important thing is how is your data interacting with the technology? How is it interacting with your people? How... Do you still have data silos?
- 2:32
If yes, then what about your applications? Because these models can take up lots and lots of data, right? So how are we doing with all those interactions? Because no longer we can afford to be in a silo.
- 2:44
And that's what... That's the exact reason why we have to get our foundation right, and our foundation is data. So let's see.
- 2:54
Now, there are certain common applications. Let's take an example. And there are so many ap- uh, applications, the possibilities are infinite, right? However, let's, you know, break it down and see with these three applications how your, uh, data requirements may differ.
- 3:11
The first example is the travel agent. So if you're building a travel agent, what kind of data would you need? First, with that travel agent, it's going to interact with your end users, so it needs to know about your customer profile, right?
- 3:27
Because if it doesn't know about your customer profile, it's literally like, um, remembering somebody's name when they have left the room. How weird that would be, right? Because you want to give personalized experience, and personalized experience require personalized data, and it also brings up a lot of responsibility because you can no longer afford to have PII information
- 3:50
disclosed. You have to be responsible. You have to maintain your brand image. In addition to that, you also need company data. For example, your travel policies. If somebody asks for a refund, whether that person qualifies for the refund of the ticket or not, it will be defined by your travel policies, which airline, so on and so forth.
- 4:10
Right? So a lot goes into when you are building like a virtual agent, for example, a travel agent in that scenario. However, if you're building a conversational, uh, contextual chatbot for improving the employee productivity for common questions, what do you really need?
- 4:25
You need data about your company, absolutely. You need to make sure that that employee has access to that data, and you are not, by mistake, giving additional access more than that is required.
- 4:38
So super important. And then you need other integrations like how this conversational, uh, chatbot will be, you know, presented, whether it's a Slack integration. It can be anything, maybe a custom application, so on and so forth, and the data can reside in different data sources.
- 4:54
So that's another thing. The third use case talks about the marketing. If you're building it for a brand, so obviously those requirements, data requirements, will be different. So now that we have established that our data requirements will be different for different use cases.
- 5:10
So now let's take a deeper dive and double-click on the travel agent. Let's say I have a travel agent. What do I need? First, we all know about the prompting.
- 5:20
I'm not going to repeat that in this session. So you need a prompt. You need a system prompt, plus your user queries, and your user queries will become part of your prompt, right?
- 5:30
So the query will be parsed into. That's again, your data. The instructions is your data. And, uh, sometimes you can have like a prompt, uh, catalog or a template based on different requirements.
- 5:42
Your agent might choose to pick up a specific prompt, you know. Who knows? The way you design or the way, uh, your business requirements are will derive how the design will look like.
- 5:52
Then the second third is context. Now, this context is no longer static. It's coming from your data again, right? And the data can be different data services, data sources.
- 6:03
And then the model that you are using, maybe you're using out-of-the-box model, but maybe you have fine-tuned the model. Again, you need data for training your model representing your company.
- 6:13
So you need data in every step before you can get meaningful output. Now, one thing that is not mentioned over here is the responsible AI part, which I'm not losing sight of, and I'm going to talk about it, but just in a moment.
- 6:28
So now we have established what we need, but you will be like, "Okay, we all know what we need. We have talked enough. How will we solve this problem?"
- 6:37
So that's where we have Amazon Bedrock. So Amazon Bedrock not just provides you with the choice of the models, but it also provides you with some of the amazing features, such as Bedrock Data Automation, using which you can build your custom data pipelines, uh, transform your data, and we'll talk about that.
- 6:56
Or if you want to fine-tune the model, you have model customizations, you can evaluate the models, you have model evaluation. If you want to build a RAG application very quickly, rather than writing the code from scratch, you can use, uh, Bedrock knowledge bases, uh, to do that for you and reduce your time to go to market.
- 7:14
And again, I'm not forgetting responsible AI, and we have Amazon Bedrock Guardrails which can help you with that. So we have to do whatever we are doing, whether we are building an agentic RAG application or we are building a, let's say, summarization application or a classification application or detecting a fraud, or it's our same contextual, uh, chatbot.
- 7:36
We all h- we have to do all of this in a very secure and in a private m- uh, manner with safety guardrails in place, so that's what we are going to do.
- 7:47
So now bear with me and t- let's take another example of a simple RAG application for building a contextual chatbot. Let's simplify a bit so that we can go deeper, right?
- 7:58
So let's say you want to build a chat application. What would you need? Data. Now, with data, we have Amazon Bedrock Data Automation, using which you can build your custom data pipelines just with a single API, right?
- 8:12
That's amazing. Maybe you have data which is, uh, in the form of the videos or text, uh, or images. You need to comprehend those images because they have information.
- 8:25
When I say images, I'm not talking about describing a portrait over here. I'm talking about you might have financial documents which have charts, which have line graphs, and you need to make business sense out of it.
- 8:37
How would you do that, right? So you need some mechanism to do it. Of course, you can do everything on your own, but how we can make it faster, right?
- 8:46
That's where Bedrock Data Automation comes into play. Then next, we have Amazon Bedrock Knowledge Bases. Now, you would say that, "Yeah, I can build, you know, a contextual-based chatbot, a RAG application," and there are so many tools out there.
- 9:00
But let's talk about how Knowledge Bases can help you. One way is, what do you really need when you are building a application? We have talked about data processing, and Bedrock Knowledge Bases have native support for Bedrock Data Automation.
- 9:13
Then we need to define a chunking strategy. Do you want to build a logic for your own chunking strategy, or you want to just leverage some of the things which are out of the box?
- 9:22
Such as if you have complex tables and stuff, you can use, uh, hierarchical chunking, semantic chunking. But then you would say, "Yeah, that's good, Mani, but sometimes we need the custom logic for chunking," so you can actually do that.
- 9:36
And then you can augment the prompt. You would need to vectorize it. You can select a particular foundation, uh, model, uh, embeddings model for, um, creating the embeddings, and we provide you the choice of using which vector store to store your embeddings.
- 9:51
Once all that has been set up, the data ingestion is set up, the incremental updates to the data is set up, because that's what Knowledge Base brings out of the box.
- 9:58
You don't have to write that custom logic, right? Once that is all set up, what do you need? You need APIs to retrieve the information, right? So when, when I say retrieve, there is actually a retrieve API, using which you can retrieve, uh, the similar content.
- 10:14
And then when you are saying, "I'm doing the search," um, yeah, semantic search is good, but how do I optimize on search? So there is hybrid search. So we provide you those options, and you can pass it as parameters rather than implementing it on your own, right?
- 10:29
And now that we have established on how this retrieve API works, you can augment it with a model and then, uh, get the response back. But there are so many different techniques like reranking, post-processing, uh, query decomposition.
- 10:43
So we have, uh, another API which we call RetrieveAndGenerate, which can do everything out of the box, as the name suggests, and also provide you with controls such as, oh, I want reranking, I want, uh, query decomposition.
- 10:56
So all you have to do is pass in the par- right parameters. If you have complex queries, it can handle that. Okay. So now we have the RAG application.
- 11:05
We have processed our data. We have the RAG application. But remember, I talked about the guardrails, right? We need to generate responses responsibly, right? So that's where Amazon Bedrock Guardrails provides you with so many features.
- 11:22
And, uh, with this limited time, I cannot go into each and every feature, but think of it. We are taking example of a contextual-based chatbot, right? I want the users or the information that is being presented to my end user to be, to not have the PII, right?
- 11:41
Or maybe not have any, um, keywords which are not good, because sometimes models can say stuff that they are not supposed to, right? [chuckles]
- 11:52
Uh, in a casual coffee chat, that's okay, but not in a formal setting when you are being recorded, right? [laughs] So that holds the same for our models as well.
- 12:01
So that's where Amazon Bedrock Guardrails comes into play. You can create, like, your own policy. You can define, uh, your own custom guardrails, uh, give examples of what is not to be shared, even ground your responses, reduce hallucinations, and then figure out how your users are interacting when they are triggered, because everything is logged.
- 12:22
And you can identify your user patterns as well with that, right? So now we have this chatbot, right? Bear with me, just visualize it. Uh, you have established, uh, the data processing using BDA, you have established a RAG application using Amazon Bedrock Knowledge Bases, and you have created guardrails, and you have integrated that with, uh, with your
- 12:44
knowledge bases because there is a native integration. So you would think that, "Yeah, I'm all set," right? Of course, you're all set. But how about the coconuts that we talked about, right?
- 12:55
They have to come into play because we mentioned that. So we will talk about coconuts. Uh, but before that, uh, I was supposed to actually transition it when I say your application [chuckles] is ready.
- 13:06
So now let's say your application is ready, and then you have images, and it's able to give you the responses and search, uh, back in the form of the images, right?
- 13:16
You have ingested the documents, so everything is ready. You're getting the responses back. But yes, now let me come back to the coconuts.
- 13:24
So yeah, the dance. I like, I like the dancing coconuts, but let's derive meaning out of it. So far, we have been talking about data processing, how data is important, how...
- 13:36
which Bedrock features can help. And now coconuts? Yes, because that is one thing that I want you to remember when you get out of this room, because coconuts are super important, and you will see how, right? [laughs]
- 13:53
So actually, what happens is a lot of, uh, people have talked with me about, um, the best practices of building RAG applications, and these are not just for the RAG applications, actually.
- 14:07
They... Some of these concept or most of these concepts can be extended to other generative AI applications as well, right? So we'll talk about that. So I want to, you know, have some time for...
- 14:20
to go into this because this is important. So now you have this RAG application, but remember, we started with the data, and we talked about how chunking strategy is so important, right?
- 14:33
And having the right strategy will make the difference in the accuracy of the generated responses. So this is an important step, right? But this is only a step. The second part is optimization.
- 14:48
I talked about reranking, I talked about parsing, hybrid search, but there are so many other techniques like query reformulation, decomposition. You need to understand how to optimize my application.
- 15:01
But tell me one thing. Can you optimize without knowing what is wrong with your app, with your application? Can you do that? Any raise of hands? No. Right. I can see people nodding heads, so I'll take no.
- 15:13
So we cannot because we need to evaluate our application. So we will come to evaluation because before evaluation is we have to see what's going on. And also, when we are talking about optimization, one is the optimizing the accuracy.
- 15:30
The second is optimizing the cost, right? And also the latency. How do we do that? Because these are the three pillars of any generative application: performance, latency, cost. Right.
- 15:43
So how do we do that? One way of doing that is caching your results. So you can use semantic cache, which is caching... Why, why semantic? Because in the generative AI world, we don't have always the exact deterministic questions, neither the deterministic...
- 16:02
deterministic responses up to a certain extent, but not in the exact wordings. So if we have to do it, what we are looking in our cache is not the exact question, but if similar questions have been asked, I don't need to go and invoke my foundation model.
- 16:17
That will charge me money, right? And that's number one. Second, that will also re- because it will take time for the model to synthesize, it will add latency, right?
- 16:29
So we need to take care of both those things. That's why semantic caching. Similar questions quickly retrieve if th- if, if this has been asked before at all. So now we have got three things.
- 16:41
The second is observability. We need to see what's going on, right? That's where you need to log everything, user queries, retrieval hits, model responses, 'cause that's your only way how you can monitor your, um, application to improve it, to, you know, refine it.
- 16:58
So that's what we are doing with observability. So earlier when my, um, you know, customers used to ask me about, um, observability or best practices, I would say, "You know, observability is the crit- critical component."
- 17:13
But now I say, "Please don't go into production or even do a pilot without observability." How will you troubleshoot? How will you improve? How will you figure out what's going on in the application if somebody complains, right?
- 17:27
So you need observability. But now when we are talking about observability, now we know we have this additional data. What are we going to do with it? We have to do something with it.
- 17:40
We need to evaluate it, right? We need to figure out based on my application, like for RAG application, we have context relevance, because if my context is not relevant, my search results are not good, doesn't make sense for me to put them in front of the model and pay, uh, the money and, you know, because everything has
- 17:59
a cost associated with that, just to get a bad response. Why would I do that? I would first evaluate my search results. If my search results are good, then only I'm going to augment my, my prompt and give it to the model, right?
- 18:13
So those are the common things that we sometimes, because we are, you know, we are quickly doing everything, we forget about it. So we have to rethink and evaluate our thinking as well as our application. [chuckles]
- 18:26
So that's super important. But if you have a summarization use case, you can have metrics related to summarization, right? So that's why I was saying, like, these coconut is actually...
- 18:35
coconuts are good. They're good for other applications as well.
- 18:39
And then never go blind. Uh, so we have talked about that. But what are we going to do? Let's say in our evaluation something happened, we figured it out.
- 18:48
We need to update. We need to update what? Sometimes our data, because if our data is stale, our answers are stale. Sometimes we have to update our strategy. So again, coming back to optimization, right?
- 19:03
And once you have done the updates, please don't, uh, feel that you are done and you have updated, it's all good. You need to test again. So one is you are evaluating your application, you have updated it, but then you also need to have a test suite, because whenever you are doing this update, these evaluations have to
- 19:23
be automatic. And if you have a defined path on how you are creating this test suite, the chances of you going wrong in your production application reduces, and the chances that you have a high-quality application out there in production increases by manyfold.
- 19:42
So test. And once you have tested and tested and tested your application, you're ready to crack your coconut, and that's how you will scale. So not just the coconut, we need coconuts.
- 19:55
And once you have cracked your coconuts, you can decorate it, and you can have your application into production and bear the fruits of success. [laughs] [outro music]