# Data is Your Differentiator: Building Secure and Tailored AI Systems

Mani Khanuja

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/ROfHHJmumcc-data-is-your-differentiator-building-secure) · [Video](https://www.youtube.com/watch?v=ROfHHJmumcc)

## Data Is Your Differentiator: Building Secure and Tailored AI Systems

A useful generative AI application needs more than a model: it needs application-specific data, controlled retrieval, observable behavior, and a repeatable path from evaluation to updates.

Familiarity with language-model prompts and basic API calls will help; retrieval-augmented generation and its main components are explained as they appear.

### Dancing coconuts and a deeper foundation

[0:00](https://www.youtube.com/watch?v=ROfHHJmumcc&t=0s)

Dancing coconuts open Mani Khanuja’s presentation, courtesy of the [**Amazon Nova** models](https://aws.amazon.com/blogs/aws/introducing-amazon-nova-frontier-intelligence-and-industry-leading-price-performance/). They are a playful example of what generative AI can produce—and a memory aid that will return when the application is ready to scale. First comes the less visible work that makes an AI application useful to a business.

The more ambitious the application, the deeper its foundation must be. That foundation is **data**: the material that represents the company, its brand, and its organization. The iceberg slide makes the supporting work visible, placing mindset, people, process, and technology beneath the business value above the surface.

![Iceberg diagram places generative AI business value above water and mindset, people, process, and technology below the surface.](https://ai.engineer/api/talk-essay-media/3ef08cc9979a5c54eb69e439f1099fe4e1e09c68f1cd0bc6215c32523c60a611)

Get your data foundation right: mindset, people, process, and technology.

Data preparation is familiar from earlier machine learning systems, but generative AI changes the requirements according to the application and its business purpose. Transformation, parsing, and loading still matter. So do the interactions between data, technology, and people. If useful information remains trapped in organizational silos, an application that could use it cannot deliver the intended experience. Getting the foundation right therefore includes how information becomes available, not just how it is formatted.

### The application determines the data requirements

[2:54](https://www.youtube.com/watch?v=ROfHHJmumcc&t=174s)

Consider a travel agent. It needs a customer profile to personalize the conversation; learning the relevant details too late is like remembering someone’s name after they have left the room. That personalization also creates a responsibility to prevent disclosure of personally identifiable information, or **PII**. The agent then needs company information: whether a customer qualifies for a ticket refund depends on travel policies and the relevant airline’s rules. A fluent answer without those inputs cannot reliably resolve the request.

An employee chatbot has a different boundary. It needs company knowledge, but each employee must receive only information they are authorized to access. It may appear in Slack or a custom application, with information drawn from several sources. A marketing application adds another purpose: producing content that fits the brand. Khanuja introduces that third case without developing a separate implementation.

| Application | Required data | Governing concern |
|---|---|---|
| Travel agent | Customer profiles and travel policies | Personalization without PII disclosure |
| Employee chatbot | Company knowledge from multiple sources | Preserve employee access permissions |
| Marketing application | Brand-relevant information | Keep generated content on brand |

**There is no single data requirement for generative AI.** The intended interaction determines which information the application needs and which boundaries it must preserve.

![Slide with three panels titled A more personal virtual agent, More relevant conversational search, and Content that’s on brand, alongside phone, chat, and tablet images.](https://ai.engineer/api/talk-essay-media/1fe1b708c3aa0b0aac8f4ad0023c509b1d4b665b9d16e1e72b0b0de5b2e1e139)

Three generative AI applications and their data needs.

### Data enters through the prompt, context, and model

[5:10](https://www.youtube.com/watch?v=ROfHHJmumcc&t=310s)

Following the travel agent into a model interaction reveals several distinct uses of data. First, the system prompt supplies instructions and the user query supplies the immediate request. Both are data. A prompt catalog or set of templates can provide different instructions for different business requirements, with the agent selecting the appropriate template.

Next comes **context**, which is no longer necessarily static text bundled with the application. It can arrive from changing data services and sources. Finally, there is the model itself: an out-of-the-box model may be sufficient, while a fine-tuned model requires training data that represents the company. Data is involved before meaningful output appears, whether it supplies instructions, current knowledge, or model customization. Responsible generation adds another requirement, addressed later through guardrails.

### From source material to a retrieval corpus

[6:37](https://www.youtube.com/watch?v=ROfHHJmumcc&t=397s)

Amazon Bedrock supplies several parts of this application architecture. Alongside model choice, customization, and evaluation, [Bedrock Data Automation](https://aws.amazon.com/blogs/aws/get-insights-from-multimodal-content-with-amazon-bedrock-data-automation-now-generally-available/) handles data processing. [Bedrock Knowledge Bases](https://aws.amazon.com/blogs/machine-learning/build-a-contextual-chatbot-application-using-knowledge-bases-for-amazon-bedrock/) provides managed retrieval-augmented generation, or **RAG**, and [Bedrock Guardrails](https://aws.amazon.com/blogs/machine-learning/introducing-guardrails-in-knowledge-bases-for-amazon-bedrock/) adds configurable safeguards. The security and privacy requirements apply across agentic RAG, summarization, classification, fraud detection, and contextual chatbots. To examine the pieces more closely, Khanuja narrows the example to a contextual chatbot.

Its source material may include text, videos, and images. Understanding an image here means more than describing a portrait: a financial document may contain charts and line graphs whose business meaning must survive processing. Khanuja describes Data Automation as offering a single API for this processing. In the launch-era workflow, that means a unified asynchronous invocation, `InvokeDataAutomationAsync`, with configuration and S3 result handling around it—not an entire pipeline without setup.

Knowledge Bases natively integrates Data Automation, then gives the application choices about how to prepare the resulting material for retrieval:

1. **Choose a chunking strategy.** Use [hierarchical or semantic chunking](https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-knowledge-bases-now-supports-advanced-parsing-chunking-and-query-reformulation-giving-greater-control-of-accuracy-in-rag-based-applications/), or supply custom chunking logic when the documents require it. Complex tables are one reason the choice deserves attention.
2. **Choose an embedding model.** Convert the chunks into vector representations used for similarity search.
3. **Choose a vector store.** Store those embeddings for retrieval.
4. **Maintain ingestion and updates.** Knowledge Bases manages ingestion and incremental processing. The application still needs an appropriate synchronization trigger; the companion chatbot implementation uses an S3-triggered Lambda to call `StartIngestionJob`.

The result is a maintained retrieval corpus, rather than a one-time upload that can be assumed to remain current.

### Retrieve first, or retrieve and generate

[9:58](https://www.youtube.com/watch?v=ROfHHJmumcc&t=598s)

Once ingestion is configured, the application needs a way to retrieve relevant information. The `Retrieve` API returns similar content, leaving the application to augment a prompt and invoke a model. Search is also configurable: [hybrid search](https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-knowledge-bases-now-supports-hybrid-search/) combines semantic similarity with text matching. In the current vector-search configuration, `HYBRID` requires a supported store and a filterable text field; it is not available for every storage configuration.

For a refund question, a Python retrieval call can expose the policy passages before any answer is generated. This function accepts the configured knowledge base ID and lets the caller choose the search mode:

```python
import boto3


def retrieve_policy(knowledge_base_id, question, search_type="SEMANTIC"):
    client = boto3.client("bedrock-agent-runtime")
    response = client.retrieve(
        knowledgeBaseId=knowledge_base_id,
        retrievalQuery={"text": question},
        retrievalConfiguration={
            "vectorSearchConfiguration": {
                "overrideSearchType": search_type,
            }
        },
    )
    return response["retrievalResults"]
```

The returned retrieval results are evidence for the next step, not a completed refund decision. The application can inspect that evidence before adding it to the model’s prompt.

Retrieval quality can be improved through reranking, post-processing, and query decomposition. For an integrated path, `RetrieveAndGenerate` retrieves context and generates the response, with parameters for controls such as reranking and query decomposition. Decomposition is useful when a complex question needs to be broken into smaller retrieval requests.

| API | Returns | Application responsibility |
|---|---|---|
| `Retrieve` | Retrieved content | Inspect context, augment the prompt, invoke the model |
| `RetrieveAndGenerate` | A generated response grounded in retrieval | Configure the managed retrieval and generation path |

The choice is about where to retain control. A retrieval-only call leaves an explicit boundary for application logic; the combined API packages more of the workflow behind [query and generation controls](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html).

![Diagram connects a user query to Amazon Bedrock and knowledge bases, then an augmented prompt, a list of models, and an answer.](https://ai.engineer/api/talk-essay-media/fe370ce6b27aaa46114f545c571249ca5bd040bc0d09f5fa1e460716bc1bab78)

Amazon Bedrock Knowledge Bases retrieval and generation workflow.

### Add safeguards without losing the access boundary

[11:05](https://www.youtube.com/watch?v=ROfHHJmumcc&t=665s)

A working RAG path still needs responsible response generation. For the contextual chatbot, Khanuja highlights preventing PII disclosure and filtering unwanted words. Bedrock Guardrails lets teams define policies, specify examples of information that should not be shared, and apply grounding checks to help reduce hallucinations. Guardrail interventions can also reveal patterns in how users interact with the application, provided the relevant telemetry is collected.

Data Automation, Knowledge Bases, and Guardrails now form an assembled chatbot, with native integration between the retrieval-and-generation path and its safeguards. One consequential boundary remains: [current AWS documentation](https://docs.aws.amazon.com/en_en/bedrock/latest/userguide/kb-test-retrieve.html) states that guardrails cover input and generated responses, not retrieved references. **Guardrails do not replace access control or source-data protection.** Sensitive retrieved material needs its own controls, especially if the interface exposes references directly.

With documents ingested, the application can search and return material in image form as well as text. That brings the presentation back to the dancing coconuts. Getting responses is the starting point for the operational practices that follow, many of which apply beyond RAG. The first is chunking: the way information is divided affects generated-answer accuracy, but it is only one step in improving the application.

### Optimize accuracy, cost, and latency

[14:48](https://www.youtube.com/watch?v=ROfHHJmumcc&t=888s)

Optimization offers many possible interventions: better parsing, reranking, hybrid search, query reformulation, and decomposition. Choosing among them requires knowing what is wrong. Evaluation identifies the weakness; optimization addresses it. The objective also extends beyond answer accuracy to the three dimensions Khanuja emphasizes: performance, latency, and cost.

**Semantic caching** targets repeated work. Users do not always express the same question in identical words, so an exact-string cache can miss a reusable answer. A semantic cache looks for a sufficiently similar question that has already been answered. When a cached result can be reused, the application avoids another foundation-model invocation and the time needed to synthesize a fresh response.

### Observe the application, then evaluate the right stage

[16:41](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1001s)

To diagnose failures, capture the chain from **user query** to **retrieval hits** to **model response**. Those records let a team investigate a complaint and determine whether the problem began with the request, the retrieved evidence, or the generated answer. Khanuja’s advice is explicit: “Please don't go into production or even do a pilot without observability.”

Telemetry requires deliberate configuration. [Bedrock model invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) is disabled by default and requires a destination such as CloudWatch Logs or S3. Enabling it does not establish complete coverage of retrieval events or guardrail interventions; those need to be accounted for in the application’s instrumentation. Full request and response logs can themselves contain sensitive data, so the privacy requirements also apply to this new dataset. Once collected, it becomes material for evaluation.

For RAG, start with **context relevance**. If search returns irrelevant material, placing it in the prompt spends generation budget on a poor input. Evaluate the search results first; only then use suitable evidence to augment the prompt and ask the model for an answer. This separates a retrieval problem from a generation problem instead of treating every bad answer as a model failure.

Evaluation must fit the task. A summarization application needs summarization-specific metrics, rather than automatically inheriting the criteria for retrieval. Khanuja does not prescribe particular summarization metrics here; the broader practice is to choose evaluation according to the behavior the application is meant to deliver.

### Update, test again, and scale

[18:39](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1119s)

Never go blind: evaluation findings must lead to action. Sometimes the data needs refreshing, because stale data produces stale answers. Sometimes the retrieval or optimization strategy needs to change. The evaluation result should determine which part of the application gets updated.

An update is not the end of the loop. A defined **test suite** should automatically rerun evaluations after changes. That makes testing a repeatable part of maintenance, rather than a manual exercise performed only before the first release. Khanuja presents this as a way to reduce production failures and improve application quality, without supplying a measured improvement.

Repeated testing leads to the final coconut callback: crack the coconut, then scale to coconuts. The complete **COCONUTS** memory aid connects chunking strategy, optimization, caching, observability, never going blind, updating, testing again, and scaling. Production is the point at which this working and tested application can bear fruit—not the point at which the first generated answer appears.

![Table lists chunking strategy, optimize, cache results, observability, never go blind, update, test again, and scale; the speaker inset partially overlaps the bottom-left row.](https://ai.engineer/api/talk-essay-media/e27f5995830b3de9fec18b1323b8902ade314cb10b87b15a0dd42c78a88a2665)

COCONUTS best practices for RAG applications.

## Resources

- [Introducing Amazon Nova](https://aws.amazon.com/blogs/aws/introducing-amazon-nova-frontier-intelligence-and-industry-leading-price-performance/) — Introduction to the Nova model family, including image and video generation with example workflows.
- [Bedrock Data Automation: multimodal processing walkthrough](https://aws.amazon.com/blogs/aws/get-insights-from-multimodal-content-with-amazon-bedrock-data-automation-now-generally-available/) — A launch-era walkthrough of multimodal extraction, blueprints, asynchronous processing and Knowledge Bases integration.
- [Advanced parsing, chunking and query reformulation](https://aws.amazon.com/blogs/machine-learning/amazon-bedrock-knowledge-bases-now-supports-advanced-parsing-chunking-and-query-reformulation-giving-greater-control-of-accuracy-in-rag-based-applications/) — Mani Khanuja's coauthored guide to semantic and hierarchical chunking, custom processing and query decomposition.
- [Build a contextual chatbot with Knowledge Bases](https://aws.amazon.com/blogs/machine-learning/build-a-contextual-chatbot-application-using-knowledge-bases-for-amazon-bedrock/) — A coauthored historical tutorial covering ingestion, synchronization, retrieval and a chatbot interface using 2024 model choices.
- [Current Knowledge Bases query controls](https://docs.aws.amazon.com/bedrock/latest/userguide/kb-test-config.html) — Configuration guidance for hybrid search, metadata filtering, reranking, guardrails and query decomposition.
- [Integrating Guardrails with Knowledge Bases](https://aws.amazon.com/blogs/machine-learning/introducing-guardrails-in-knowledge-bases-for-amazon-bedrock/) — A walkthrough of adding guardrail policies to a Knowledge Bases retrieval-and-generation workflow.
- [Configure Bedrock invocation logging](https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html) — Instructions for enabling supported model invocation logs and configuring CloudWatch Logs or S3 destinations.

## Slides

### Common Applications of Generative AI

A travel agent uses customer and company data, conversational search draws on enterprise sources, and brand content relies on inventory and past marketing. Each application has different data needs.

[2:53](https://www.youtube.com/watch?v=ROfHHJmumcc&t=173s) [Examples of a virtual agent, conversational search, and on-brand content, with their data sources.](https://assets.ai.engineer/sites/aiecode2025/sha256/43f7de87e8a137f4f171785f4a9b8935badbcbfaf1fa5a2111eaf676246c5e69.webp)

Common Applications of Generative AI

A more personal virtual agent
The objective
An online travel agency wants to generate personalized travel itineraries

The data
Customer profile
Past trips
Web history
Travel preferences

Company data
Flight and hotel inventory
Promotions
Similar traveler details

More relevant conversational search
The objective
Provide answers to customers and employees based on enterprise data

The data
Slack
Internal guides
SharePoint

Content that’s on brand
The objective
Create marketing content and ads that reflect my company’s brand voice, look, and feel and keep up with changing inventory

The data
Real-time inventory
Brand guidebook
Previous adds and content

[illegible]25, Amazon Web Services, Inc. or its affiliates. All rights reserv[illegible]
6

Interface text:
TRAVEL
Morocco
[illegible]

### Example of a Generative AI Application

A retention question is combined with the call transcript, customer data and vectorized data, then sent to a model to suggest actions, including a one-time discount of 25% for three months.

[5:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=313s) [A customer-retention example combines a prompt, context, and a model to produce recommendations.](https://assets.ai.engineer/sites/aiecode2025/sha256/8368685f0c7131d01bcc8caea02d7f3b6b8f4e326ac20749c129c89be44821a9.webp)

Example of a Generative AI Application

PROMPT + CONTEXT + MODEL = OUTPUT

PROMPT
“This customer is considering closing their account.

What steps could we take to retain them?”

Agent question

CONTEXT
Current Call Transcript
Customer data
Vectorized Data

Data services provide relevant customer data and conversation state

MODEL
Foundation Models
Fine Tuned Models
Scratch Build Models

Query LLM and retrieve output
*For RAG, query vectors

OUTPUT
1. Reiterate value of their Insurance Policy
2. Offer a one-time discount of 25% for 3 months
3. Apologize and reassure them of your commitment
4. Offer escalation to a supervisor to diffuse concerns

Present recommendation to agent

[illegible]25, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.

### Amazon Bedrock

Amazon Bedrock combines model access through one API with customization, RAG and agents. The speaker also points to Bedrock Guardrails for responsible AI.

[6:37](https://www.youtube.com/watch?v=ROfHHJmumcc&t=397s) [Amazon Bedrock overview listing model choice, customization, RAG, agents, and security.](https://assets.ai.engineer/sites/aiecode2025/sha256/dbb6dde6c8396da4495590add09499c1d97c8b5364a6ea866febdc0740877883.webp)

Amazon Bedrock
The easiest way to build and scale generative AI applications

Choice of leading FMs through a single API
Model customization
Retrieval Augmented Generation (RAG)
Agents that execute multistep tasks
Security, privacy, and safety

aws

[illegible] Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.

### Document Processing Using Amazon Bedrock Data Automation

Amazon Bedrock Data Automation extracts structured data from multimodal content and customizes outputs through a single API. The speaker gives financial documents with charts and line graphs as an example.

[7:58](https://www.youtube.com/watch?v=ROfHHJmumcc&t=478s) [Data Automation extracts structured data from multimodal content and supports customized outputs and managed workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/205c4b653198bdd65105fad859da364fe5164f25ffcce628fcf736839a766508.webp)

Document processing using Amazon Bedrock Data Automation

Amazon Bedrock Data Automation

Extract, transform and generate structured data from multi-modal content

Generate customized output based on your application’s requirements and business rules

Streamline application workflows with a fully managed, single API experience

[illegible]25, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.

### Amazon Bedrock Knowledge Bases

Bedrock retrieves context from a knowledge base, adds it to the user’s prompt, and sends the augmented prompt to a chosen model. The slide also highlights managed ingestion, session context, and citations.

[8:49](https://www.youtube.com/watch?v=ROfHHJmumcc&t=529s) [Diagram of a RAG workflow using Amazon Bedrock Knowledge Bases, with model choices and feature summaries.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ab5c37696dea379229590ba2453654b2445ef2106ef770546e55661c421a11f.webp)

Amazon Bedrock Knowledge Bases
Native support for RAG

USER QUERY
AMAZON BEDROCK
Amazon Bedrock knowledge bases
AUGMENTED PROMPT
MODEL
ANSWER

Workflow markers: 1, 2, 3, 4, 5, 6

Models:
Meta—Llama
Anthropic—Claude
Amazon—Nova
Cohere—Command
Mistral AI - Mistral
AI21 Labs - Jurassic

Securely connect FMs to data sources for RAG to deliver more relevant responses
Fully managed RAG workflow including ingestion, retrieval, and augmentation
Built-in session context management for multiturn conversations
Automatic citations with retrievals to improve transparency

[illegible]25, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.

### Amazon Bedrock Guardrails

Guardrails evaluate prompts and model responses using filters for harmful content, denied topics, sensitive information and words. Grounding and reasoning checks address hallucinations and factual claims.

[11:10](https://www.youtube.com/watch?v=ROfHHJmumcc&t=670s) [Amazon Bedrock Guardrails features include content filtering, factual checks, denied topics, sensitive information filtering, word filters, and grounding checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/c785d82a44985265ee15d86e81ecadd9ee2d0877c219be66eaf62bc976ca01cd.webp)

Amazon Bedrock Guardrails

Evaluate prompts and model responses for agents, knowledge bases, FMs in Amazon Bedrock, and self-managed or third-party FMs

• Configure thresholds to filter undesirable and potentially harmful text and image content, jailbreaks, and prompt attacks
• Identify, correct, and explain factual claims in responses using Automated Reasoning checks
• Define and disallow denied topics with short natural language descriptions
• Remove personally identifiable information (PII) and sensitive information in generative AI applications
• Define a set of words to detect and block in user inputs and model responses
• Filter hallucinations by detecting groundedness and relevance of model responses based on context

[illegible]25, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and Trademark.

### Best practices for RAG applications: COCONUTS

COCONUTS connects RAG practices into a cycle: chunk and optimize data, cache similar queries, observe and evaluate results, then update and retest before scaling.

[13:48](https://www.youtube.com/watch?v=ROfHHJmumcc&t=828s) [Table of RAG practices covering chunking, optimization, caching, observability, evaluation, updates, testing, and scaling.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b29196521c042c98529656426bbaee38b75db2fab73d5b391f019974386161b.webp)

Best Practices for RAG Application: COCONUTS

Chunking strategy: Break your documents into meaningful, query aligned chunks.
Optimize: Look for strategies such as parsing, hybrid search, re-ranking and so on.
Cache results: For reoccurring queries semantic caching can help with cost and performance optimization.
Observability: Log everything, user queries, retrieval hits, model responses – monitor to improve.
Never go blind: Evaluate your RAG application using relevant metrics such as context relevance, answer relevance, correctness and others.
Update: incremental updates to vectors stores, stale data = stale answers, recursive retrievals, query reformulation, re-ranking and so on.
Test again: Test, Test and Test.
Scale: Increase the load slowly, make it repetitive, crack your coconut, while increasing load..!

### Thank you

Mani Khanuja closes with a decorated coconut, echoing her metaphor for taking an application into production.

[19:56](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1196s) [AWS closing slide with a thank-you message and a photograph of coconuts and purple flowers.](https://assets.ai.engineer/sites/aiecode2025/sha256/532503fbc2a762fcd79ac6c40123fea47a937ae4dfb55af62df96ed06de5f826.webp)

aws

Thank you!

Mani Khanuja
LinkedIn: manikhanuja

[illegible] Web Services, Inc. or its affiliates. All rights reserved. Amazon Confidential and [illegible]

## Full transcript

[0:00](https://www.youtube.com/watch?v=ROfHHJmumcc&t=0s) [on-hold electronic music] So today we are going to talk about data as your differentiator or maybe about coconuts.

[0:22](https://www.youtube.com/watch?v=ROfHHJmumcc&t=22s) Let's see. Right. So that's why I was like, I'm confident that I'll, uh, wake you up a little bit. And by the way, these coconuts dance also. [laughs] Courtesy of Amazon Nova models.

[0:33](https://www.youtube.com/watch?v=ROfHHJmumcc&t=33s) Okay. Enough about coconuts, and, um, we'll connect the dots and revisit our coconuts because they're dancing, and I love dancing coconuts, right? Remind us of the beach and the, you know, vacation and everything.

[0:48](https://www.youtube.com/watch?v=ROfHHJmumcc&t=48s) But let's come back from vacation now. Serious work to do. Okay. So we have been talking about generative AI, and, um, we all know that generative AI adds a lot of business value if done right.

[1:04](https://www.youtube.com/watch?v=ROfHHJmumcc&t=64s) Right? That's a, that's a big if. And, um, it can take your business to the highest level, as you can see in this image, right? But however, when you are building something that big that can change how your users interact with your applications, then think about the foundation, right?

[1:24](https://www.youtube.com/watch?v=ROfHHJmumcc&t=84s) The foundation has to be even deeper. And what is that foundation? That's your data. Because your data is representing your company, your brand, your organization. So you have to get the data together.

[1:40](https://www.youtube.com/watch?v=ROfHHJmumcc&t=100s) And we all know this since the very beginning of machine learning world, and you would say, "Mani, what is new that you are, that you are, that you are, you know, presenting over here?"

[1:50](https://www.youtube.com/watch?v=ROfHHJmumcc&t=110s) So the new thing is that the data requirements for building generative AI applications are different. Generative AI is different, and so there are special treatment that you need to give to the data based on your, uh, application and the business requirements.

[2:07](https://www.youtube.com/watch?v=ROfHHJmumcc&t=127s) So that is exactly what we are going to talk about today. And data is not just about, okay, I have to transform my data, I have to load my data, I have to parse it, I have to load it, you know, all that stuff.

[2:19](https://www.youtube.com/watch?v=ROfHHJmumcc&t=139s) That is still there and very important. But the most important thing is how is your data interacting with the technology? How is it interacting with your people? How... Do you still have data silos?

[2:32](https://www.youtube.com/watch?v=ROfHHJmumcc&t=152s) If yes, then what about your applications? Because these models can take up lots and lots of data, right? So how are we doing with all those interactions? Because no longer we can afford to be in a silo.

[2:44](https://www.youtube.com/watch?v=ROfHHJmumcc&t=164s) And that's what... That's the exact reason why we have to get our foundation right, and our foundation is data. So let's see.

[2:54](https://www.youtube.com/watch?v=ROfHHJmumcc&t=174s) Now, there are certain common applications. Let's take an example. And there are so many ap- uh, applications, the possibilities are infinite, right? However, let's, you know, break it down and see with these three applications how your, uh, data requirements may differ.

[3:11](https://www.youtube.com/watch?v=ROfHHJmumcc&t=191s) The first example is the travel agent. So if you're building a travel agent, what kind of data would you need? First, with that travel agent, it's going to interact with your end users, so it needs to know about your customer profile, right?

[3:27](https://www.youtube.com/watch?v=ROfHHJmumcc&t=207s) Because if it doesn't know about your customer profile, it's literally like, um, remembering somebody's name when they have left the room. How weird that would be, right? Because you want to give personalized experience, and personalized experience require personalized data, and it also brings up a lot of responsibility because you can no longer afford to have PII information

[3:50](https://www.youtube.com/watch?v=ROfHHJmumcc&t=230s) disclosed. You have to be responsible. You have to maintain your brand image. In addition to that, you also need company data. For example, your travel policies. If somebody asks for a refund, whether that person qualifies for the refund of the ticket or not, it will be defined by your travel policies, which airline, so on and so forth.

[4:10](https://www.youtube.com/watch?v=ROfHHJmumcc&t=250s) Right? So a lot goes into when you are building like a virtual agent, for example, a travel agent in that scenario. However, if you're building a conversational, uh, contextual chatbot for improving the employee productivity for common questions, what do you really need?

[4:25](https://www.youtube.com/watch?v=ROfHHJmumcc&t=265s) You need data about your company, absolutely. You need to make sure that that employee has access to that data, and you are not, by mistake, giving additional access more than that is required.

[4:38](https://www.youtube.com/watch?v=ROfHHJmumcc&t=278s) So super important. And then you need other integrations like how this conversational, uh, chatbot will be, you know, presented, whether it's a Slack integration. It can be anything, maybe a custom application, so on and so forth, and the data can reside in different data sources.

[4:54](https://www.youtube.com/watch?v=ROfHHJmumcc&t=294s) So that's another thing. The third use case talks about the marketing. If you're building it for a brand, so obviously those requirements, data requirements, will be different. So now that we have established that our data requirements will be different for different use cases.

[5:10](https://www.youtube.com/watch?v=ROfHHJmumcc&t=310s) So now let's take a deeper dive and double-click on the travel agent. Let's say I have a travel agent. What do I need? First, we all know about the prompting.

[5:20](https://www.youtube.com/watch?v=ROfHHJmumcc&t=320s) I'm not going to repeat that in this session. So you need a prompt. You need a system prompt, plus your user queries, and your user queries will become part of your prompt, right?

[5:30](https://www.youtube.com/watch?v=ROfHHJmumcc&t=330s) So the query will be passed into. That's again, your data. The instructions is your data. And, uh, sometimes you can have like a prompt, uh, catalog or a template based on different requirements.

[5:42](https://www.youtube.com/watch?v=ROfHHJmumcc&t=342s) Your agent might choose to pick up a specific prompt, you know. Who knows? The way you design or the way, uh, your business requirements are will derive how the design will look like.

[5:52](https://www.youtube.com/watch?v=ROfHHJmumcc&t=352s) Then the second third is context. Now, this context is no longer static. It's coming from your data again, right? And the data can be different data services, data sources.

[6:03](https://www.youtube.com/watch?v=ROfHHJmumcc&t=363s) And then the model that you are using, maybe you're using out-of-the-box model, but maybe you have fine-tuned the model. Again, you need data for training your model representing your company.

[6:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=373s) So you need data in every step before you can get meaningful output. Now, one thing that is not mentioned over here is the responsible AI part, which I'm not losing sight of, and I'm going to talk about it, but just in a moment.

[6:28](https://www.youtube.com/watch?v=ROfHHJmumcc&t=388s) So now we have established what we need, but you will be like, "Okay, we all know what we need. We have talked enough. How will we solve this problem?"

[6:37](https://www.youtube.com/watch?v=ROfHHJmumcc&t=397s) So that's where we have Amazon Bedrock. So Amazon Bedrock not just provides you with the choice of the models, but it also provides you with some of the amazing features, such as Bedrock Data Automation, using which you can build your custom data pipelines, uh, transform your data, and we'll talk about that.

[6:56](https://www.youtube.com/watch?v=ROfHHJmumcc&t=416s) Or if you want to fine-tune the model, you have model customizations, you can evaluate the models, you have model evaluation. If you want to build a RAG application very quickly, rather than writing the code from scratch, you can use, uh, Bedrock knowledge bases, uh, to do that for you and reduce your time to go to market.

[7:14](https://www.youtube.com/watch?v=ROfHHJmumcc&t=434s) And again, I'm not forgetting responsible AI, and we have Amazon Bedrock Guardrails which can help you with that. So we have to do whatever we are doing, whether we are building an agentic RAG application or we are building a, let's say, summarization application or a classification application or detecting a fraud, or it's our same contextual, uh, chatbot.

[7:36](https://www.youtube.com/watch?v=ROfHHJmumcc&t=456s) We all h- we have to do all of this in a very secure and in a private m- uh, manner with safety guardrails in place, so that's what we are going to do.

[7:47](https://www.youtube.com/watch?v=ROfHHJmumcc&t=467s) So now bear with me and t- let's take another example of a simple RAG application for building a contextual chatbot. Let's simplify a bit so that we can go deeper, right?

[7:58](https://www.youtube.com/watch?v=ROfHHJmumcc&t=478s) So let's say you want to build a chat application. What would you need? Data. Now, with data, we have Amazon Bedrock Data Automation, using which you can build your custom data pipelines just with a single API, right?

[8:12](https://www.youtube.com/watch?v=ROfHHJmumcc&t=492s) That's amazing. Maybe you have data which is, uh, in the form of the videos or text, uh, or images. You need to comprehend those images because they have information.

[8:25](https://www.youtube.com/watch?v=ROfHHJmumcc&t=505s) When I say images, I'm not talking about describing a portrait over here. I'm talking about you might have financial documents which have charts, which have line graphs, and you need to make business sense out of it.

[8:37](https://www.youtube.com/watch?v=ROfHHJmumcc&t=517s) How would you do that, right? So you need some mechanism to do it. Of course, you can do everything on your own, but how we can make it faster, right?

[8:46](https://www.youtube.com/watch?v=ROfHHJmumcc&t=526s) That's where Bedrock Data Automation comes into play. Then next, we have Amazon Bedrock Knowledge Bases. Now, you would say that, "Yeah, I can build, you know, a contextual-based chatbot, a RAG application," and there are so many tools out there.

[9:00](https://www.youtube.com/watch?v=ROfHHJmumcc&t=540s) But let's talk about how Knowledge Bases can help you. One way is, what do you really need when you are building a application? We have talked about data processing, and Bedrock Knowledge Bases have native support for Bedrock Data Automation.

[9:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=553s) Then we need to define a chunking strategy. Do you want to build a logic for your own chunking strategy, or you want to just leverage some of the things which are out of the box?

[9:22](https://www.youtube.com/watch?v=ROfHHJmumcc&t=562s) Such as if you have complex tables and stuff, you can use, uh, hierarchical chunking, semantic chunking. But then you would say, "Yeah, that's good, Mani, but sometimes we need the custom logic for chunking," so you can actually do that.

[9:36](https://www.youtube.com/watch?v=ROfHHJmumcc&t=576s) And then you can augment the prompt. You would need to vectorize it. You can select a particular foundation, uh, model, uh, embeddings model for, um, creating the embeddings, and we provide you the choice of using which vector store to store your embeddings.

[9:51](https://www.youtube.com/watch?v=ROfHHJmumcc&t=591s) Once all that has been set up, the data ingestion is set up, the incremental updates to the data is set up, because that's what Knowledge Base brings out of the box.

[9:58](https://www.youtube.com/watch?v=ROfHHJmumcc&t=598s) You don't have to write that custom logic, right? Once that is all set up, what do you need? You need APIs to retrieve the information, right? So when, when I say retrieve, there is actually a Retrieve API, using which you can retrieve, uh, the similar content.

[10:14](https://www.youtube.com/watch?v=ROfHHJmumcc&t=614s) And then when you are saying, "I'm doing the search," um, yeah, semantic search is good, but how do I optimize on search? So there is hybrid search. So we provide you those options, and you can pass it as parameters rather than implementing it on your own, right?

[10:29](https://www.youtube.com/watch?v=ROfHHJmumcc&t=629s) And now that we have established on how this Retrieve API works, you can augment it with a model and then, uh, get the response back. But there are so many different techniques like reranking, post-processing, uh, query decomposition.

[10:43](https://www.youtube.com/watch?v=ROfHHJmumcc&t=643s) So we have, uh, another API which we call RetrieveAndGenerate, which can do everything out of the box, as the name suggests, and also provide you with controls such as, oh, I want reranking, I want, uh, query decomposition.

[10:56](https://www.youtube.com/watch?v=ROfHHJmumcc&t=656s) So all you have to do is pass in the par- right parameters. If you have complex queries, it can handle that. Okay. So now we have the RAG application.

[11:05](https://www.youtube.com/watch?v=ROfHHJmumcc&t=665s) We have processed our data. We have the RAG application. But remember, I talked about the guardrails, right? We need to generate responses responsibly, right? So that's where Amazon Bedrock Guardrails provides you with so many features.

[11:22](https://www.youtube.com/watch?v=ROfHHJmumcc&t=682s) And, uh, with this limited time, I cannot go into each and every feature, but think of it. We are taking example of a contextual-based chatbot, right? I want the users or the information that is being presented to my end user to be, to not have the PII, right?

[11:41](https://www.youtube.com/watch?v=ROfHHJmumcc&t=701s) Or maybe not have any, um, keywords which are not good, because sometimes models can say stuff that they are not supposed to, right? [chuckles]

[11:52](https://www.youtube.com/watch?v=ROfHHJmumcc&t=712s) Uh, in a casual coffee chat, that's okay, but not in a formal setting when you are being recorded, right? [laughs] So that holds the same for our models as well.

[12:01](https://www.youtube.com/watch?v=ROfHHJmumcc&t=721s) So that's where Amazon Bedrock Guardrails comes into play. You can create, like, your own policy. You can define, uh, your own custom guardrails, uh, give examples of what is not to be shared, even ground your responses, reduce hallucinations, and then figure out how your users are interacting when they are triggered, because everything is logged.

[12:22](https://www.youtube.com/watch?v=ROfHHJmumcc&t=742s) And you can identify your user patterns as well with that, right? So now we have this chatbot, right? Bear with me, just visualize it. Uh, you have established, uh, the data processing using BDA, you have established a RAG application using Amazon Bedrock Knowledge Bases, and you have created guardrails, and you have integrated that with, uh, with your

[12:44](https://www.youtube.com/watch?v=ROfHHJmumcc&t=764s) knowledge bases because there is a native integration. So you would think that, "Yeah, I'm all set," right? Of course, you're all set. But how about the coconuts that we talked about, right?

[12:55](https://www.youtube.com/watch?v=ROfHHJmumcc&t=775s) They have to come into play because we mentioned that. So we will talk about coconuts. Uh, but before that, uh, I was supposed to actually transition it when I say your application [chuckles] is ready.

[13:06](https://www.youtube.com/watch?v=ROfHHJmumcc&t=786s) So now let's say your application is ready, and then you have images, and it's able to give you the responses and search, uh, back in the form of the images, right?

[13:16](https://www.youtube.com/watch?v=ROfHHJmumcc&t=796s) You have ingested the documents, so everything is ready. You're getting the responses back. But yes, now let me come back to the coconuts.

[13:24](https://www.youtube.com/watch?v=ROfHHJmumcc&t=804s) So yeah, the dance. I like, I like the dancing coconuts, but let's derive meaning out of it. So far, we have been talking about data processing, how data is important, how...

[13:36](https://www.youtube.com/watch?v=ROfHHJmumcc&t=816s) which Bedrock features can help. And now coconuts? Yes, because that is one thing that I want you to remember when you get out of this room, because coconuts are super important, and you will see how, right? [laughs]

[13:53](https://www.youtube.com/watch?v=ROfHHJmumcc&t=833s) So actually, what happens is a lot of, uh, people have talked with me about, um, the best practices of building RAG applications, and these are not just for the RAG applications, actually.

[14:07](https://www.youtube.com/watch?v=ROfHHJmumcc&t=847s) They... Some of these concept or most of these concepts can be extended to other generative AI applications as well, right? So we'll talk about that. So I want to, you know, have some time for...

[14:20](https://www.youtube.com/watch?v=ROfHHJmumcc&t=860s) to go into this because this is important. So now you have this RAG application, but remember, we started with the data, and we talked about how chunking strategy is so important, right?

[14:33](https://www.youtube.com/watch?v=ROfHHJmumcc&t=873s) And having the right strategy will make the difference in the accuracy of the generated responses. So this is an important step, right? But this is only a step. The second part is optimization.

[14:48](https://www.youtube.com/watch?v=ROfHHJmumcc&t=888s) I talked about reranking, I talked about parsing, hybrid search, but there are so many other techniques like query reformulation, decomposition. You need to understand how to optimize my application.

[15:01](https://www.youtube.com/watch?v=ROfHHJmumcc&t=901s) But tell me one thing. Can you optimize without knowing what is wrong with your app, with your application? Can you do that? Any raise of hands? No. Right. I can see people nodding heads, so I'll take no.

[15:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=913s) So we cannot because we need to evaluate our application. So we will come to evaluation because before evaluation is we have to see what's going on. And also, when we are talking about optimization, one is the optimizing the accuracy.

[15:30](https://www.youtube.com/watch?v=ROfHHJmumcc&t=930s) The second is optimizing the cost, right? And also the latency. How do we do that? Because these are the three pillars of any generative application: performance, latency, cost. Right.

[15:43](https://www.youtube.com/watch?v=ROfHHJmumcc&t=943s) So how do we do that? One way of doing that is caching your results. So you can use semantic cache, which is caching... Why, why semantic? Because in the generative AI world, we don't have always the exact deterministic questions, neither the deterministic...

[16:02](https://www.youtube.com/watch?v=ROfHHJmumcc&t=962s) deterministic responses up to a certain extent, but not in the exact wordings. So if we have to do it, what we are looking in our cache is not the exact question, but if similar questions have been asked, I don't need to go and invoke my foundation model.

[16:17](https://www.youtube.com/watch?v=ROfHHJmumcc&t=977s) That will charge me money, right? And that's number one. Second, that will also re- because it will take time for the model to synthesize, it will add latency, right?

[16:29](https://www.youtube.com/watch?v=ROfHHJmumcc&t=989s) So we need to take care of both those things. That's why semantic caching. Similar questions quickly retrieve if th- if, if this has been asked before at all. So now we have got three things.

[16:41](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1001s) The second is observability. We need to see what's going on, right? That's where you need to log everything, user queries, retrieval hits, model responses, 'cause that's your only way how you can monitor your, um, application to improve it, to, you know, refine it.

[16:58](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1018s) So that's what we are doing with observability. So earlier when my, um, you know, customers used to ask me about, um, observability or best practices, I would say, "You know, observability is the crit- critical component."

[17:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1033s) But now I say, "Please don't go into production or even do a pilot without observability." How will you troubleshoot? How will you improve? How will you figure out what's going on in the application if somebody complains, right?

[17:27](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1047s) So you need observability. But now when we are talking about observability, now we know we have this additional data. What are we going to do with it? We have to do something with it.

[17:40](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1060s) We need to evaluate it, right? We need to figure out based on my application, like for RAG application, we have context relevance, because if my context is not relevant, my search results are not good, doesn't make sense for me to put them in front of the model and pay, uh, the money and, you know, because everything has

[17:59](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1079s) a cost associated with that, just to get a bad response. Why would I do that? I would first evaluate my search results. If my search results are good, then only I'm going to augment my, my prompt and give it to the model, right?

[18:13](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1093s) So those are the common things that we sometimes, because we are, you know, we are quickly doing everything, we forget about it. So we have to rethink and evaluate our thinking as well as our application. [chuckles]

[18:26](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1106s) So that's super important. But if you have a summarization use case, you can have metrics related to summarization, right? So that's why I was saying, like, these coconut is actually...

[18:35](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1115s) coconuts are good. They're good for other applications as well.

[18:39](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1119s) And then never go blind. Uh, so we have talked about that. But what are we going to do? Let's say in our evaluation something happened, we figured it out.

[18:48](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1128s) We need to update. We need to update what? Sometimes our data, because if our data is stale, our answers are stale. Sometimes we have to update our strategy. So again, coming back to optimization, right?

[19:03](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1143s) And once you have done the updates, please don't, uh, feel that you are done and you have updated, it's all good. You need to test again. So one is you are evaluating your application, you have updated it, but then you also need to have a test suite, because whenever you are doing this update, these evaluations have to

[19:23](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1163s) be automatic. And if you have a defined path on how you are creating this test suite, the chances of you going wrong in your production application reduces, and the chances that you have a high-quality application out there in production increases by manyfold.

[19:42](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1182s) So test. And once you have tested and tested and tested your application, you're ready to crack your coconut, and that's how you will scale. So not just the coconut, we need coconuts.

[19:55](https://www.youtube.com/watch?v=ROfHHJmumcc&t=1195s) And once you have cracked your coconuts, you can decorate it, and you can have your application into production and bear the fruits of success. [laughs] [outro music]
