AI Engineer World's Fair 2026
Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes Leone, AWS
Read the talk
Five code-level controls for more reliable AI agents
A travel-agent walkthrough shows how tool selection, graph queries, independent validation, deterministic rules and runtime steering address different failure modes.
From a talk by Elizabeth Fuentes Leone
Before you start: Familiarity with Python functions, tool-calling agents and basic vector retrieval will help you follow the examples.
What are you paying the model to read?
Every agent response has two token costs: the context sent to the model and the answer it generates. Irrelevant context increases the bill; missing or poorly selected context can also leave the agent answering without what it needs. Reliability starts before the model produces its first word.
Five interventions change what happens around that response: select relevant tools, compute answers through graph queries, validate execution independently, enforce business rules before tools run, and steer recoverable requests toward a valid outcome. These intervene at different points in the agent loop, from context assembly to execution and recovery.
Elizabeth Fuentes Leone, an AWS Developer Advocate, demonstrates each intervention with a travel assistant built using Strands Agents, the AWS-maintained open-source agent framework. The companion repository provides the examples behind the walkthrough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Select tools before invoking the agent
The travel assistant has 29 dummy tools covering flights, hotels, payments, weather and cancellations. In the baseline, every message carries all their descriptions into the model’s context, whether the request needs them or not. Conversation history adds another growing source of input tokens.
A tool is more than a Python function from the model’s perspective. Strands takes the function’s name, description, docstring and typed parameters and generates a tool schema. That schema is what the model reads when choosing an operation. Fuentes Leone estimates roughly 3,000 tool-description tokens per call for the full registry, falling below 300 when only three relevant tools are supplied. These are illustrative schema-overhead estimates, separate from total usage across an agent run.
The semantic-selection demo uses a Jupyter notebook, with an application version also available. Strands calls an OpenAI model through its provider integration and an API key; Bedrock is another supported path. Sentence Transformers creates embeddings locally, and Faiss stores the tool vectors. Ollama offers a local model option without hosted inference-token charges. The registry and its index-building, search and swapping helpers are demonstration code; the travel tools themselves simulate operations.
The spoken evaluation uses 90 queries against 29 tools, with an expected tool assigned to each query. The baseline exposes the full registry and gives the agent a simple instruction to use the correct tool. Each question gets a fresh agent, so this comparison excludes accumulated chat history. Strands usage accounting tracks input and output tokens; the live narration does not establish a consistent total-token savings figure or a precise accuracy percentage.
The filtered version moves tool discovery ahead of agent invocation:
- Embed the incoming question.
- Search the tool index for the top three matches.
- Resolve the returned names to callable tools.
- Construct the agent with those tools and send it the question.
For a normalized Faiss index, the selection boundary can be expressed directly in Python:
python
from collections.abc import Callable, Mapping, Sequence
import faiss
from sentence_transformers import SentenceTransformer
def select_tools(
query: str,
encoder: SentenceTransformer,
index: faiss.Index,
tool_names: Sequence[str],
registry: Mapping[str, Callable],
k: int = 3,
) -> list[Callable]:
if index.ntotal == 0:
return []
query_vector = encoder.encode(
[query], normalize_embeddings=True
).astype("float32")
_, positions = index.search(query_vector, min(k, index.ntotal))
return [
registry[tool_names[int(position)]]
for position in positions[0]
if position >= 0
]
The tool descriptions used to build the index must use the same encoder and normalization. The demonstration reruns the same questions with the smaller tool set, so the changed variable is the available tools rather than a rewritten task prompt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replace the tool set without discarding the conversation
A fresh agent per question avoids a second problem: tool accumulation. In a conversation, continually adding the latest search results can eventually put the entire registry back into context. The demo’s swap_tools helper replaces the active set instead. It clears the registry’s tool collections and installs the newly selected tools before the next invocation, while preserving the conversation messages. It is a locally defined helper around Strands state, not a built-in method to assume exists on every agent.
This keeps tool exposure bounded, but it does not make the whole context constant in size. The conversational run still sends growing chat history, and its token usage rises. Fuentes Leone reports improved tool-selection accuracy while explaining why the test remains difficult: some questions deliberately blur searching, booking and checking, and several generic dummy tools have similar names. Filtering removes some of those competitors unless the query actually matches them.
For deployment, the same discovery step can use a larger self-managed vector store, such as Postgres, or Amazon Bedrock AgentCore Gateway semantic search. Gateway provides managed indexing and discovery for registered tools. The current documented flow requires enabling semantic search and invoking x_amz_bedrock_agentcore_search; it returns relevant tool definitions for subsequent use. Discovery and execution remain separate operations, even when the indexing infrastructure is managed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compute over the dataset, not the retrieved sample
Vector retrieval is useful for an open question such as finding information about a topic. It is a poor basis for answering “What is the average guest rating across all hotels in Paris?” unless the retrieved context contains every relevant record. A nearest-neighbor search that returns three chunks from 300 documents has selected a small subset, not established a complete population for counting or averaging. Without a relevance threshold, it can also return nearest matches when none actually answer the question.
The hotel GraphRAG demonstration replaces that sampling step with a structured query. Documents become nodes, relationships and properties in local Neo4j. The model generates Cypher, and the database executes it over the matching records. The database performs the aggregate; the model explains the returned result. Correctness still depends on the graph representing the source accurately and the query expressing the intended question.
The comparison uses two agents with different tools. One tool embeds the question using Sentence Transformers and searches Faiss. The other describes the graph-query interface to the OpenAI model, accepts generated Cypher, submits it through a Neo4j driver and returns the records. This separates language interpretation from the database operation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four questions that expose retrieval’s limits
The first question asks for the average guest rating across all Paris hotels. The graph returns an average rating of 4.7 for the demonstration’s Paris hotels. The notebook shows the retrieval-based answer calculating the same average from ratings of 4.9 and 4.5, while the graph answer presents the computed result directly. In this small case, retrieval happens to include the needed hotels; once the matching population exceeds the retrieved subset, the same calculation no longer answers the all-hotels question.
The next tests exercise different consequences of that distinction:
- Precise counting: Asked how many hotels have swimming pools, the retrieval agent says it did not find specific information. The graph agent reports no matching hotels. The difference is between missing evidence in retrieved text and a computed empty match in this database.
- Relationship traversal: Asked for room types and prices at the highest-rated hotel, the graph path must first identify the hotel and then follow its room information. The live run includes an error or notification before returning a hotel answer and reporting unavailable rooms; it does not establish a clean, complete price result.
- Out-of-domain requests: Asked about Antarctica, retrieval produces a longer uncertainty response. The graph query finds no matching records, allowing the answer to say there are no hotels listed there in the demonstration database.
The shorter graph responses illustrate the benefit of returning computed facts rather than asking the model to perform arithmetic or fill in missing relationships. No measured output-token comparison accompanies these examples.
The graph itself is built from plain text files with LLM-assisted extraction. Fuentes Leone passes the documents through Neo4j’s Simple Knowledge Graph Pipeline, using OpenAI to help extract graph structure instead of manually constructing every node and relationship. That makes graph construction easier, while leaving the extracted structure as another part of the system whose fidelity matters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate doing the work from approving the result
A tool can return an error while the agent tells the user that everything succeeded. The failure is especially hard to detect when the same agent both performs the operation and judges its own response: a missing booking can become a confident confirmation. An independent validation pass checks whether the reported outcome follows from the execution evidence.
The demonstration divides the work among three roles, with Strands Swarm handling handoffs rather than a manually written orchestration loop.
| Agent | Responsibility |
|---|---|
| Executor | Use tools to fulfill the request |
| Validator | Review what the executor actually did |
| Critic | Approve or reject the result |
The roles are supplied through system prompts. The test data includes a valid booking and failure cases involving unavailable hotels, nonexistent hotels and missing bookings. Only Strands with its OpenAI integration is needed for this example.
The executor is the swarm’s entry point, and the demonstration configures a six-handoff limit. Alice’s Grand Hotel booking passes from execution to validation and then approval. For a missing or nonexistent record, the executor encounters the error, the validator catches it, and the critic rejects the result. The user receives a clear failure instead of the fabricated success produced by the single-agent comparison. This is a check on the response after attempted execution; it does not replace enforcement before an operation with side effects.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Enforce business rules before a tool runs
A maximum of ten guests per reservation is an executable condition. Writing it in a system prompt or tool description only gives the model text to interpret. A pre-tool validation hook checks the actual proposed arguments and cancels the invocation when they violate the condition. This is the neuro-symbolic pattern in the demonstration: language-based planning combined with deterministic checks at the execution boundary.
Strands hooks provide the interception point. A HookProvider registers callbacks through HookRegistry; BeforeToolCallEvent fires before the proposed tool executes. AfterToolCallEvent is also available but is not used here. The callback examines the tool name, arguments and relevant state before deciding whether execution may proceed.
The rules operate on simulated booking state: check-in must precede checkout, a booking may contain at most ten guests, and confirmation requires verified payment. A cancellation rule forbidding cancellation within 48 hours of check-in is defined but not used in this run. The core booking and confirmation predicates can remain separate from the framework callback:
python
from datetime import date
def booking_errors(
check_in: date, check_out: date, guests: int
) -> list[str]:
errors = []
if check_in >= check_out:
errors.append("Check-in must be before checkout.")
if guests > 10:
errors.append("Maximum 10 guests per booking.")
return errors
def confirmation_errors(payment_verified: bool) -> list[str]:
if not payment_verified:
return ["Payment must be verified before confirmation."]
return []
The hook routes a proposed book_hotel call to the booking checks and a confirmation call to the payment-state check. An eleven-guest proposal fails before the booking tool runs. Keeping these predicates outside the model makes the enforcement decision inspectable and repeatable.
The comparison tests confirmation without payment, an over-limit booking and a valid five-guest booking. The added hook configuration is the intended difference between the baseline and guarded agents. The baseline’s basic prompt permits confirmation without payment; the guarded response blocks it. The over-limit case similarly produces a guarded response citing the ten-guest maximum.
The final notebook table exposes an additional failure that the narration’s description of the valid case does not capture. In the displayed three-case run, baseline accuracy is 1/3 and guarded accuracy is 2/3; the guard blocks both invalid operations but also rejects the valid booking. Enforcement must be evaluated for valid requests as well as invalid ones. The code can reliably apply a rule while the overall configuration still prevents work that should be allowed.
Fuentes Leone connects this pre-tool enforcement pattern to AgentCore Policy at the infrastructure level. The next problem is what happens after a rejection: binary approval or cancellation is appropriate for some constraints, but other requests have a valid alternative that the agent could find without making the user start again.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return corrective guidance when recovery is possible
A room that accommodates four cannot hold six guests, but two rooms might satisfy the request. A full flight might have a suitable alternative departure. In the preceding hook configuration, rejection stops the attempted operation. Steering adds a recovery path: send the reason and corrective guidance back to the model, then let it propose an admissible alternative.
The Agent Control demo also moves rule management out of the agent’s Python source. Its SDK connects to a local control server, where rules are registered through an API. Updating those rules does not require editing and redeploying the agent. Two controls demonstrate the distinction: steer requests above ten guests toward an acceptable arrangement, but deny confirmation without prior payment.
The concrete request is to book AnyCompany Lisbon for 15 guests. The hook-only baseline blocks it because a single booking exceeds the limit. The steering integration introduces two components: AgentControlPlugin captures agent events and sends them to the control server; AgentControlSteeringHandler receives steering decisions and delivers them back to the model. Together they close the feedback loop between a rejected proposal and the agent’s next action.
For that same 15-guest request, the local demo reports a successful reservation split into two rooms: ten guests in one and five in the other. The original over-limit proposal is not itself a successful booking; the reported success follows the adjusted arrangement. Hard constraints still govern each operation, while steering gives the agent a way to preserve the user’s overall goal.
Keep the 15-guest request while changing the booking arrangement
Constructed example: Stable object IDs and the first-room/second-room labels organize the comparison; no booking identifiers or unshown tool-call records are asserted.
Book AnyCompany Lisbon for 15 guests.
Operation: Return steering feedback for the over-limit proposal and let the agent revise the room arrangement.
Hotel
AnyCompany Lisbon
AnyCompany Lisbon
Requested guests
15
15
First room
15 guests proposed; exceeds the limit of 10
10 guests
Second room
Not present
5 guests
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry the controls into a deployed agent
The production architecture moves the local agent into Amazon Bedrock AgentCore Runtime. Runtime can host Strands or other agent frameworks, with short-term and long-term memory and CloudWatch observability supporting operation beyond a notebook.
Gateway routes tool calls to Lambda functions. Steering rules live in DynamoDB, so the proposed architecture can pick up rule changes on the next call without redeploying the agent. Neo4j AuraDB supplies the external graph database option; Fuentes Leone mentions its free tier. The deployed policy and its recovery behavior still need to be checked independently: the current production example describes reducing a 15-guest request to ten, whereas the local demonstration preserves all 15 guests through the two-room split.
AgentCore deployment requires AWS credentials. Fuentes Leone points to possible promotional credits and describes notebook and AWS Cloud Development Kit deployment paths. The inspected repository documents CDK infrastructure deployment and a notebook for local testing, so use its current CDK instructions for provisioning rather than assuming the notebook deploys the infrastructure. Credit availability is separate from the architecture’s operating costs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose the intervention at the point of failure
The five techniques answer different diagnostic questions. If every request carries irrelevant tool schemas, filter before invocation. If an answer requires a count or aggregate that was never computed, query the dataset. If execution errors become success messages, separate execution from validation. If a business rule must hold, enforce it before the tool runs. If rejection leaves a feasible task unfinished, return guidance that lets the agent try a valid alternative.
The learning path is incremental: work through the five local demonstrations, available as notebooks and applications, then carry the controls into deployment. Each step gives a distinct behavior to inspect—what entered context, what the database computed, what execution actually returned, which rule fired, and whether recovery preserved the request. Those observations make the final question practical: which of these failures can you already reproduce in your own agent?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Five local agent-reliability demonstrations and a sixth example for deployment on Amazon Bedrock AgentCore.
FAISS-based tool filtering, comparison notebooks and a script for examining token usage with conversation history.
Compare vector retrieval with Neo4j queries for hotel FAQ questions, aggregates and relationships.
Examples of external rules that guide an agent to retry a request or deny an operation.
CDK deployment instructions for a booking agent, DynamoDB rules, Lambda tools and optional Neo4j graph retrieval.
Neo4j's guide to extracting entities and relationships from unstructured text and writing them into a graph.
Current instructions for enabling and invoking natural-language discovery of registered gateway tools.
Further reading
- RAG-KG-ILPaper
A hybrid retrieval, knowledge-graph and multi-agent framework evaluated on NHS-grounded health questions.
Read the complete timestamped transcript
- 0:00
Hi. Today, we are going to talk about how to stop AI agents' hallucinations with five techniques beyond the prompt. Each one is a code change, not a prompt change.
- 0:16
Let's see. Every time your AI agent responds, you are paying for the words going in and the words coming out. In your bill, you will see those calling tokens.
- 0:30
Yeah. And the more tokens you send in, the more you pay. And if what you send is not quite right, too much or missing something important, your agent start to hallucinate. [chuckles]
- 0:47
There are five techniques to help reduce tokens waste, improve accuracy, and catch failure before users see them. And each one is a code change, not a prompt change at all.
- 1:05
So let's see it. First, we have semantic tool selection. You filter which tool go into context on every call. The model only see what is needed for that specific query.
- 1:23
Second, we have GraphRAG for persist queries like aggregation, counts, multi-hop reasoning, and you replace the text retrieval with a structured graph query.
- 1:38
The model gets a compute, uh, verifiable answer, not a sample, as RAG do.
- 1:47
Three, multi-agent validation. A second agent can check every response before it reaches the users. And four, neuro-symbolic guardians. You rule live in Python, not in the prompt, and the model cannot skip them.
- 2:09
So five, runtime guardians, because if you don't want to block, you can steer, and you don't c-- you don't need to block everything. And when a rule fires, the agent self-correct and complete the task.
- 2:26
No hard stop, no user retries. So for each technique, I will show you the agent without it and then with it so you can compare. And all the demos are...
- 2:40
I'm using a travel agent that I built using Strands Agent. And Strands Agent is a open source agent framework that we maintain on AWS.
- 2:56
And I am Elizabeth Fuentes Leone. I am a Developer Advocate for AWS. I'm focused on agentics application. And here in this QR code, you will find everything that you will need to recreate all these techniques that I'm going to show you in a moment.
- 3:21
So let's get into... Let's get into it.
- 3:32
So semantic tool selection. Our travel agent has 29 tools: flights, hotels, payments, weather, cancellations. All, all are dummies tools, are not like a travel agent for real.
- 3:50
But every time that user sends a message, all the 29 tools description go into the context windows.
- 4:00
The model reads all of them before deciding what to do. And if your agent has memory, the rows to every conversation adds more context that gets sent with every single message.
- 4:20
And you pay for every single one of those tokens, whether the model ends up using that tool or not. To understand where those tokens come from, you need to see what a tools actually looks like to the model.
- 4:39
In Strands, you write a function with the tool decorator, that's the tool, and a name, a description, and a docstrings typed parameters. Then Strands take that and generate a schema with name, description, parameters, and the schema is what goes into the context windows on every
- 5:04
call. Each tool schema is about seventeen or two hundred tokens, depending on how many parameters it has. If our travel agent has twenty-nine tools, that adds up to somewhere around three thousand tokens per call just for the tool
- 5:29
description. Before your message, before the response, every single call. By creating a tool database, we can filter the tools that the agent may needs before the agent is invoked.
- 5:48
With this filter, the models sees only three most relevant tools.
- 5:55
Tokens usage drops from thousands to fewer than three hundred. Let me show you in the code Here in my QID. This is the... Let me clear all the output.
- 6:12
So first, we need to install the requirements. Here is the requirement file. This is a Jupyter Notebook because it's more simple to show everything, but it is one application that you can run if it is more comfortable for you.
- 6:27
So here are the requirements. I have the Strands agent, and because I'm using OpenAI as a model invocation, I'm using the API from OpenAI, and I need a Strands agent for an OpenAI.
- 6:41
You can use Open-- a Strands agent with, oh, my God, almost all the model provider and with a Strand- with Amazon Bedrock, of course, because we as AWS, we are the maintaining of this framework.
- 6:55
And to use Amazon Bedrock, you don't need to add, uh, the model provider. And because I need to embedding, I need to create embeddings for my vector tools database, I'm using the sentence transformer.
- 7:12
This is a super simple model that runs locally, so it's free. So you want... You can run this locally in your computer without expense another model embedding. And, eh, you can use Strands with Ollama too.
- 7:28
So if you have a local model, you can run everything in this, in your computer for free without spend any tokens. And I use infise as my vector store.
- 7:42
It's local, super simple. And well, right now I'm gonna use Neo4j. This is for other, eh, demo that I'm going to show you in a moment. And because I'm using some environments, I'm using Python of.
- 7:56
And let's see. So here, I'm going to run this. Let me put it bigger. I know that you're having some problems. Put bigger. Little mer, little less.
- 8:07
And here, this is better. So I install my requirement. I already did that. And I'm using OpenAI, so I need my a-my API, my API key, API key. And here, I invoke my Strands.
- 8:23
I need the agent because I'm going to build an agent. I'm using OpenAI, and I have a bunch of tool, dummy tools here. I le- I'm going to show you in a moment.
- 8:33
And I'm going to use this because I have some, eh, functions that I'm using to, uh, create, build a index for my vector store, and I need a search tools.
- 8:46
When I use the vector store, I put my query there. I search for the, I search for the tool that I'm going to use using vector, um, um,
- 8:55
search for vectors, and I swap the tools. I'm going to show you that in a moment. So let me show you all my dummy tools. Where is the engage?
- 9:05
Register. Here. Here I have all my tools. Uh, this is for swap, search, build index, and where are my tools? Choo, choo, choo, choo. Here. So here are all the dummy tools I create for this.
- 9:21
You know, it's super... This is a demo, please. This is not something that you can use to go to production. No, please. And okay. So what is this? This one.
- 9:32
So I run this, and I build my semantic index. I already did that by, uh, yeah. So I have twenty-nine tools here, twenty-nine dummy tools, and I'm going to test this with a lot of different queries, and this is the ground truth.
- 9:49
Eh, I have a ground truth because I, I know what is the best, um, the best tool to answer the question so we can know if our agent is okay or not okay.
- 10:00
So let's run this. We have, eh, ninety queries and twenty-nine tools. So this is some helper functions. I don't care. Yeah, I care, but I'm not going to explain you that.
- 10:13
So this is my agent, my traditional agent, where I'm putting all my twenty-nine tools inside the agent. I have my model, some helper function. This is my agent. I only...
- 10:26
This is the way I create a agent using a Strands agent. I put all my tools, and my system prompt is, "You are a travel assistant. Use the correct tool to answer question."
- 10:36
Oh, my God. Super. [chuckles] And the model. Let's run this. And this is, uh, whatever. This is some, eh, function to count the tokens because in-inside the Strands agent you can, you can count the tokens.
- 10:51
You, you can know how many tokens are using for in and out the, um, the agent. This is some helper function. And yeah,
- 11:00
this is every time that I put a, uh, a prompt inside this agent, I'm only sending the prompt and the agent respond. Then I, I use a new agent because this is a, uh, a for, so I'm not having a conversation with the agent.
- 11:15
I only said, uh, a question and I receive an answer. So for each question is spent around two thousand tokens. So in the end, not always it give me the right answer here.
- 11:31
So yes, I have, um, a not so good accuracy and the average is one thousand tokens. Now, let's use my new agent with the semantic approach.
- 11:44
The thing, the first thing that I do is for every single query, for every single, eh, prompt, I send the query to my search tools and I'm going to receive the top Q, the top K three, uh, most relevant, um, with more probability to response my query because this is a semantic search inside my vector store.
- 12:09
And- Then I use that response, I'm going to select that name, and I'm going to put that tools inside my agent. So I only going to use the three tools that this semantic search retrieved me for my agent, and I send that only tools.
- 12:28
And this is my function helper, and that's it.
- 12:32
So I am going to send the, all the queries again the same. And for the first question, I have 4,000 tokens to read blah, blah, blah. And yeah, we have a huge difference because I don't send them all the 29 to- tools in all the queries.
- 12:51
So yeah, pam-pam. And okay, semantic memory. Yeah. Okay, now come on, finish. So well, let's, let's, let's go to the next one. So this agent is only sending a question, and I receive a response.
- 13:08
I don't having a conversation with this agent. What happen if I start to have a conversations, if the agent remember me? So we need to, we need to send only the tools that this agent is going to use because if I give the, all the tools I'm using in the conversation history, uh, then I...
- 13:28
It's going to be in a moment that I'm going to have the 29 tools inside my agent. So we are not resolving, like we are not having a, a, um, we are not resolving our problems in the agent when we're having a conversation with the agent.
- 13:44
So we need to put the tools, and then we need to remove the tools, and we can do that with the swap_tools. And let's run this. And what is doing swap_tools?
- 13:58
I will run this. No. Let me, let me show you first the swap_tools
- 14:04
function here in the register. I delete this because with Strands,
- 14:11
in each invocation, you have complete control of the status because this is a agentic loop, and you can use this agentic loop
- 14:23
how or... Because this is an agentic loop, and you can do almost whatever you want. You can put and remove everything inside the loop with some few lines of code.
- 14:40
So let's see. Here we have the agent. We have the tool register through the tool registers. It's something inside the agent status. So we can clear the tools, and that's it.
- 14:55
So in the next invocation, we can clear the all tools, and we can add the new tools.
- 15:03
Here, where is my... I have a lot of things here.
- 15:09
So I run this. It's going to take a moment.
- 15:28
And yes, we can see that in each invocation, my amount of token is increasing. Why I have more? Because I'm sending the chat history as well. So I send the tools that I need, only the tools that I need, and the chat history.
- 15:46
So that's why I can see that, uh, my amount of token is bigger and bigger.
- 15:59
And that's it. The accuracy is better. And yeah, we have more tokens.
- 16:08
So yes, probably we don't have the best accuracy here because this is a super demo with super dummy tools, and some queries are ambiguous on purpose because search for something, book for something, check something, and the demo has a generic tools, dummy tools with some similars name.
- 16:33
And when all the 29 tools are visible, the model sometimes pick the wrong one. And with the filtering, those generic tools, uh, only appears when the query actually match them.
- 16:48
So you can run everything, and you can test this. So this is only the run only local. But what happen when you want to put this demo in production?
- 17:01
You can, of course, you can build a bigger vector store. I don't know. You can use Postgres. Yeah, I think it's too much. But we on AWS, we have Amazon Bedrock Agent Core.
- 17:14
So Amazon Bedrock Agent Core is a service dedicated only to agents in production. And inside of Agent Core, we have Agent Core Gateway. So Agent Core Gateway, Gateway allows you to build this index, uh, alone.
- 17:36
So you only have to say, "Hey, this is my tools," and Agent Core Gateway is going to build everything for you. And it can have the, the vector search inside.
- 17:48
So the routing layer is inside the Agent Core, and it handles the tool selection automatically. So you register your tools once, and it find the right one for each request.
- 18:01
It's the same principle but without infrastructure to manage. Now let's see the next one.
- 18:11
GraphRAG. You know, RAG is Retrieval-Argument Generation. Is how agents access your own data. You take the user question, search your documents for the most similar content using vector search, and pass what you find to the model.
- 18:31
The model answer from that, it's worked well from open question, "Find me something about this topic," but there is a category of question where that breaks down. What is the average rating across all hotels in Paris?
- 18:48
How many hotels have a pool? Vector search always returns something, even when nothing is truly relevant, and the agent only sees the top N chunks of your old data at a time.
- 19:06
It cannot aggregate, count, or traverse relationship across all the full dataset, so it estimates, and it present that estimate as a real fact, you know? A real answer.
- 19:22
So here we have RAG. You build the vector store, and it retrieve three chunks from three hundred documents, and the model guesses.
- 19:34
If we are using GraphRAG, you can run a query across all the data and returns a compute results. Graph address this differently, because instead of retrieving text chunk, you can build a knowledge graph from the documents, nodes, relationship, structured data.
- 20:00
For the demo, I'm going to use Neo4j locally, and the model is going to write a Cypher query to search it. Cypher query is Neo4j query language. It's similar to SQL.
- 20:16
So the graph run that query across all the data. The model gets back a compute verified results, not a sample. And before running this demo, I'm... we need to install the dependence.
- 20:32
Let me show you. Let, let's go to the code.
- 20:38
So graph. This is the notebook that I want to sh-share with you, so we need to install the requirements again. What we have here, let me see.
- 20:48
This is the first one, yeah. Requirements. I have... I'm going to use OpenAI again. Let me just close this. I'm going to use Neo4j, and we need Neo4j for GraphRAG.
- 21:01
And for the, uh, agent that is using normal RAG, we are going to build a super simple, uh, vector store in Faiss, and we are going to use again the sentence transformer.
- 21:15
And let's see. I already have all. I invoke my OpenAI.
- 21:21
This is my tools. I, I... because I want to create some tools here, I'm going to use OpenAI, and I'm using graph database. And this is my... I have my Neo4j locally.
- 21:34
And let's run this. And this is something to check Neo4j. So I'm building my, uh, vector store. This is my Faiss, and this is my Neo4j that I have here locally.
- 21:47
I don't know if I can show you. I think not. And I have a tool, a normal tool created with Decorator. This is to search inside the, my vector store.
- 21:58
You see that super... I, I have the query. I co- I create, I dec- I create the, the embedding for my query. Then I search inside the vector store, and I have the query for the knowledge graph.
- 22:10
So here the model need to understand that to search in the query, in the graph knowledge base, it need to build a Cypher query. So I put that in the context because we already know how to build, uh, tools, right?
- 22:26
I put that in the context. I have my driver to send the data to, uh, to connect with the vector store and to send the Cypher query that the model is going to create for me, and this is to read the results, and that's it.
- 22:41
That's the only tool that I need to search in my knowledge graph. Okay. This is my model OpenAI. This is my RAG, my RAG agent, and this is my graph agent.
- 22:52
I have two different agent to compare the results. Let's see what, uh, what happen. Yeah. I'm ready. So let's do... Let's run the first test, aggregation.
- 23:10
Here, what is the question? What is the average guest rating across all hotels in Paris? So meanwhile this agent is, is... Oh, it's already done. So what I have here,
- 23:23
I have the average guest rating across the hotels listed in Paris is, um, it calculates. You know, when something, when I'm using RAG, it go to the, to the vector store.
- 23:36
It receive the N, uh, possible answers for this question. Because this is an aggregation, it's going to use the data that it receive to build a mathematical, uh, operation.
- 23:50
This is the thing that I have here. So everything that we see here is the, is the model reasoning. So when I run the graph agent... Oh, let me...
- 24:01
The agent is only going to give me the answer, you know, only these tokens here. Because the Cypher query, it can give me the mathematical operation itself. It can do that.
- 24:13
So we don't need the agents, the LLMs, the model, uh, do that for me because the Cypher query already give me the right question. So it is forty-seven and the average.
- 24:27
So here in this, um, first, we are lucky Because probably they are, yeah, only two hotels. But what happened if this vector store have more than two or three hotels in the, in the vector store.
- 24:44
So we are not going to give a real answer. It's going to c-- the LLM is going to calculate with the only three answer that it received. So it's going to build this operation with only three hotels.
- 24:59
But if the vector store have more than three, we are going to have some problems. Something that, um, and I, um, yeah. So let's run the next one, the precise counting.
- 25:13
Something similar to how many hotel have a swimming pools as a amenity? How many again? So it give me the, the traditional, "It's appear that the search did not returns any specific information about hotels in Paris."
- 25:30
Okay, what happened with the other one? "There are currently no hotel that offer swimming pools." No, why? There is no hotels. So the other one is like, "Mm, would you like to ask about hotel with other specific amenity or information?"
- 25:46
It's like, mm, maybe it is or maybe I don't know. Yeah. It's not so accurate, okay? So multi-hop reasoning.
- 25:58
Oh, what is my question? What are the room types and price for the highest-rated hotels? What are the rooms types and price? These two question. So it search the fact I had to have only one.
- 26:14
"The higher-rated hotels in any company Paris," blah, blah, blah. I, I don't speak French. With the guest rating is that one. Whatever. I currently don't have... Okay, blah, blah, blah.
- 26:26
A lot of data there. So what happened with the other one? Receiving notification for... Oh, I have error. [chuckles]
- 26:35
Uh, here in the second one I received the, the question. "The higher-rated hotel is Harmony," blah, blah. "They offer following types. But unfortunately, these room are not available." So I receiving a que- a answer.
- 26:49
I don't receive a lot of blah, blah, blah there. I only receive the answer what I need. And for the next one, out-of-domain detection.
- 26:59
What is the question? Tell me about hotels in Antarctica. A spoiler: there is no hotel in Antarctica. There is zero hotels in Antarctica. And let's see. "It's appear that the search did not return specific information about hotel."
- 27:14
Okay, because there no. Ah. "As such, currently do not have detail," blah, blah, blah.
- 27:20
If you are looking for particular type experience or specific inquiry about visiting Antarctica, no, please let me know and I can assist you for it. Okay, a lot of tokens that is pending there in that answer for LLM.
- 27:34
What happened with the other one? "There are currently no hotel listed in Antarctica." Of course, because it create the cyber query, the cyber query, and it receive zero. So it no, it give me a honest answer.
- 27:49
Here's some summary that, uh, Claude create for me for this in Jupyter Notebook. But here is something that I want to show you before go to the other one.
- 28:00
So here, something that I love for Neo4j, because why I'm using Neo4j? Because in the library,
- 28:09
the Neo4j give me, uh, it can build, it use a LLM, I'm using the OpenAI as well, to build a knowledge graph. So how I build my knowledge graph, I only have a bunch of data, a bunch of TXT, only text, and I send that data to the Neo4j library that I miss it.
- 28:36
Here, I send that data to this, and Neo4j using this all, this, um, library just right here, it can understand all my data, and it can build the graph for me.
- 28:51
So I don't need to create that using the simple Knowledge Graph Pipeline inside the Knowledge Graph library. So, uh, that's why I'm using Neo4j. It's amazing and it's super simple to use, so I invite you to use that.
- 29:08
So let's go to the next one. So multi-agent validation.
- 29:17
Sometimes a agent fails and nobody find out. It calls a tool, the tool returns an error, and the agent does not surface that error. It generate a confident success response instead.
- 29:33
The user think it work, you think it work, it did not. The agent acts and validate its own output in the same loop. There's no separation, no second opinion, so when something goes wrong, it rationalize and tells you, "It's okay, it work."
- 29:55
Here is what happens. Inside a single agent, when it fail, it calls the tools, gets an error, rela- re-rationalize it, and returns a success response. The user never sees the error.
- 30:14
You can address this by adding a validation layer. You can have three agents in sequence. One acts, one checks, and other approve or rejects.
- 30:30
Strands Agent has a built-in class for this called Swamp. It manage the handoff between agents automatically. You just define the role of each agent in a system prompt.
- 30:48
And let me show you that. So this demo only needs a Strands Agent with OpenAI integration. Let's see the requirements here. We only need a Strands Agent. We don't need anything more.
- 31:04
Let's go to the notebook here. And the key important thing here is the... I already run this. Let me see here, is the swarm. The swarm is what lets you to connect multiple agents together without create like a for or while manually to put the all these agent
- 31:29
together. It can build the chain and manage the handoff between them automatically. So here, we are going to create three agents. Let me go that because we have the normal and we have some ground true data.
- 31:47
So here is to create a, a single agent. So we have... We are going to test three different scenarios to valid a booking. They sa-- they we expect true.
- 31:56
To unavailable hotels, we expect false. And no existing hotels and missing booking, we expect false too. This is how we create a single agent. We need a prompt, some tools.
- 32:08
And we are using the tools that we have here
- 32:12
and here. We know that this is the data, so the agent can, can give us the answer that we are looking for. And let's see. Let me go to the other ones.
- 32:23
Uh, wait. Here. So how we build the swarm.
- 32:28
I already run this. I want to show you the swarm. So we are going to build three different agents: the executor, the validator, and the critic.
- 32:37
The-- We have a system prompt. You are an executor agent for a hotel booking system. Use the provided tools to fulfill requests accurately and blah, blah, blah. The validator is you are a validator agent.
- 32:52
Review what the executor did and output exactly on/off.
- 32:58
And we have the critic that is going to say yeah, approve or not yet. Let's run this. And oh, what is here? Oh, model. Didn't define the model. Sorry.
- 33:10
Sorry, please forgive me the life because I didn't run this one.
- 33:17
Let's run, uh, because this is the, the normal and the model is there, right? What? Build path, yes. Run that.
- 33:29
What's happened with you? Let me go this. Let me copy and paste this one. I don't know why this give me problem.
- 33:39
Multi-agent swarm. Is it... I'm the same name. Come on. Yeah, you can see this is life. I don't going to edit this. [chuckles]
- 33:48
So yes. Okay. We have... Mm. Thank you. And the swarm. How we, how we create the swarm? We have swarm, and the swarm function, we put all the agents together, and the entry point is the executor.
- 34:05
So all everything is going to start in the executor, and then it's going to hand off
- 34:10
and six time because you can handle that too. So get the swarm final response. Let's do a, a for and let's send all the response. And book around hotels.
- 34:22
I have booked around hotel for Alice tonight. Booking blah, blah, blah. Hand off. Valid. Okay. Let's go. Verdict, approve. So the critics is approving the this. This is the other one.
- 34:38
And we can run some comparison here if you want.
- 34:44
So here we can see that the swarm handles the flow between them, between all the, uh, three different agent. It was what happens when the single agent tries to confirm something that does not exist in the system.
- 35:00
And now the same request through the swarm, the executor gets the error, the validator catches, the critic reject this, and the user never see a fabricated response. So you can...
- 35:14
We can test here with a single agent that it's, uh, substituted on no entry and return success.
- 35:23
And the swarm, it have a executor that got the error, validate, say, "Hey, come on, man," hallucination, and critic, they reject. So the user sees a clear failure, the fabricate confirmation, so missing the user here.
- 35:41
And you can run this to compare the two different agents. So here we can see that the single agent, it have everything like, uh, yeah, it's okay. It's correct.
- 35:54
But at the multi-agent swarm, you can see that the last one is saying, "Mm, you know, missing issue here. Uh, something is happen." Now, let's go to the next one.
- 36:06
Okay. Neuro-symbolic guardians. You have a rule for your agent. Let's, uh, say maximum ten guesses per reservation. You write it in the system prompt. You even write it in the tool description.
- 36:22
And the agent still calls the tool with fixed ten. Not because it is ignoring you. Because prompts probably are suggestion, not constraints. The model process them as a text No as a logic it has to execute.
- 36:42
It's probabilistic. Only code execute logics. A rule in the prompt, the model reads it as a suggestion. A rule in the code, the model cannot skip it. Neuro-symbolic guardians rules put the rules in the code.
- 37:02
Strands Agent has a feature called hooks. The f- the function that Strands calls automatically at specific moments in the agent loop. It is this case right before a tool executes.
- 37:18
So you have a hook, you write the rule, check the parameters, and if they fails, you cancel the call. Let me show you this in the code.
- 37:31
Here we have the Jupyter Notebook. We have the requirements here. Um, we have the same requirements that the previous demos. We only need... I will start everything again. We only need Strands and OpenAI.
- 37:45
We don't need anything else. And, um, my API.
- 37:53
And here is the thing important. In this demo are these three hooks provided in the Strands base class, so you s- that allows you to create the hook. So we have hook provider, hook register, and before_to_call event.
- 38:10
And the tool regi- hook register is what Strands pass you to register your callback, and before_to_call event is the event that fires every time the model is about to execute a tool.
- 38:27
That last one is what makes it possible they intercept the call before it runs. So you have the before_to_call event and of course we have after_to_call event that we are not going to use that here.
- 38:42
And let's see. The rule in the prompt, remember, the model reads as a text, and it may follow or not follow that. So let's see this. I'm going to run this.
- 38:55
So here we have some simulated state that we are going to use in this agent. And let me... This is the symbolic rule. So booking rule. If, if rule...
- 39:08
Booking rule as a... This is something that
- 39:11
we create. Let me show you the rules here. So we have the rules. Rule one, validate dates. Check is month before checkout equals validate date checkout. This is something that it check.
- 39:27
This is, uh, something that is going to invoke when the rule is, is, uh, is most... We have to use it. And,
- 39:39
uh, we have another rule for max guests. Maximum tech guests per booking. So if I want to use, I don't know, if I want to booking, uh, uh, I want to do a booking for 11, it's going to block me.
- 39:56
It's going to reject the booking. And we have some confirmation rule, payment before confirm. You know you can't confirm if you don't have the payment. Cancellation rule. Cancellation within...
- 40:10
cannot cancel within 48 hours of check-in. So this is something that I've created here, you know, to this demo. So I have booking rules and I have confirmation rule.
- 40:21
I don't use in here the cancellation rule. So create validation hook. We create the validation hooks here, right, like this. Neuro-symbolic rule. And we use the hook provider. So we have a bunch of code here that we add the booking rules, the confirmation rules in the state.
- 40:45
And you can check this by yourself later, but we want to see the demo running so that we define the clean tools that we are going to use the- for the booking hotels and the presents payment.
- 40:58
This is the, the normal tools, you know. And we need to add the tools to the hooks. I think I add that.
- 41:07
Yeah. Tool name, book hotel. So this thing is going to get triggered when the book hotel is using the...
- 41:19
Yeah, the book hotel. Okay? So let's create the agents for the comparison. So we are going as the other demos. We have three different scenarios. Confirm booking without payment.
- 41:33
Payment must be verifiable for the confirmation. This is the rule that it must trigger this. We have booking hotel exceeding guest limit, and we have valid booking for five guests.
- 41:46
So yeah, we have the three scenarios, and we have the normal agent, the normal agent, the baseline agent, and we have the agent with the neuro- the neuro-symbolic guardians.
- 41:58
So we have the hook, the neuro-symbolic hook.
- 42:03
Uh, this is the same here. Yeah. And
- 42:07
we add this. If you can see, the normal agent is only three lines, the tools, the model. And this is the line that give us the difference between these two agents.
- 42:19
So yeah, really run that. Now let's go to
- 42:25
confirm a booking without payment. So I'm running this, my agents. The first one is obvious. Come on, man. I... Yeah, because I... Yeah, it confirm the booking without payment because it doesn't have any rules and the prompt is super basic.
- 42:41
And the other one is Let's open this, and is booking blocked. Payment must be verified before the confirmation. Thank you. So it's okay. Now let's run the other one.
- 42:57
Test second scenario. What is the question? Booking hotel excess guest limited. So the question is... Ah, yeah, I have more than, so
- 43:12
here, uh, the hotel have successful booking for... Yeah, because the prompt is basic, and I don't put in any rule there. And for the other one, it seems that the Grand Hotel has maximum capacity of 10 guests per booking.
- 43:25
Additional booking must be made at least one day in advance. I don't remember the day. Probably this is something that I give them. And to validate booking is both agent execute all rule pass because it, yeah, it's only a validation for booking.
- 43:42
So book hotel for blah, blah, blah, all rule passes.
- 43:47
It seems that the booking still needs to make... Ah, I don't... I didn't remember the question. So booking is made provide... Ah, yeah. Okay.
- 43:57
Okay, I get it. And okay. Now run all that in this question. Confirm without payment, booking safe max, and what happened here? This is all the scenarios that we just run, so this is, uh, something to compare the results.
- 44:12
So valid booking kinsa ka, uh, five guesses. Allowance, correct. Wrong,
- 44:19
50. Blocked, wrong. Correct, confirm. So the... We have here the comparison between the two agents.
- 44:34
So what we are, what we have here is same model, same tools, same prompt, and the different the outcome because the rules are in Python knowing the prompt. This pattern of enforcing rules in code before the tools run is also what Amazon Agent Core Policies service that we
- 44:58
have does at infrastructure level. So the same concept, but managed for you in production, and you only have to create the rules. So the hooks are all or nothing.
- 45:12
They block everything or approve. But sometimes you want the agent to adjust and keep going, not stop and leave the user waiting. That is what I will show you in the next.
- 45:32
Runtime guardians, runtime steer, steer don't block. This is the next one. Hooks blocks unconditionally. The agent stop, and the user has to retry.
- 45:49
For a hard constraint, that is exactly what you want. But sometimes the rule is softer. Maybe room fits for four guests, but a group of six could go book two different rooms.
- 46:08
Or a flight is full, but there is availability on the next one. You do not... You don't want to block everything. Probably you want the agent to find a option and complete the task.
- 46:23
That is steering. Here we have the hook that fires and the task fail. And with Agent Control steer the models and the task complete. The other difference here is operational because with the hooks it's as changing a rule means changing code and redeploy the all harness,
- 46:48
all the agent. And with the Agent Control, we use the name of the open source library that we are going to use here. Rules are registered on a local server via API.
- 47:02
You update them without touching the agent code because the agent picks them up immediately. Let me show you that here in the code.
- 47:14
Yeah, this is the notebook. In this demo, we only need a extra packet, the
- 47:23
Agent Control SDK. So the Agent Control is the one that helps us to create the steering. And here we are using the setup control. This is a, a little application, a little app that I create with the local server and the steering rules.
- 47:46
Here we have the local server. We have the control. There are the steering rules. First, we have the,
- 47:54
the steer max guests. Steer, you know, guide. So guide agent to reduce guest count when exceeding maximum of 10,
- 48:06
and it's going to steer. And we have some control that deny. For example, deny no payment. Block, blocking confirmation without PR payment.
- 48:20
And here, well, this is to, um, to create the, um, the service, the server. And let's go to the, uh, notebook. So I already... Well, let me go here.
- 48:36
Yeah. So I need the environment. So this is the agent. Uh, we're going to have error here. Wait, wait, wait. I don't want to use Bedrock. Why? Why? No, let me comment this one because this is going to give me a error.
- 48:57
And let's run this. Yeah, so we have the hooks as we did in the demo. So we have book any company in Lisbon, and we have some prompts. You are a hotel booking assistant.
- 49:10
When booking first, describe what you will book and blah, blah. And this is the prompt for my agent. Where is my agent? Agent control. This is for the... Well, this is some, some helper.
- 49:23
And this is my hook that you already know because we create this. So we have the system prompt, and we have the hooks. So let's test this agent with the, uh, book any company Lisbon for 50 guests.
- 49:38
And if you remember, this only can book for, uh, uh, less than 10 guests. And of course, it's blocking. Now, let's go to the agent control new agent.
- 49:54
Here we have two key importance or two key imports that are important for the agent control SDK. So we have the agent control plugin that captures agent events and sends them to the agent control server.
- 50:11
And we have the agent control steering handler that listen for a steering decision for the server and delivers them back to the model.
- 50:21
Together here, they are what connects Strands to the agent control steering logic. So
- 50:31
let's go this with this. So the steering agent
- 50:39
is going to... Uh, we book a room for any company Lisbon, uh, 50 guests. Let's see what it's doing. Yeah, I have successful booking for a stay any company Lisbon for 50 guests for May.
- 50:53
So it's the reservation have been split into two rooms. So it took that it itself, you know, one room for one and other room for five, and that's it.
- 51:07
So use hook for hard constraints, agent control for softer rule. Now, we have five technique all running locally, but how do you take this to production without maintaining service, without building infrastructure?
- 51:29
Let me show you how. Everything that I just built here in the previous demos runs locally. For the production version, Amazon Bedro- Bedrock... Amazon Bedrock Agent Core give you the runtime, a gateway, a short-term and a long-term memory, a CloudWatch observability
- 51:54
built-in, no servers to manage. Here the architecture. The Strands Agent runs inside the runtime, and you know, inside the runtime you can put every framework that you want, not only, uh, run, uh, Strands.
- 52:10
The gateway routes tools calls to the Lambda function automatically as a tools. And the steering rules from the previous demo live in the DynamoDB. So you change them there, and they are live on the next call.
- 52:29
And you don't need to deploy anything. And you want to use Neo4j? Of course, you can use Neo4j AuraDB. That is a external graph database. We also has a free tier.
- 52:44
And the code is in the repo here.
- 52:51
It's in the repo. And you will need AWS credential if you're using Amazon Bedro- Bedrock Agent Core. And in the resource link, there is some credits. I hope that you can find it because I always try to h- to give away some credit for AWS, so you can deploy everything for free.
- 53:15
And, uh, if you are new here in AWS, I have a repository that you can use to deploys everything of this architecture, uh, using a notebook too. But if you are familiar with CDK, Cloud Developer Kit, you can do it that as well to deploy everything at once.
- 53:36
Uh, both option are in the repository, of course, and you can go deeper in Agent Core with all the documentation that I left there.
- 53:45
So please don't stop at the demo and try going to production with Amazon Bedrock Agent Core.
- 53:52
So let me bring it all back. Tokens waste on every request. You can fix it with semantic tool selection.
- 54:04
And confident answer, it's never computed. Possible... It's possible because you are asking how many. So you can use GraphRAG and query the data. Don't sample it. Sometimes we have fabricated success confirmation.
- 54:22
You can use multi-agent validation and second pass corrected. And rules the models are query skipped. You can use, uh, neuro-symbolic guardians to enforce in the code.
- 54:39
Don't trust in the prompt. And for the hard blocks that stop the user, you can use runtime steering, self-correct, and finish.
- 54:51
Each demo that I just showed you is in the, in the repository as a notebook and an application as well. Start with the demo one, then go to the demo five, and if you want, deploy it into productions.
- 55:07
So have you tried any of this in your own agents?
- 55:12
Thank you to join me in this session, and happy building.