AI Engineer World's Fair 2025
Building Agents with Amazon Nova Act and MCP — Du’An Lightfoot and Banjo Obayomi
About this talk
Du’An Lightfoot and Banjo Obayomi lead a hands-on AWS workshop on building autonomous browser agents with Amazon Nova Act, Model Context Protocol servers, and Strands Agents. They explain AWS agent-building options, provision participant environments, configure Nova Act API access and headless browser execution, discuss integration with Amazon Bedrock, and demonstrate multi-step Google Maps searches and MCP-based PowerPoint generation while answering audience questions.
Chapters
- 0:00Introductions and AWS agent-building concepts
- 9:42Workshop setup, Strands Agents, Nova Act API access, and headless execution
- 22:23Agent limitations, website policies, credentials, and browser integrations
- 45:18Local MCP servers, AWS Lambda, and the Strands–Bedrock relationship
- 1:03:00Interactive browser automation and Google Maps agent demonstration
- 1:25:05AWS region discussion and MCP-based PowerPoint generation
Talk transcript
- 0:00
[ on-hold music] Building agents with Amazon Nova Act and MCP. [sighs]
- 0:23
I'm excited today because we're gonna build intelligent, autonomous AI systems that can help you build, scale, and improve your applications and business. My name is Du’An Lightfoot, and I'm joined by
- 0:43
Oh, you gotta swap it. Yeah.
- 0:46
Swap.
- 0:47
Yeah. [laughs]
- 0:51
There we go.
- 0:52
There you go.
- 0:53
My name is Du’An Lightfoot, and I'm joined by
- 0:56
Hey, I'm Banjo Obayomi. I'm a solutions architect here at AWS.
- 1:01
Now, this is the AI engineer warfare, and I've been in tech over fifteen years, and right now is the most exciting time for me in my entire career. And one of the reasons for this excitement is agents.
- 1:19
How many of you right now are bui-building agentic systems?
- 1:24
Uh.
- 1:25
I love it. [laughs] So when we talk about agentic AI, I think it's important that we level set from an AWS perspective. There are three key terms we need to think about.
- 1:39
First, the ability to plan. An agent gets a prompt, it gets an objective, and it determines the actions that need to be taken. So it creates the plan, and then it takes actions on those actions by using things like tools.
- 1:56
Now, the last piece, the third piece, and probably the most interesting, is the reasoning, where the agent is able to evaluate the results and determine if it needs to update the plan and take additional actions until the objective is complete.
- 2:16
This is an agent. Now, when we actually break down the architecture,
- 2:23
I think it's important to take a look at this because we have the user input, we have the agentic system, we have the possibility of some type of human in the loop, and then we have the generated response.
- 2:36
Now, when we dive a little deeper, there's some components of this agentic system. We have the LLM.
- 2:44
We have a knowledge base with external information that we wanna, may wanna provide. We have guardrails to say to the model, "Don't do this," or to ground the model with the truth from our knowledge base to say, "Okay, is this actually relevant information?
- 2:59
Is this accurate to the information we're receiving from the knowledge base?" And then we have access to additional tools, memory, or we may need to talk to additional agents or LLMs like Amazon Nova Act through something like MCP.
- 3:14
And we have the ability to design our own flows for these systems.
- 3:19
Now, the most interesting piece that I think a lot of us are probably focused on when we're building these systems is around the continuous evaluation framework. Like, how do we know if we're using the right LLM?
- 3:32
How do we know if our prompt is consistent, accurate, or even optimized for the performance we're expecting? And then how do we even judge our system? How do we rate that and determine that, that it's actually solving the problems that we need or intend?
- 3:47
Now, once we have this, we need to log this information and then have some type of subject matter expert and determine how can we improve this system, and this is the iterative approach.
- 3:57
So we're always trying to improve and optimize our agentic system.
- 4:04
Now, continuing on this, continuing on with this story. Now, there are some use cases that we should be building these systems for. Like, if it's complex tasks and we don't know which tools should be used, how many tools should be used, and we want the model to leverage its reasoning capabilities, well, this is a great use case
- 4:21
for agentic system. But if it's something that is just one step, our traditional if this, then that approach is probably the best solution, right? We don't always need to provide some type of agentic system for something that can be done with a traditional solution.
- 4:40
Now, when we talk about agents on AWS, there are three approaches and perspectives we should think about. First is gonna be the specialized, using something like Amazon Q. How many of you are, have used Amazon Q?
- 4:54
The, there's Amazon Q in the console to help solve your problems on AWS. In the console, there's Amazon Q developer inside of your IDE. And right now, one that I'm, I would think most excited about is Amazon Q CLI agent.
- 5:08
How many of you have used that?
- 5:10
Whoo.
- 5:13
For me, if you, if you are into increasing your productivity, using a CI, a CLI agent has helped me tremendously. You -- From editing a video, it can do that.
- 5:26
Summarizing a document, reading my ent-entire code base. Like today, for one of my demos, I had some code and I was trying to figure out why wasn't it working.
- 5:35
I said, "Analyze this code and tell me what you see. Let me know the APIs that it's calling." Well, I looked up the APIs. Well, it didn't match my APIs in the API gateway.
- 5:44
So when the code was deployed, it didn't, wasn't deployed with the right APIs. So the agent was able to help me save a ton of time by just analyzing the code and telling me what it saw, because I'd never seen the code before, right?
- 5:55
So that's what these tools are able to, um, to help us do. The next is fully managed. If you're using Amazon Bedrock, you're able to leverage Amazon Bedrock agents to build and manage agents inside of AWS.
- 6:07
And today, what we're gonna be focused on is the DIY, the do-it-yourself approach by using Strands Agents. This is-- allows you to not just leverage Amazon Bedrock, but also leverage models through other providers using Lite LLM.
- 6:25
Now, when we talk about Strands Agents, Strands Agent was, um, announced about a month ago. I wanna say something about a month ago. This is open source, extremely lightweight.
- 6:35
So if you use other agent fla- frameworks, it's like that. But the implementation is , you'll see in the code how easy it is to build a agentic system or a, a agent itself in a few lines of code and already get started.
- 6:49
I built a multi-agent solution in about under fifty lines of code.
- 6:56
And so when we break down Strands Agents, there are three components. We have a prompt, we have a LLM, and we have tools. So you create a function called, let's say, a get weather tool, right?
- 7:08
You define your agent, you give it a prompt, and it's already implemented. And you'll see in the code as Banjo goes through it here in a moment.
- 7:19
Now, taking it a step further, as Danielle presented today on Amazon Nova Act, these models are able to do some really cool things, and this is another thing that I'm excited about.
- 7:30
Amazon Nova Act is a research preview model, and the capabilities of this allows you to use a prompt or give instructions and take complex tasks and do things like browse the Internet to find research or to research or to search on amazon.com to find the top list of widgets, right, and then
- 7:55
return them and then add them to your cart. So you'll see how we can leverage this, not just using the SDK for Amazon Nova Act, but also by leveraging MCP.
- 8:05
Which leads us into the last piece, which I think
- 8:09
when we're talking about agents, I don't think we would be here today as fast as we have moved if it wasn't for MCP. How many of you are leveraging MCP today?
- 8:19
Model Context Protocol. How many of you have built your own MCP servers?
- 8:25
I built several. Um, I got two that I use all the time. One, how many of you use Obsidian?
- 8:33
Okay, so for my documentation, I built a Obsidian MCP server. This allows me to save all my documents, reference all my documents, and just my entire workflow is streamlined because of this MCP server I use right there.
- 8:49
But I also use one for my bookmarks. I built a bookmark manager because every Friday, I'm restarting my computer, and I lose my bookmarks. I save them and I forget about them.
- 8:58
But now I can just say, "Save this bookmark." It gives it a description, gives it a title, give it a date, and I can even add notes so I can remember where this bookmark.
- 9:07
So now when I open up QCLI, I can say, "Hey, I'm looking on a topic. I'm looking for, um, some information on MCP. Can you tell me all the bookmarks that I have?"
- 9:14
And then it'll find it. Can you tell me the ones I saved last week? And so these, this is the power that we have today. But with that being said, I think it's time that we all start building.
- 9:25
Banjo's gonna take over, but if you, you open your laptops and log on to this link, this is gonna take you to a workshop environment where you have access to an Amazon account where Banjo is gonna walk you through building out today's workshop.
- 9:41
I thank you for your time.
- 9:42
Cool. All right, so, uh, this is gonna be a hands-on workshop, so we've provisioned an AWS account for everybody here, so you don't have to install anything on your computer.
- 9:52
Everything is g- gonna be done through the browser. And I always say the hardest part of the workshop is just getting started, so some of my colleagues are also here.
- 9:59
So raise your hand, AWS folks that are here to support. Uh, so we're gonna take some time to just get logged into an environment. We're gonna set up a VS Code server, enable models, uh, get the Nova Act API key.
- 10:10
So again, this is the hardest part of the workshop, just getting started. Uh, so let's take some time to just get into the environment, and I'll follow along as well.
- 10:19
So, and this is the-- Again, everything is... You don't have to install anything on your computer. You don't have to use your own AWS account. Everything is provisioned for you.
- 10:26
So but while that's loading, I'm gonna just briefly walk through the three modules of the workshop. Uh, so the workshop is really about how you can use Nova Act.
- 10:34
So the first module is just getting started with Nova Act. We're gonna make an API call for that. The second part of the module is gonna make an MCP server that can leverage, uh, Nova Act.
- 10:43
And then finally, we're gonna use the Strands Agent to cook everything together. So that's kind of the, the three steps we'll go through at this workshop. And all the code is available via the link on GitHub, so you can try it out on your own as well.
- 10:55
Uh, but yeah, trying to get started here. If you can't follow along, I'm gonna be doing it up here, so don't worry too much. And again, all the code is available, so you can try it out line, uh, offline.
- 11:06
Okay. So the first things first, uh, if you're following along, make sure to click this Open AWS Console button. Again, we've provisioned a AWS account. You know, don't log into your own AWS account.
- 11:17
Don't try to create a new one. Everything is, uh, pre-visioned here already. So I'm gonna click, clicking that button to open up your AWS account.
- 11:26
So logged into my AWS account. Uh. So the first thing we do in a AWS account, we're gonna enable, uh, Amazon Bedrock models. So Amazon Bedrock, think of as a serverless API to access different foundation models, and you can build lots of, uh, generative AI applications in it.
- 11:47
So it has capabilities like knowledge bases, guardrails. You can build agents on top of it for anything you need to build, uh, AI agents or generative AI applications, Amazon Bedrock has capabilities for that.
- 11:59
But for this workshop, we're just gonna enable specific models. So I'm gonna enable specific models. You can click the Amazon models, and then we'll use the Claude 3.7, uh, 3.5 Haiku, and 3.5 Sonnet.
- 12:14
So those are the ones we're gonna use for this workshop.
- 12:18
And that's gonna request access there. And again, all the instructions are also in this workshop as well. Uh, so we could follow along, but I'm just gonna go through it just for sake of time.
- 12:35
And then the next part, once we get the model access, there's a VS Code server that has everything set up already. So I'm just gonna go in there.
- 12:47
And if the URL and password is there, you can log into your VS Code server with everything installed.
- 13:06
And I'm also gonna log into Amazon Q. So Amazon Q is our, uh, ID extension to help you write code.
- 13:13
Uh, if we have time, you can sign up to a Builder ID completely free. You don't need, uh, AWS account. You don't need to put in your credit card.
- 13:21
You can just, uh, log in through there. I already have an account, so just s- it feed it up, but it puts a nice little, uh, AI agent there.
- 13:29
It can ask questions, update code, et cetera. So it's, uh... I'll show you some examples. I'll just go through some of the code.
- 13:36
So, so who's gotten to this point, setting up all the models, workshops? 'Cause this is-- Once you get all of this done, then that's when the real fun begins.
- 13:46
So just getting a pulse if I need to slow down or-
- 13:51
Slow down.
- 13:51
Slow down a bit. Okay, I'll wait a bit. Again, raise your hand if you're stuck anywhere, questions. We have, uh, agents that can come around and support you. So I'm gonna just pause for a little bit.
- 14:04
Any general questions while w- we're waiting? [muffled speech] Oh, yeah. So this workshop, uh, again, all the code is available online. Uh, this workshop available as well, so you can also look through that.
- 14:18
There's a website called workshops.aws. And when you go there, you can do something like, uh, Nova Act, and then it's the only workshop that shows up. So you can always go to workshop.aws, just search, uh, Nova Act, and this workshop will show up.
- 14:34
So you can see all the instructions, all the code, and run this, uh, on your own.
- 14:54
Okay. And then last thing, uh, 'cause we're gonna use Nova Act, we actually need to get a Nova Act API key. So if you go to novaamazon.com, it-- This is a website that you can use the Amazon Nova model.
- 15:07
So you can do like chatting, generating images, uh, speaking with Nova, uh, generate videos. But then also, this is where the, the Act, uh, API key is generated. So if you're following along and you wanna, uh, generate your key, again, it's free to log in.
- 15:23
You can use your amazon.com, uh, like when you order something on amazon.com account to log into this, and then you can just generate a key here, and it'll be able to a-access that.
- 15:49
Oops. Okay. So I'm gonna walk through what module one is. Uh, before I get in, has everybody gotten in here? Just quick pulse check. If not, you know, I'll continue.
- 16:00
I know the Wi-Fi is slow, so it might be hard, so I'll just continue on. Uh, but yeah, the first one we got to see how Nova Act works, uh, how, how the actual code looks like, uh, generated the key, uh, need to export the key, and then kind of running the first script, which is actually gonna
- 16:16
open amazon.com, uh, and we're actually going to look for the first coffee maker. So let me see how that, uh, code looks like. Let's go here.
- 16:27
Make this bigger. Oops. So very simple code. Uh, with Nova Act, it's again, it's all on Python SDK. I-- So I decide what a page to go to.
- 16:44
So go to amazon.com. I say, "I want you to search for a coffee maker." I say, "Select the first result." And I say, "Get the title of that product page."
- 16:52
So, uh, very simple. If you've ever done kind of web automation before of something like, uh, Selenium or Playwright, you probably have to like look for this div tag, you know, look at this h1 tag, grab this information.
- 17:03
A lot of manual processes of actually inspecting the actual website. Here, I'm just saying, "Click the search bar, find something." Like I don't have to specify click this tag, do that.
- 17:12
So it makes it much more easier to engage with the website as a natural human would, instead of like looking through divs and trying to find, uh, this p tag specifically.
- 17:21
So, uh, this is a great way to just, uh, you know, use Nova Act right out of the box. So I'm going to, uh, run this so I can see.
- 17:35
Examples. All right. All right, so added my key. Go over what happens when I run this file.
- 17:59
Give it a second. Oops, we failed. All right, let's start over. [muffled speech]
- 18:09
Straight one. Ah, okay. I know it. You gotta run out of this.
- 18:24
Let's start that over. Yeah, question?
- 18:37
Um, so just explain why is Banjo running that command? It's running F- Xvfb. It's a frame buffer where it runs your X11 system.
- 18:46
Yes.
- 18:47
What happens there, uh, Nova Act actually goes and clicks a mouse on a browser. That's why it needs to be run like that, otherwise it has no GUI. So this is just kind of a way to emulate, um, um, a graphical user interface on, on this Linux box.
- 18:59
Thank you, Darko. Yeah. Since we're running everything in the, the cloud on a browser, I'm saying, you know, open a browser again, but it's already in a browser, so that's why it crashed, so I had to put that, uh, frame buffer command.
- 19:10
Uh, and yeah, the workshop kinda walks through why we did that. But you can see, uh, what is going on when Nova Act says, "I'm gonna search for a coffee maker.
- 19:18
I'm at the Amazon homepage. My task is to search for this." So it's understanding what it's doing. I see the search bar has coffee maker. I'm at the search bar here, and now it actually, uh, puts the, the actual log of the actual HTML file, so it's taking screenshots.
- 19:32
You can see what it looks like. It got the first results. I'm on the coffee maker page. It selected it, and now we got the title of the-- Now it says, you know, "What's the title of this product page?"
- 19:47
All right, got this Black Decker 12 coffee maker. My task is to return the title of the product page. Product title, it got that, and it ended the session.
- 19:56
And then it also creates a, a video log that I can actually look at to see what it did for each-- for everything it did in this WebM file.
- 20:05
So yeah, question?
- 20:07
So does it reason about what it's looking, uh, in terms of pixels, or
- 20:15
does it just analyze the HTML?
- 20:17
Yeah. Can you repeat that? There's gonna be a microphone. Yeah.
- 20:21
Thank you.
- 20:22
So the question was, does it reason about the page in terms of pixels or in terms of text?
- 20:31
Yeah. So it's actually looking through the actual, uh, the page itself. So you see in this video, it sees-- It looks at the page, it can see what's in the page.
- 20:39
So it's, it's a large language model trained, so it can actually see the actual-- the page is doing. So it's not looking at, like, like, the H1 tag or whatnot.
- 20:47
It understands the context of that particular page. It can see that's a search box. Okay, I'm gonna go click to that search box. So yes, it understands the pixel level of what's on that actual page.
- 20:58
So this is kind of the video. Uh, it's hard to see, so we'll make it bigger.
- 21:03
Sped up. Uh, so it opened the page. It goes to the-- It's able to type in coffee maker there. Um, it gets that information, clicks the button. So even with all the ads and everything, the video, it can understand the, the task, clicks that, and it gets the information back.
- 21:20
So that's-- And that was couple lines of code, so you can extrapolate to other type of workflows you can do for searching through things.
- 21:28
Sorry, I have a-
- 21:29
Another question. Yeah.
- 21:30
Yeah. So when you-- What I've experienced with these kind of frameworks is that when you run this on a server environment, um, services like Cloudflare will block the access and maybe do a CAPTCHA challenge.
- 21:42
How do we solve that using Q?
- 21:43
Yeah. So with, uh, so using Amazon Nova Act, so it doesn't do CAPTCHAs, it doesn't do things of that nature. So it's, it's meant for, like, workflows to understand.
- 21:51
But yes, it's not gonna bypass CAPTCHAs and other things of that nature as well. So it's made for, like, going to amazon.com or looking through a booking site. But if something that, like, requires, like, a human, or it wants a-- It do not-- You can't bypass that.
- 22:04
How would you personalize-
- 22:06
You, you, you wouldn't use Nova Act for that use case. If you need to pass a CAPTCHA or something else, I'd use another technology. This is not meant to, like, overtake humans, you know?
- 22:14
It's more like I'm helping them augment things, but not... If there's a CAPTCHA involved, then you have to use a different technology for that.
- 22:20
It's also a preview. It's also a preview.
- 22:23
It's-- Yeah. So this is a research preview as well, so if that's a very u- good use case, you know, leave feedback on the Nova, the website. So yeah.
- 22:30
Is, is human-in-loop possible at all with it yet?
- 22:35
Well, this, this one, it's... No, 'cause I'm writing all the code here. So, but again, this is Python code, so I could probably put in something here like, you know, ask something, make an API call here.
- 22:45
So this is, you know, it's a Python code. So you might be able to create some type of, uh, workflow that might augment, like, wait for a human response or, or whatnot.
- 22:53
Because the browser's happening in, like, headless mode, but could you make it work with a browser the human is also seeing at the same time?
- 23:00
Yes. Yeah.
- 23:00
So it can pause and wait for somebody to put in, like, a password or credentials or do a CAPTCHA-
- 23:05
Yes
- 23:05
... and then once it receives that works, continue on the workflow?
- 23:08
You could do, yeah. Because the-- Right now I, I ran it in headless mode, but yes, it can also run, uh, you know, it would open up the browser.
- 23:13
If I ran this on my MacBook, it would open up a Chrome browser and go through that session.
- 23:18
Also, if you're running it, and you want it to bypass something that has two-factor, if you're already logged into, say, amazon.com, and then you run a code, it's gonna use your credentials in that browser session to continue on to perform that task.
- 23:33
So that's something that you can do as well.
- 23:36
Cool. So let me... Oops. And then one other thing, you can also do multi, uh, you know, parallel execution. So my last-- my next example is actually I'm trying to find multiple monitors, and I want to compare them all at once.
- 23:53
So I'll show you how that code looks like. Um, open.
- 24:01
Whoop. So I can check for the monitor, extract information. I'm setting, you know, I want-- I'm defining what I want. So again, I'm, you know, saying I want to find the price, the rating, the size.
- 24:14
Uh, go to amazon.com. Uh, I set it headless mode this time, so I don't need to do the frame buffer. Uh, I start multiple threads. It looks for each monitor simultaneously, because each of these are individual tasks, so it can parallelize them instead of waiting it to, to go through.
- 24:30
I, I define the list of monitors I wanna go through, start the thread, and then it starts executing and finds the, the results of the monitors. So I can run that in the background.
- 24:44
So starting the three parallel threads, and it's-- Oh, so again, running in headless mode, so it's gonna be able to do this in the background, but we can see kind of what the model is thinking, how it navigates through the webpage.
- 24:55
Banjo, question for you.
- 24:56
Yep.
- 24:58
I, I, I have tried to use Nova, uh, in the past April, and, uh, it worked for the first time, but once I did it again, it triggered the CAPTCHA.
- 25:09
Is this something that has been already resolved, or is this happening? Because I think the website, and it was Amazon in this case, it was detecting it was a bot.
- 25:18
And, uh, is there like an llms.txt or robots.txt that can declare that this has been-
- 25:23
So Nova Act, there is a GitHub repo, so you could go there and just grab that. But it's, it's working now. Like, I'm running it, you know, I just-- I-- This is just live code I'm doing right now.
- 25:32
Like, uh, I just exported my API key, started running it. So, uh, you can try it in the workshop. Uh, but yeah, I mean, it's, it's ready to go.
- 25:40
We're building right now. [chuckles] And you can kinda see that it's going on in the background, what it's doing. Uh, I've looked at this monitor, the Dell monitor. I'm at the Amazon homepage.
- 25:51
It's like it's going through, looking through the search result. It's saving things. So you can see it's running in parallel. It got the information for the, one of the first ones.
- 25:59
So it's, it's going as, uh, I just set that up, and it can execute that. So if you have some type of, uh, I don't know, like daily news thing, you need to go to the website and get news or something and, like, have a report, Nova Act.
- 26:10
And there's no API for that. This is one way you can encodify how to do that kind of search and get the information.
- 26:17
You have a question?
- 26:18
Um, yeah. I'm wondering, so h-how successful is this in terms of, like, more ambiguous tasks? 'Cause I, I ran the Amazon demo, and that worked.
- 26:25
Cool.
- 26:25
But I'm wondering, could I just add Google there and it-
- 26:27
Sure
- 26:28
... and, and how, like, how vague and, and sort of how much does it know when it's navigating through? Like, I was thinking, like, if I wanted to, to return a pair of sunglasses that broke-
- 26:38
Mm-hmm
- 26:38
... would, would I, would I be able to just say, like, start in Google and then find this company's website, find a way to, you know, engage support, open a ticket?
- 26:47
Like, h-how much sort of-- Like, how vague can you be and how smart is it currently? W-Would that, would that, like-
- 26:54
Yeah, I mean, the more instruction you give, obviously the better, but it's able to understand how to navigate a website. That's what the model's trained on. So if you say, you know, "Go to this sunglasses website," it doesn't-- it probably wasn't trained on a specific sunglass website, but it can understand that button is support, you know?
- 27:07
This button is click a ticket. So it understands kind of the, the general knowledge of how to navigate the website, but if there's something very intricate about that website, you're gonna have to encode it in the text.
- 27:16
Like, make sure you click find an expert or whatever, so it understands how to navigate websites.
- 27:22
Got it. And does it understand when it's failed?
- 27:24
Yeah, someti- sometimes. [chuckles] I've seen it sometimes get stuck in a loop and like, "Oh no, I keep scrolling, I keep scrolling, I keep scrolling," and it doesn't know when to stop.
- 27:32
So it-- Again, this is in research preview, so things are getting better. The model's getting updated behind the scenes. But it's not like, it's not AGI. [chuckles] So that's...
- 27:40
Got it. And one last question. Um, how, how is it in terms of navigating, like, uh, distrustful parts of the internet? I mean, uh, there's a lot on the internet that we see and we know is not to be trusted or it's something not to be followed.
- 27:52
Sure.
- 27:52
How have you sort of worked around that problem?
- 27:54
Yeah, 'cause again, it is a model in the background, so it's gotta understand, like, if you're doing some typing, it's not gonna wanna click that or it might be s-- there are safeguards in place.
- 28:02
So that's built into the model. But again, uh, it is in research preview. You still have to explicitly say what buttons to press for certain actions. But again, the model, it is an LLM train.
- 28:12
It's gonna be able to understand the, the nuances and say if it can't take this action or can't do that, that, that could happen. But I haven't seen that use case, but if you keep pushing it, maybe you'll find those, those things.
- 28:22
Well, the thing I had in my mind is, like, if you go to a, a site where you have to download a link, sometimes there's an ad that says-
- 28:28
Sure
- 28:28
... "Download a link," and you know that that's just an ad trying to get your attention.
- 28:31
Of course.
- 28:32
Would the model know, or i-is that some, is that some present problem?
- 28:35
Yeah, if like, for example, like in the, the amazon.com, it shows an ad for something. When I said find the first thing, it was able to scroll past that ad and click something.
- 28:42
So the model understands the task you give it. So yes, it can understand that.
- 28:46
Thank you.
- 28:49
All right, so this, this just finished. Yeah, that's really quick. It showed it, it got-- It was able to find all the models given the size, the rating, the price range of the monitors.
- 28:57
So again, it, it executed that all in parallel. It got me the nice information, and that, that's kind of the idea of, like, it can do parallel execution in the background, so you don't have to wait for it, and you don't, don't see it actually clicking through the, the task, and you get your information.
- 29:11
Banjo, one more.
- 29:11
All right, one more question, then we'll move on to the MCP part.
- 29:15
So Nova is specifically meant to be used with a browser, correct?
- 29:20
Uh, so Nova Act. So Amazon Nova is a family of models on Amazon. So if you go to this website, novaamazon.com, uh, you see there are different foundation models like Nova Pro, Premier, Lite, Micro.
- 29:33
These are like the text understanding models, so like your typical LLM calls. There's also an image model called Nova Canvas to generate images. There's a video model called Nova Video, which can generate, uh, videos.
- 29:46
Uh, and then it's also a speech model, text, uh, speech-to-speech called Nova Sonic. So Nova is a foundation of found, uh, f-foundation models by Amazon to do all these type of tasks, and Act is just another one for browser automation.
- 29:59
Are there plans to expand this, like, beyond the browser so that we can someday take actions in Slack or IDE or anything outside of the browser?
- 30:11
Maybe. Some of the team is here, so maybe talk with them later. [chuckles]
- 30:15
All right, thank you.
- 30:18
All right, so I'm gonna move on to the MCP part.
- 30:22
Banjo.
- 30:22
Yep.
- 30:23
Nova Act is only available in US?
- 30:26
Yes. Right now, Nova Act is only available in the US. It's in preview, so it's just getting started. So if you log in from, like, a different, uh, account, like address, like UK or something, it might not, it won't work.
- 30:37
So it only works in the US at the moment. Yes.
- 30:42
All right. One more question over there, and then I'm gonna move on.
- 30:53
Oh. [laughs] Your amazon.com is different. I don't know, yeah, because it is opening up a different browser, so it, it could have clicked something differently. Yeah, so- Could be an employee discount.
- 31:07
Yeah. [laughs] That's right. [laughs]
- 31:08
We can actually look at the video preview or video playback to see what your results were like.
- 31:14
Yeah. One more? Oh. One more. Okay. One more quick.
- 31:18
Quick, quick one.
- 31:19
Yeah.
- 31:19
Are there plans to support persisting browsing data such as cookies in the cloud browser?
- 31:26
So when it-- Right now it's opening up its own, uh, browser, but you can also set, like, your own, like, Chromium profile and open up that browser. So everything you have saved there, like if you wanna log into your stuff, you can set your own custom browser, but by default it opens up a new, like, completely new
- 31:40
browser without anything saved. All right. So I wanna show, uh, I actually made an MCP server for Nova Act. So a module tool is going through, uh, MCP, and I can kind of show you what I did for the MCP server.
- 31:57
Uh, in fact, we can use Amazon Q here. So I'm gonna ask it, uh:
- 32:02
Can you tell me about the Nova Act MCP server?
- 32:14
Can you tell me what it does? What it does and... Oops.
- 32:23
So tell me about the Nova Act MCP server.
- 32:27
So you can see it's going through, um, integrates with Nova Act browser at MCP. It has a browse session tool, browser action, execute parallel tasks, take screenshots, close browser, list results.
- 32:38
So I created these different a-aspects of the MCP server so I could use something like Claude Desktop or Cursor or Amazon Q CLI to just say, you know, "Open amazon.com and find information for me."
- 32:49
So it's, it's portable, it understands, uh, so I don't have to actually write code. I can just say, "Go to amazon.com and find me the co- the first coffee maker."
- 32:58
It will actually write all that code I did in the initial one to do that or the multi-monitor. So I wrote a, wrote a bunch of code to do this.
- 33:04
If I just said, you know, "Well, get me these three monitors and get the price," it would actually write all the Nova Act code it needs to do that using the MCP server.
- 33:12
So that's kind of the power of MCP, that I just describe a task and then I can-- it will encode the actual browse action things it needs to. So, and then I also made an MCP client that can actually in-interpret that.
- 33:24
So... Oops. Uh, it connects to the MCP server, it runs the, the code, and it's able to use Query Bedrock. Um, I am using a model, so I'm using Claude 3.5 Sonnet here because I'm us- as an MCP client, it needs to have an LLM, um, behind that, and then it's able to, you know, understand which tools
- 33:42
to use, uh, run the code and open up the browser and whatnot. So let me just run the example here.
- 33:50
So module two. So we are... Open the file. Just did that.
- 34:01
We ask Amazon Q to explain the file to us, and now we're actually gonna run it. So Python three, and then I can open this up.
- 34:24
Okay. So let's pr- be adventurous. So somebody give me a query to try since [laughs]
- 34:29
anyone has an idea. I'm gonna just ask it and it'll do something. So someone give me an idea of what to run on Nova Act. [laughs] Fix Wi-Fi? [laughs]
- 34:39
Uh, how would you fix the... [laughs] Can you find a website to fi- fix website? Can- Find
- 34:48
one. Find... Let's see. Website to fix Wi-Fi. Use headless mode.
- 34:58
Oh, that's better, huh? But let's see. All right. It goes to google.com, "How to fix Wi-Fi problems, troubleshooting guide," in the box and press Enter. Return a list of the websites, title, descriptions.
- 35:15
All right. It's going through that. So it opened google.com. Uh, how to fix Wi-Fi problems. I see an empty search bar where I can type queries for search information.
- 35:26
I should type how to fix Wi-Fi problems. So you can see it's understanding what to do. It'll-- Oh, it hit a reCAPTCHA page. So okay. [laughs] The search results are not available, blah, blah.
- 35:35
So, so see, it looks like it got stuck on a reCAPTCHA page. So this is like a headless agent. So if someone asked a question about can it pass CAPTCHAs or whatnot, you see that it's, it got stuck doing that.
- 35:47
It looks like it's stuck in a loop now, so it, it keeps-- sees the CAPTCHA again, so... I should skip the clip button to skip the CAPTCHA window. The CAPTCHA is still open, so it, it's probably gonna be stuck here unless I close it.
- 35:58
So you can see there are limitations. It's not gonna pass CAPTCHAs and whatnot, but that, that was a good query to show that it... Oh, did it fill it?
- 36:06
Uh, it's still open, so it's gonna be stuck here, so I'm just gonna close it out. But you can see, you know, it, it can't pass everything. It can't navigate the website.
- 36:13
So something like that w-will, will, will not work. So that was a great test example to show.
- 36:20
If I use the, the baked-in one, you know, find that coffee maker under fifty dollars, it'll be able to go through that and use headless mode. But any questions on that, seeing how the MCP server is working?
- 36:30
I didn't have to write code. I just said, "Do something," it actually wrote the code to, to do it for me.
- 36:36
Any questions? Question over here. Are you able to bring it into the browser and finish the CAPTCHA and let it keep going? Yeah, yeah. So a question about if I can actually go into the browser and do it myself.
- 36:45
Yeah. If I ran this locally on my machine, it will actually be able to, it'll open up the browser, and I can actually click the button, and it'll continue doing that.
- 36:53
Right now I'm running it within the browser, so [laughs] I'm dev-doing, dev-doing everything in headless mode, so we can't interact with that.
- 37:02
So you can see it's able to find search under $50. It can actually look at the website. It's found, uh, search results on amazon.com. So yeah, so that, for that use case where we're not passing CAPTCHAs, it's able to con-continue and find the information there.
- 37:16
Can it actually order something for you?
- 37:20
So a question about can it actually order something. If I use my own browser session and, like, logged in at my ba- amazon.com account and said, "Yes, order this for me," you know, click through, it'll be able to understand that thing, but I would have to put in a info- I would have to use my own browser
- 37:34
session. So I c- I wouldn't want it to log in by myself. So yeah.
- 37:44
Another question? Or a question here.
- 37:48
If you give Nova Act the authentication for Amazon, for example, like you give it your login details, then can it log in and complete that action for you?
- 37:56
Yeah, if I, if I like say, "This is my username, this is my password, enter that into that field," it, you know, will be able to understand, you know, this is a sign-in button and I have this information.
- 38:05
But again, this is all Python code, so yeah, you can encode it, you can make it an environment variable so it won't read it directly. So a lot of ways to do that.
- 38:12
Does it also, like, understand 2FA? Let's say it asks you to go to your Gmail and you... Will it then open the Gmail website, check the email if you're logged in again on your session, and then input it, or is it not that-
- 38:23
And you can-- Well, if there's no CAPTCHA, like we just saw with the CAPTCHA, yeah. So there's no nothing blocking, so... But yeah, again, Nova Act is free to use.
- 38:30
There's a lot of creativity in this room, so I think we should have like a Nova Act hackathon. I think that'll be [chuckles] You know, do something crazy with Nova Act. [laughs]
- 38:41
All right. So-
- 38:43
One more question.
- 38:44
Yep, one more.
- 38:45
Uh, can I book a flight when my price alert is less than $100? [laughs] It's like a continuously check.
- 38:52
You probably use something else for that. But yeah, I mean, Nova Act can open up that website. It can just have a query every day, you know, open Google Flights and look at the quickest thing, and if something is below this threshold, you know, send me an email.
- 39:03
So again, this is all a Python script. So you can set up something that triggers, like, once a day, like in a Lambda function. And so yes, totally possible.
- 39:11
So Nova Act is very flexible, and because it can run in headless mode, you don't need to have that UI. So that's really what makes it helpful for interacting with websites that don't have a native API.
- 39:29
Thanks. Yeah, this is pretty cool. I'm a little bit confused, like, we have the Nova SDK, SDK API key-
- 39:37
Mm-hmm
- 39:37
... and we are also doing some stuff in Bedrock.
- 39:40
Ah, yeah. So-
- 39:41
So how does this actually work?
- 39:42
Yeah, yeah, yeah. So in the, the [lips smack] Nova API key is separate, but for this MCP client I did, it actually needs a large language model to understand what's still happening.
- 39:53
So if I go to Claude. Oops. I actually said I'm actually using, uh, Claude Sonnet 3.5, uh, for my MCP server. So that's how... Because I just asked it, you know, "Find that website for me."
- 40:06
How, how does it know that without any of the code doing that? So it's using a large language model underneath the hood to actually find that information. So that's where we use Bedrock for.
- 40:16
Trying to find it in the code, but
- 40:19
Sonnet. Yeah. I set the model ID. So your assistant, you have an AI assistant helping. You have tools. You're using Claude 3.5 Sonnet. You're making an API call to Bedrock whenever something happens.
- 40:29
So that's where the, the LLM we're using. But Nova Act is separate from that. So this M-- Like, if you're using, you know, Claude Desktop, it's running an LLM in-inside of that to be able to understand that for the MCP server.
- 40:44
A question here. A question here.
- 40:56
Uh, the question is, uh, does it integrate with browser plugins as well? Like, could it integrate with LastPass if you have the LastPass plugin-
- 41:02
Mm-hmm
- 41:02
... fill in the credentials through LastPass, and then continue?
- 41:06
I haven't tried that, but again, it, it do- you can set up to use your own browser. So if you do that and if that's integrated, it might be able to do that and click through that.
- 41:13
But I have not tested that, but something to, to try out.
- 41:16
Thank you.
- 41:17
Okay.
- 41:18
Uh, the biggest problem you would face is two-factor.
- 41:21
Mm-hmm.
- 41:21
Like, even if you gave it a password, like if you're using something like Google Authenticator or something, that would be, like, the biggest problem, a CAPTCHA. But other than that, if you provide it an environmental variable, or if you give it instructions on how to access LastPass in the browser, it should be able to do it.
- 41:38
Right. And, uh, one more question then we'll go on to the last module.
- 41:45
Yeah.
- 41:53
So clearly there are a lot of different, uh, agent architectures that you could use. Um, and what I can imagine using this as, uh, like you have a coordinator agent set up somewhere that's running in the overall app.
- 42:05
And then when something pops up and says, "Hey, you need to go and look this up online, go and check it," uh, it should mod... So my question is, how modular...
- 42:15
I- I mean, it's just Python, so it should be pretty modular, right? Is that the way in which you're imagining the architecture to be? Is just if I was coding a coordinator agent in LangChain or LangGraph, for example, it would then call your sub-agent and get and, and, and run its stuff and then get f- and then
- 42:33
get a text-based output that I throw into my message queue.
- 42:38
Right, right.
- 42:38
That's how it all integrates together. Is that right?
- 42:40
Yeah, that's one way you can do it. So Nova Act, again, it, it... Right, it's just Python, so it could be a tool, it could be an API call.
- 42:45
In the next module, we're actually gonna show you how to actually make an agent from that. So good, good tee-up right here. Uh, so [chuckles] um, Du'An talked about the Strands, uh, at the beginning.
- 42:55
So Strands is a new agentic framework launched by, uh, AWS. So let me open up the link. Uh, it's easy as a pip install strands, and the first agent is, like, agent equals that.
- 43:07
So it's very-- It's a model-first, uh, way of interactive agents. If you used a l- a lot of agent frameworks in the past, there's a lot of bootstrapping and making sure everything is correct and, like...
- 43:18
But that was necessary for kind of the older models. Like, if you think back to, like, like Llama Two, for example, like how, how far models have evolved since then, so.
- 43:26
But now we, we can pass a lot of the, you know, bootstrapping we did previously. The agent can figure that out, so we don't need all these very, uh, heavy ways and like, you know, make sure everything's typed and every-- so whatnot.
- 43:38
So here's a very simple example of how I actually spun up, uh... And also it has MCP native support. So in this example, I actually have two MCP servers.
- 43:49
Uh, I have the AWS documentation and AWS diagrams MCP server. So if you go to this, like, AWS Labs MCP, these are their official, um, AWS MCP servers. And there's a bunch of different ones from, like, uh, cost analysis, uh, Nova canvas, diagramming, CloudFormation, uh, lots of different ones here.
- 44:08
Uh, so again, it's all on GitHub, uh, AWS Labs MCP. But the example I'll do here is I'm actually-- Uh, I made, like, a solutions architect agent. Your role is to help customers understand building on AWS.
- 44:20
And I define these two MCP servers here. Uh, give it a prompt, and I say, "This agent has all the tools in the MCP server. It has a Bedrock model," um, using Claude Haiku here.
- 44:33
And what's cool about Strand, it can also use, like, LiteLLM and Ollama, so it has access to lots of different things, or you can run it locally. And of course, it has access to Amazon Bedrock, so that's what we're using here.
- 44:44
So all those three things makes the agent, the tools, the model, and the system prompt. And then I can say, uh, "Get the documentation for AWS Lambda, and then create a diagram of a website that uses Lambda."
- 44:55
So let me run this code. CD. CD strand.
- 45:18
Okay, so it uses UP to install the MCP server locally. A lot of people, I don't know, where does MCP run? This is running locally, but there are other ways to run it, like in a Lambda function and whatnot.
- 45:29
But for just testing it out, it pulls down the, the MCP server locally and runs it. And you can see it's already executing. So let's make this a bit bigger.
- 45:38
Uh, so it says, "Okay, I'm gonna help you with that. First, I'm gonna search the AWS Lambda documentation, uh, read the documentation, then I'll create a diagram illustrating a static site."
- 45:51
So you can see it does a POST request and do the search. So the MCP server defines where everything is. I don't have to, like, feed it in the well-architected framework.
- 45:59
The AWS documentation is always updated, so it just knows call the search function. It got the Lambda welcome file. It, it put that in. It's able to generate the diagram.
- 46:09
It ac- it generates the diagram and tells us what is going on or how the workflow looks like. It tells me to save the diagram to this location. So I can open it up, generated diagrams.
- 46:20
Oops. And now it's very small. Let me see if I can make this bigger.
- 46:28
Oops. There we go. So it was able to generate the diagram for me. So all through the about, uh, you know, forty lines of code, I have two MCP servers.
- 46:38
I have my prompt and is able to understand that, get that, and just generate something for me with that. Uh, so that's very easy to get started with Strand of building ag- agentic workflows.
- 46:47
I know agent means a lot of different things to different people, but, you know, as long as you have tools, the model, the system prompt, do some type of action, and Strand makes it extremely easy to do that.
- 46:58
If I use other frameworks, it could be a lot more code to do something like that, especially integrating MCP natively like that. I'm gonna pause here for any Strand questions.
- 47:14
There's company.
- 47:18
Um, I know Bedrock already had its kinda agents SDK. So is Strand replacing that, or is this now the pr-- Or is this replacing that, or is it supposed to complement that?
- 47:27
Like, is this the preferred way of creating agents with models in Bedrock?
- 47:31
Yeah. When, when it comes to preferred way, it always comes down to your use case. So the Bedrock agent has a lot more, I guess, opinionated ways to do things.
- 47:39
It's, you can do it through the console. It has, uh, built-in support, uh, right there in AWS, while Strand is more of it's an open source framework. So you can download the code.
- 47:48
You can use other models through that, like LiteLLM or Ollama. If you use Bedrock agent, you can't run that offline. So there's different use cases, different developer tooling. I mean, me as a software engineer, I like, you know, code first doing things.
- 48:00
So it does depend on your use case, what you're trying to do in your experience.
- 48:03
Can, can you show the code real quick? That-
- 48:05
Yeah, yeah. This is the code. Yeah.
- 48:07
Just show the agent. So this is an open source framework. If you go where it says agent, you-- and it says model. Right now we're using a Bedrock model, but you can use another model with LiteLLM.
- 48:19
Yep.
- 48:20
So you don't need AWS at all-
- 48:22
Yeah
- 48:22
... in that instance.
- 48:23
You can use Ollama. You can use OpenAI. You can use-
- 48:26
Right
- 48:27
... Anthropic. So it does matter, yeah.
- 48:33
Yeah. So there's documentation, Anthopic, LiteLLM, uh, lot of different model providers, Ollama, OpenAI. So it's an open source framework, so you can use it whatever you want. So but yeah, that's the idea with Strand open source model agent development kit.
- 48:47
One question.
- 48:49
Suppose I wanna build a text-to-SQL agent, and I have, um, say, fifteen tools already built in that I want this agent to be able to use. If I use this framework, um, how can I make sure- That the agent know when-
- 49:06
Yeah
- 49:06
... to use the right tool and the sequence.
- 49:09
Yeah, great question. Uh, so I didn't-- This example, I have a, a weather agent. So one thing, you said you already have tools. What I like about Strands a lot is I can write a Python function I already have, and let's put this tool decorator, and that's it.
- 49:24
You know, you don't have to put anything else. It understands this is the, uh, what you need to do. And then when I'm going to that agent, I have this tools, and it has put in the, the native tools we're gonna be using.
- 49:35
HTTP request is a, is a standard tool in the Strands framework. So in this example, I'm, like, asking, what is the weather in Seattle? And then also, how many words are in this response?
- 49:45
Uh, this is OpenAI-- uh, apiweather.gov, where you don't need an API key and it can find the information for you. So I'm gonna just update this. San Francisco. And it's sort of wrong, but it should figure it out.
- 50:01
Uh, weather example, wea- weather word count. And I was very specific, you know, find the weather first, and then how many words are in the response. So it's able to use that tool, it gets the forecast, and then it knows to use that word count tool next.
- 50:13
So we're passing a lot of the information to the model. The models are very smart now. We don't have to say, "Do this, do this, do this." The mo-- Let the agent figure it out.
- 50:21
That's kind of the role of the agent. You give it the context and the tools necessary, it figures out the best way to solve the problem.
- 50:28
But then wouldn't it be prone to hallucination when you give it twenty tools and then-- 'Cause we've tried that with AWS Bedrock. No, the similar things. When you bind more, I think more than ten tools, it's-
- 50:41
Sure
- 50:41
... going to-
- 50:42
Sure. There's always, you know, a balance, but I've-- Again, the models are much better. Like, try using Claude 4 Sonnet. Is it hallucinating as much? Like, these newer models are much better for understanding the concept and understanding what tools when.
- 50:54
The older models, sure, they get confused. There's so many things. But I'm very confident on these newer models, they can understand your use case and what tools are available and figure out the best way to solve the problem.
- 51:04
So then with this framework, there wouldn't be a way for you to orchestrate a customized flow, but more like you give the control to the agent.
- 51:14
You could. If you wanna have, like, specific, like, do this specific way, uh, there are different ways in Strands, uh, with something called workflow mode, where you actually say, uh, you know, "This is the workflow I wanna do.
- 51:27
Fo-- Research results, analyze things, write a final report." If you have to do something very sequential, uh, Strands has that. I won't have time to go through all the different, you know, ways to do multi-agent collaboration and whatnot, but this, for that specific where, like, I wanted to do XYZ first, the workflow, uh, way can do that.
- 51:44
So yes.
- 51:45
Then is it possible to say, um, I, I don't have a predefined workflow, but I know it needs to figure out the right workflow then-
- 51:55
Then, then that's what I just did there. You know, I just gave it-
- 51:57
Okay
- 51:57
... a sentence, it figured it out.
- 51:58
I see. I see. Okay, perfect. Thank you.
- 52:01
So, uh, you haven't touched on it, but, um, Claude 4 has something called interleaved thinking, I believe that's what it's called, where it can handle multiple tools, processing much better than most models today.
- 52:13
So if you're passing in twenty tools, it's able to work through the agentic loop to really figure out which tool to run, and it's also able to run parallel tool calls.
- 52:24
So rather than just say, "Okay, here's the objective. Let me run this tool," it can say, "Here's the objective. Let me run this tool, this tool, this tool, then this tool," and then process the results and determine what needs to happen next.
- 52:37
So I would try Claude 4, which he, like, um, Banjo mentioned.
- 52:42
Then last example really quick. Uh, again, you know, Strands, I've made my Nova Act MCP server, and it can actually run that, you know. I define this is the MCP server, use the Nova Act MCP, you know, use the Claude.
- 52:55
So same type of thing. I can have another agent, you know, use, uh, Nova Act as well. Uh, so Strands make it very easy to build these agentic workflows.
- 53:04
Uh, so that's-- Re- really enjoyed the, the developer experience of using Strands. And you know, I already have the MCP server. We see the same ex-exact example before. So once you have the MCP server, it's very easy to just plug it in, into different, uh, architectures, and Strands make it very easy to, to accept that.
- 53:23
Uh, but yeah, those were the, the three modules really about how to use Strands, uh, um, MCP, then Amazon Nova Act. Again, uh, Strand is open source. You can download it, pip install strands.
- 53:34
Uh, if you just type strandsagents.com, it'll take you to the documentation. Uh, again, also Nova Act, novaamazon.com. It's free. You can log in.
- 53:47
And then I think that's all the time we have. But-
- 53:49
Banjo
- 53:49
... we do have a, a survey. Uh, and you can get AWS credit codes by filling out this survey. So-
- 53:55
Banjo.
- 53:55
Oh, yeah.
- 53:55
I have, I have a question.
- 53:56
Yeah, yeah.
- 53:56
About that workflow thing in-
- 53:58
Yeah, yeah
- 53:58
... uh, Strands. When you create these individual agents, can you define which tools are passed on to each agents?
- 54:05
Yeah, yeah. It's a great question, Dirk, about different agents. Uh, we're running out of time, but I'll quickly show. Uh, I have a multi-agent example, I believe.
- 54:15
Oh.
- 54:16
I think you had it in the docs.
- 54:17
Yeah, yeah.
- 54:17
Yeah.
- 54:17
It's in the docs, yeah.
- 54:18
Yeah.
- 54:20
Yeah, each of these is a different agent. So you know, this is an agent. You can have a different system prompt. You can have different tools. So you're just defining the agent, and then yeah, you can have different tools, different whatever there, different models, and then the workflow would just call that.
- 54:33
So yes, completely customizable. That's the good thing about Strand. It's very easy to customize and build, uh, scalable solutions like that.
- 54:40
Thank you.
- 54:41
And then again, uh, here's the survey. You can get AWS credits for filling out this thing. Tell us how we did, what you liked, uh, what you wanna learn more.
- 54:48
And now go build. [audience clapping] All right. Yeah.
- 55:04
Any other questions by the way? I think we have a minute. [laughs]
- 55:15
Thanks for the presentation. Um, so as these systems develop, I think that it's reasonable to assume that, um, they would emerge as an increasingly effective vehicle for committing fraud online at scale, which would push businesses to implement, uh, more things like CAPTCHA, which kind of decreases the surface area that tools like this would be applicable.
- 55:37
So what is the long-term strategy for that?
- 55:40
Well, you already saw we failed a CAPTCHA today.
- 55:42
Yeah.
- 55:43
Like, you know, we're not trying to bash CAPTCHA. We're not trying to break things, you know. A responsible AI is very important to Amazon. So no, we're not trying to let this tool commit fraud.
- 55:51
You know, you have to have an API key, so it could be monitored. So use cases like that will be shut down.
- 56:00
Are we done? [microphone feedback] Oh, I think we're done. Yeah, so [laughs] thank you all. [laughs] [audience applauding]
- 56:08
Thank you.
- 56:08
I think it's finished, yeah. Oh, we can keep going? We have more time? Oh, you s- [laughs] The cl- the clock, the, the clock ran out, so I thought we were kicked out. [laughs]
- 56:21
All right. Well, more questions then, I guess. [laughs]
- 56:25
I thought... Yeah. Another question. Um.
- 56:35
Um, so regarding Nova Act, let's say that I have a headless browser in the cloud. Is there a way to connect Nova Act to my custom browser instance in the cloud?
- 56:45
Yeah, yeah, yeah. You can cl- There's a way to, like, put your own browser instance. So yeah, Nova Act supports that. So-
- 56:51
Oh, okay
- 56:51
... totally possible. Yep.
- 56:52
Thanks.
- 57:02
Mm-hmm. Mm. Mm. All right, let me go to Nova Act GitHub page.
- 57:11
Nova... And there's just some examples there.
- 57:17
I think, right? 'Cause it says we start at 1:00, and then you have 120 minutes.
- 57:20
I know. I just told him.
- 57:22
Okay.
- 57:22
But I think what happened that time is when he was like-
- 57:25
Oh, yeah. No, you guys can keep rocking it.
- 57:27
Okay.
- 57:33
Yeah.
- 57:34
Yeah.
- 57:34
Yeah. So yeah, there's a way to set up your own user agent for Nova Act, so definitely possible.
- 57:44
So we still got time, so people can keep building or if they got questions.
- 57:48
Yeah.
- 57:48
I think, I don't know what happened with the time, but we still got, we still got time.
- 57:54
Yeah. All right. Well, apparently I have still more time, so I don't know if anyone actually got into the workshop, so we can still, uh,
- 58:01
build some stuff, or I can try some o- other examples. Mm. [sighs]
- 58:26
Try to make Nova Act. [laughs] I tried to make a Streamlit app with Nova Act, so we can try if that works.
- 58:43
Oops. So one example I tried, I tried to make a Streamlit app
- 59:08
that, uh, looked for, like, the top five, uh, PlayStation games on GameFAQs and then create an image, like a nice graph for me. But it ca- it can fail, so, uh, I think that's one of the issues there.
- 59:20
I think it failed at one of the steps there. Uh,
- 59:25
let's see. Oh, that Nova Act got an error, so
- 59:30
it couldn't navigate gamefaqs.com. So it does, it does fail at some of the things, so that's, you know, again, research preview. You have to be more specific on how it goes through things.
- 59:40
Uh, but yeah, let me show you where the, the code is just so you can have an example. Let me pull up the code.
- 59:46
You want something that has more-
- 59:52
Yeah. Let me try, let me set up my local machine so we can see how it works. Yeah.
- 1:00:01
For Nova Act, how much does that-
- 1:00:03
Yeah. You got a question?
- 1:00:05
Oh, yeah. Go for it.
- 1:00:06
How much does Nova Act depend on, like, uh, semantic HTML and, like, good web design to actually work?
- 1:00:13
I mean, it understands the actual page, so it can click through those things. But if the, if the page, like, doesn't have, like, a search box or button, it'll not be able to navigate.
- 1:00:21
So as long as the page-- it can see the page, understand where to click, and then click those correct buttons, so...
- 1:00:26
I guess maybe a follow-up is, is there any, like, efforts to do, like, experimental, like, engagement on the page? So if it comes on a page that it's not familiar with, maybe it would try and act like a human would to, like, click on things or try things out and learn from it?
- 1:00:41
Depending on what you, you put in that prompt 'cause again, you're creating that workflow, what is to do. So it, it, if you say, you know, "Explore this website and find things," it will, it'll try to click through that.
- 1:00:50
But again, it's up to kind of what that initial prompt is that you have for it.
- 1:00:55
Yeah. When you're using Nova Act, you're kind of giving it step-by-step instructions when you're using the SDK. So that way if you kind of know it's an obscure website, you can give it those instructions that it need to perform.
- 1:01:08
Rather than the MCP server, um, is using natural language to infer what needs to be done, so it's not specific instructions coming from you unless you provide it.
- 1:01:27
Yeah. So I'm gonna run it locally on my machine just to show an example. Uh, let's see. What about haptics examples? Maybe...
- 1:01:39
Oh, let me hide my key for a second [chuckles] 'cause this is being recorded. [chuckles]
- 1:01:47
All right. Python get coffee.
- 1:01:58
Thank you. Thank you for coming.
- 1:02:04
All right, so I'm just running it locally on my machine, so without headless mode, so you can see it opens up the browser. Uh.
- 1:02:13
It's able to type coffee maker.
- 1:02:19
So what we're looking at now is not in headless mode. This is actually Nova Act actually performing the task in a browser.
- 1:02:26
So yeah. A lot of questions about how does it work, you know. And we could try a more complicated example. I just wanted to show it can work on your machine.
- 1:02:39
And then you can see the log, you know. I'm just looking for... And if I, like, change the page while it's doing something, it's gonna, like, mess up. So I'm gonna click the page and see what it does like.
- 1:02:47
So someone asked about if I click things of that nature, what's it gonna do now? So...
- 1:03:00
So see, it, it crashed now 'cause I bro- I changed a different page, didn't know what to do. So example, you can interact when it's, when it's going through the, the motion as well.
- 1:03:08
And then I believe I have an, uh, concerned MCP server. I set up a Claude instance. Oops. [thud]
- 1:03:26
And then I have an-- my Nova Act MCP server is there, so I'm able to actually, you know, I click this. You can see all the tools it has available, so I can ask it to, like, navigate a website.
- 1:03:36
So, um, anyone having a complex example can see the MCP server. So I know some people have been asking some complex examples, so get, get-- go ahead and give me one. [chuckles]
- 1:03:50
Yeah, you got, you got one? [chuckles]
- 1:03:52
Well, one question I had is, uh, can Nova support, like, drag and drop functionality?
- 1:03:59
You can try it. Do you have a specific website that has, like, drag and drop? [chuckles]
- 1:04:03
Um, for instance, I guess like a movie. [background chatter]
- 1:04:16
Bot draw.io. Uh, let's go to draw.io and make a cool diagram. Use Nova Act.
- 1:04:28
And let's see what happens. So... Great, so it's go to draw.io. All right, it opened the page.
- 1:04:45
Do I have to accept something? Nope, it's going. Oops. All right, it opened draw.io. Let's see.
- 1:04:53
And then I'll-
- 1:04:54
Do we still want to start one?
- 1:04:55
-make this smaller. Wait for page to load. Look at my initial setup for attempting selections. All right, it's going. Uh.
- 1:05:09
Oh, it crashed. [laughs] What happened? [chuckles] Oh, do I have to allow? Allow always.
- 1:05:17
Oh, it took a screenshot. I need to continue the browser session to see what's available. Let's look at the screenshot. All right, it's opening up again. Uh, it's going to draw.io.
- 1:05:49
Yeah, if I keep clicking away, it clicks back to the di-- the browser session. So I need, like, two monitors.
- 1:05:57
Let's see. Is it gonna figure out how to use draw.io?
- 1:06:04
Wait for page, take screenshot, look for template options, come up with blank page. All right, it's... So it's kinda-- I didn't give it any specific instruction. I just said make something cool, so maybe that's too hard to interpret for this website.
- 1:06:16
Maybe I have to say, "Click the sq-- click the square button and then drag the square to the center," or something. I might have to be more explicit for that.
- 1:06:27
So it seems, it seems to have frozen. Uh, oh, wait, it's clicking something. All right. Click New.
- 1:06:38
Oh, okay. It's doing stuff. Again, it-it's not, like, super real-time. It's going. It's not, like, instantaneously, but it, it's, it is clicking through the buttons, clicking through stuff. All right.
- 1:06:53
Did it do anything? Oh, the Claude. [chuckles] So it looked like it fai- [laughs] So yeah, it looks like [chuckles] Claude failed that one, so I won't blame Nova for that. But that's the i-- that's the idea.
- 1:07:05
So [chuckles] thanks for trying to do something hard. [laughs]
- 1:07:08
All right. Another question back there?
- 1:07:11
No, we're all right.
- 1:07:11
Oh, yeah.
- 1:07:12
Can we, can we try another one?
- 1:07:13
Yeah, let's try another one. Sure.
- 1:07:14
All right. Can we do, um, you know, o-on Google Maps find the top three rated coffee shops with-within a mile radius of this hotel?
- 1:07:23
Top three coffee shops Um, snare the Marriott Marquis in San Francisco.
- 1:07:38
Uh, you'll figure it out. [chuckles] All right. Open Maps Google, search Marriott Marquis San Francisco, wait for results to load. So it has a plan, it's gonna do something. So let's see.
- 1:07:53
It opened Google Maps. All right, type Marriott Marquis San Francisco, so it's able to type that.
- 1:08:11
Okay, it searched, it found the Marriott Marquis.
- 1:08:20
So there's a copy button, let's see if it clicks that. I'm curious.
- 1:08:36
Hmm. Looks like it's frozen. Give it a couple more seconds.
- 1:08:45
What did it click? It got this 15 minutes. Oh, it's trying to type in that box. Okay.
- 1:09:03
All right, it's typing coffee shops. All right. All right, it's going.
- 1:09:14
All right, so... All right, it opened the coffee shops, and let's see if it can get those top three. There's a four eight, four seven, another four seven. Let's see if it can get that. [chuckles]
- 1:09:39
Did it crash? I think it didn't, but I think [chuckles] I'm gonna blame Claude. Claude Desktop crashed. [chuckles] Might need a different MCP client. Uh,
- 1:09:52
yeah. I think, yeah, I think Claude Desktop doesn't like doing that. But again, because it's an MCP server, I can open up a different MCP client. So I can open, like, Cursor, for example, and ask it questions through that.
- 1:10:07
Uh, Cursor. Then let me close this. And then
- 1:10:22
you see it has the MCP tools. Oops, it has it up. Let me just open up a new one. I can do the same thing and use Nova Act.
- 1:10:41
And then it's calling the MCP tool again. So that's the beauty of MCP. I already have the server, I can just use a different client. It can understand all the information it needs to and do the exact same command.
- 1:10:51
So, uh, it's gonna do the same thing. Cursor might be smarter than Claude Code. [chuckles]
- 1:10:58
But yeah, it's able to do the exact same type of thing, so...
- 1:11:04
Another question over here?
- 1:11:05
Yeah, I've just got a question on the, the Nova Act model.
- 1:11:08
Yeah.
- 1:11:08
That, that model, is that, that... Is that running in the cloud or is that running-
- 1:11:12
Yes. So Nova Act. Question was where is Nova Act running? And yes, it's running in the cloud. Uh, so yeah, it, uh, it's just... You get that API key, and it's doing the call behind the scenes in the AWS cloud.
- 1:11:21
Yeah.
- 1:11:21
So then what, what does it upload to the cloud?
- 1:11:24
Well, it's asking the, the questions and like, you know, go to Google Maps, and then it's able to... I understand that, and it's actually clicking those buttons and doing the actions.
- 1:11:32
So the, the actual i- uh, intent of what you're trying to do in the specific, uh, action.
- 1:11:39
Okay. And if, if I was, if I was using it locally as opposed to, say, my-
- 1:11:43
You couldn't use Nova Act locally. It has to be, uh, connected to internet to use it.
- 1:11:50
Okay. But if I... For example, though, if I, uh, wanted it to, like, look at my Gmail-
- 1:11:56
Oh, yes
- 1:11:57
... and I logged in with it, then be able to have access to, like, see-
- 1:12:01
Ah, yeah. I, I see what you're saying. Yeah, yeah. If you... It's... I mean, it is, you know, it's a API endpoint. It's been passed to AWS, so, you know, only passing information that you feel like it's not gonna be...
- 1:12:10
We're not training the data or taking any of that nature, but it's going to the AWS cloud and processing, you know, what to click on this button locally on your, like, browser.
- 1:12:19
Mm-hmm. Okay. Cool. Okay. Thank you.
- 1:12:24
So looks like it's not... Yeah, see now it even search- certain the rating. It actually knows which rating to press, so...
- 1:12:32
Hmm.
- 1:12:33
So the Nova Act is going to just give the, give the plan, right?
- 1:12:37
Right. Yeah, yeah. Well, Nova Act is, is executing. Like, in this MCP server example, I say, you know, "Find the top three coffee shops in Marriott, near the Marriott Marquis."
- 1:12:47
And then I'm passing that information to the, the LLM to understand that plan, and then it uses Nova Act to interact with the browser. Because, like, Cursor or Claude Code or Amazon Q, they can't interact with the specific, uh, you know, website by itself.
- 1:13:02
It uses, it uses Nova Act to do that.
- 1:13:04
Right. But, like, given a question though, like, how, how does it come in, uh, come up with a plan?
- 1:13:09
Oh, the MCP server, like, the, the client. So I picked the model. In the example we had, the MCP client we had as we showed the model.
- 1:13:17
Right.
- 1:13:17
I can use a Claude 3.5.
- 1:13:18
Yeah.
- 1:13:19
That's coming up with the plan. Same thing here. You know, I asked, you know, "Help me find the top three coffee shop near the Marriott Marquis."
- 1:13:25
Mm-hmm.
- 1:13:26
This... The model that, uh, Cursor is using is coming up with that plan, and then I'm using the Nova Act MCP server-
- 1:13:32
To act on it.
- 1:13:33
Exactly.
- 1:13:34
Right.
- 1:13:34
So this is the plan. Search for Marriott Marquis, click the Marriott Marquis, you know, search for the things, and you see all this information Nova Act returned, and it actually- It will return this time.
- 1:13:43
So I think the problem was with Claude desktop, but it got the three top three coffee stops there. So-
- 1:13:48
Right. What are all the tools that, uh, Nova Act can do today?
- 1:13:52
Uh, so the MCP server is one I wrote, so, uh, but the, uh, idea between Nova Act, it can, it can interface with the web browser. That, that's the tool.
- 1:14:00
I see.
- 1:14:00
The browser is the tool, and it can... Anything that on the website, it can actually click through, go through the example, et cetera.
- 1:14:05
I see. Got it.
- 1:14:09
You got the repo? Do you got an architecture that shows the MCP just so they can see it?
- 1:14:14
Yeah. So I mentioned, uh, there's an official AWS MCP server. So, uh, this AWS Labs MCP, and, uh, there are a lot of different, um, MCP servers here. For the one, the Nova Act one, I created my own one.
- 1:14:28
Uh, let me go back to the Nova Act examples or
- 1:14:32
where do the... Ah, here. When I used Amazon Q to explain, you know, the am- the MCP server for, like, what, what's going on, what tool is the browser session performing an action on the browser.
- 1:14:44
So this is a good, uh, thing to talk about. So can you dive deeper on the browser action function? And then we can see, 'cause this is how it's actually acting.
- 1:14:57
So, um, Amazon Q browser action is designed to perform actions. It has this, uh... What's cool about it, it just does-- Oops. Let's go into the code. It performs a single action in the Nova Act browser.
- 1:15:15
So it's executing that action. It stores this act.act is like what Nova says, you know, "Click the search bar, do this, X, Y," you know. My-- The MCP client understands how to use this act.act that passes the correct action.
- 1:15:30
So we saw the example here. One of the action was like, uh, go to Google Maps or click this button or do that search. That's how it's able, you know, these actions, and then the Nova Act MCP server is translating that to actually click that button.
- 1:15:44
So the MCP server provides all the interfaces it necessarily needs, so then these MCP clients can interact and do actions and do things. Yeah. And Nova Act is just the model in the background that's able to click those buttons.
- 1:15:59
Uh, e-extending this question, it-- So your MCP server, so Claude, uh, or, um, Cursor running locally, right? It's calling your MCP server that's also running locally. Is your MCP...
- 1:16:12
And your MCP server is the one that spun up the, I guess, the Chromium instance, right?
- 1:16:16
Yes.
- 1:16:17
Is it, is your MCP server taking screenshots of what you see in Chromium and shipping them to Nova, to Nova Act?
- 1:16:24
The, the screenshots are locally, and then based on that, like you can see, it's actually getting the, all the information, uh, the final page information. So it's not storing your screenshot data and sending that.
- 1:16:33
Everything that, it's running locally, and it's clicking those buttons based on what's on the browser sensing.
- 1:16:38
Got it. But is, is any of, any of the information in Chromium, does that, any of that need to be sent into any form to Nova-
- 1:16:43
No, no.
- 1:16:43
Okay.
- 1:16:43
It does everything running lo- Yes, running locally. That was the-
- 1:16:46
Okay
- 1:16:46
... good, good distinction. Yes.
- 1:16:47
Okay. Perfect. Thank you.
- 1:17:00
And let me open up the... Where was I looking? Ah.
- 1:17:15
So one of the things about making MCP servers is you have to provide a lot of context. So, uh, for Nova Act, like I say, you know, when writing actions to Nova Act and be descriptive of what to do, you know, click the hamburger menu icon, go to order history.
- 1:17:29
Don't buy my order. So the more, you know, uh, s- concise and pres- prescriptive of what you wanna do, it's better, you know. Search for hotels in Houston, sort by average customer.
- 1:17:37
Like, so the better specific it is, uh, that's how the MCP's, uh, client is able to make those great requests and find the information. So type coffee maker, search box, Enter.
- 1:17:47
So, so the more prescriptive you are of Nova Act, the better results it's gonna be, and I encoded that all into this, uh, MCP server, so the clients can leverage that.
- 1:17:59
So I think that's probably one of the hardest things about making the MCP server is just making sure you provide the next context of when to use the tool, how to use the tool, the inputs and outputs.
- 1:18:08
But once you solve all that, it's very easy to just plug and play the different MCP clients like we've done here.
- 1:18:39
Question? Yeah.
- 1:18:41
Um, with the same example of the end-to-play, how do you make sure or how does the model make sure that Nova Act executed the action successfully?
- 1:18:52
Right. So when Nova Act is doing something, it's passing back the log of everything it's doing, so you know what, what steps it did. So the starting page, the a- the results, the action result ID.
- 1:19:03
So it's keeping a log of everything it did. Uh, power, so it's able to get that JSON to understand what the ID, what the result is, so you can see what it's doing, so it can move on to the next step.
- 1:19:13
Yeah. Yep. Make this bigger. Another question? Yeah.
- 1:19:28
Sorry, a quick question.
- 1:19:29
Yeah.
- 1:19:30
Is this able to do, uh, like, uh, automated UI testing because of this?
- 1:19:36
Well, you-- well, with Nova Act, you know, you can define, like, what you want it to do. So you're gonna have to define, you know, go to this button, click this, does this work?
- 1:19:43
So you can define that workflow. So I mentioned before, like, back in the day, like, if I'm writing a Selenium code, I have to say, "Oh, click this H1 tag, do this."
- 1:19:51
Like, now you can just write in natural language. You know, click this button, click that button. So yes, it can handle that use case, uh, specifically of, like, opening the browser or checking these things.
- 1:20:00
And but you have to, like, you know, this Nova Act, search for coffee maker. You know, you have-- you specifically have to write what buttons to press.
- 1:20:07
Yeah. Thank you.
- 1:20:19
Let's see. I guess if we have time, I could show some multi-agent collaboration with Strands. That could be something cool. Uh, I think I have a repo for that.
- 1:20:29
So should be, uh, go to the AWS Labs page. Where is that?
- 1:20:39
Nice worksheet. And then Claude. Cool. Okay.
- 1:20:58
I'm just gonna copy this code and put it into our environment.
- 1:21:27
So in this example, I'm actually going to show how Strands does multi-agent collaboration. So one y-- uh, way-- I'm actually gonna create a PowerPoint presentation based on, uh, you know, a cloud migration request.
- 1:21:40
I wanna, like, move my, um, infrastructure on-premise to the cloud. Give me a presentation of how I would do that. And so for this, I created three different agents.
- 1:21:50
I created a cost analysis agent, so I have a system prompt there, a solutions architect agent to s- um, map out what you're gonna be doing, and then each of these, uh, tools is an actual agent.
- 1:22:01
So this, uh, cost analysis has the docs, uh, MCP server, the cost analysis MCP server. It has its own prompt. The presentation agent has its own system prompt. It has a tool from the, uh, a PowerPoint MCP server that I'm using.
- 1:22:16
And there's an architecture agent that also has, you know, its own specific tools, system prompt, et cetera. So, uh, different agents for different, um, uh, things in the workflow.
- 1:22:27
And then I have this orchestrator agent, what I've called the migration orchestration agent. It has a prompt. It-- I tell it what tools it has access to. And then the cool thing with Strands is I make this orchestrator agent, and then the tools are just other agents in that.
- 1:22:41
So it knows when to call this agent for this particular tool, when to do that. And I say, you know, [lips smack] "I want to migrate my work, my, uh, workload," so write the right tools to find that.
- 1:22:53
So I made a fictional company called Shop Easy eCommerce. They have on-premise Java, MySQL database. They want a zero downtime migration. Like, all this, all these little constraints in there.
- 1:23:05
And I wanted to make a migration plan and a PowerPoint presentation that I can present to my executives of how this would work. And I just decided-- and I'm the orchestrator agent will find out what to do.
- 1:23:17
I don't specifically say, "Do this one first, do that first." We'll let the, the agent figure that out. So let me run that. Strands. And it should be multi-agent.
- 1:23:34
Great. So cloud partition, agent as tools. All right. Again, so all the MCP servers running locally. It downloads its Unity UX. It starts with the architecture design first, generates a diagram.
- 1:23:55
Gonna use WAF, so take some time. It might fail, but it will just update, update itself.
- 1:24:02
Making another judgment. All right. I think it couldn't generate the diagram there, but it's saying, "All right, I'm just gonna...
- 1:24:17
This is what the diagram should have. This is what we're gonna doing."
- 1:24:27
Now it's gonna do a cost analysis. Cost analysis are based on the things we did there. So it's, it's a-- this c- this workflow takes maybe a couple minutes to run, but you can see it's calling all these agents, uh, different things.
- 1:24:38
It's understanding what to do, what actions it needs to take first. It's finding pricing for EKS because it has a, a cost analysis tool that knows where to find that information, so it has the up-to-date pricing all the time.
- 1:24:51
Finding for Aurora, for its database. So it's able to understand all that information and get the real-time up-to-date information just because we have that, uh, pricing MCP server from the AWS Labs example.
- 1:25:03
Does region matter for this?
- 1:25:04
Pricing.
- 1:25:05
US West two.
- 1:25:07
This is it. Oh, cost analysis.
- 1:25:10
Oh, just pre-
- 1:25:11
Yeah, cost analysis MCP server, documentation, all the stuff you need for finding the right price on AWS. It has all that information, and the agent was able to just use that once going to generate a report.
- 1:25:29
So it's still running. Again, this does take a while because I've asked it a very complex question, a lot of things going, uh, so it does take a couple minutes to run through all that.
- 1:25:38
It gets its mon-monthly spend predictions, monthly savings, et cetera. So it's able to understand all the information and get all up-to-date information based on the plan we provided.
- 1:25:51
And the last thing, now it wants to create an executive presentation. So it downloaded the PowerPoint MCP server, and now it's gonna make a, a PowerPoint presentation based on that.
- 1:26:00
Is it going to be a while for the next tool again?
- 1:26:02
Yeah.
- 1:26:03
So adding the title slide. So, you know, add a placeholder. So generating PowerPoint is a very [chuckles] popular use case, and there's an MCP server that can go ahead and just do that, add bullet points, et cetera.
- 1:26:15
So give it a couple... another minute or two.