Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab

Hello, everyone, and thank you for coming to our workshop, Building with Perception Agents. Uh, my name is Emile Baizel. I lead a team of solutions architects, uh, within Amazon AGI Lab, and our team is focused on two things. The first is working with some of our earliest adopter customers who wanna work with some of our early models and products to help them get into production. And the second part is closing the feedback loop from our customer engagements, uh, building evals for our research team to train our models on,

and bringing product, uh, and engineering feedback to those teams as well. And I'm joined by one of our stellar solutions architects, Shruti Arora, who's gonna be running the workshop today.

Uh, just a bit about what we're gonna be talking about today. I'll be giving a quick overview of the Amazon AGI Lab and talking about the first product that we shipped last year, Amazon Nova Act, and then we'll get into Perception Agents, and that's what the focus of today's workshop is, and then we'll get into the workshop.

So a bit about the lab. So we are a frontier research and product team that combines startup speed with the scale and the resources of Amazon. And we are really focused on building knowledge work agents that can perceive, reason, and act in a variety of different environments to complete real-world tasks. Um, our offices are in San Francisco and Boston. We've got a few other talks going on throughout the week. We have a booth at the expo, so stop

by to collect some swag, meet some folks from the team, and hear some of the talks there.

So I wanna share a bit about our approach and how we think about agents

across three different ways. So the first is around collaboration. So we really think about building agents that are teammates much more than just a black box. And when we say teammates, we really want an agent that can understand what our goal is, our objective is, and can share how they're thinking about achieving that goal, and, and then grounding all of that in actual customer, uh, use cases. And so we work very, very closely with a large number of customers

to really help ground all the work that we're doing with agents. And the second is around perception. So agents need to understand their environment. And so for us, because it's a perception-based agent, the environment spans multiple devices. So we started with browsers last year, but you can imagine any visual screen is an environment for our agent to carry out knowledge work in, so mobile devices and also computer use is the other one.

And the third one is reliability, and we have a great customer quote that said, "If my agent works fifty percent of the time, it doesn't work at all." And that's because you're either not gonna run it at all, or if you do run it, you're gonna end up, uh, debugging it and spending all this time, and you're gonna basically do away with all the gains that you're getting from that agent. And so we spend a lot of time building and training our models in, uh, simulated environments for reinforcement learning. Um, and again, a lot of those are based on real-world, uh, use cases, which we get from not only

our external customers, but a lot of internal Amazon teams that are carrying out a lot of, uh, knowledge work.

Cool. So with that backdrop, I wanna talk about the first product that we shipped last year. So Amazon Nova Act is a first-class AWS service that allows developers to build and run reliable browser-based workflows at scale. So using natural language, you can drive a browser, uh, workflow to populate a browser, navigate, search with it, fill out forms, and call a human in the loop if you need a human for a expense report

approval or if you get stuck or other things like that. And there's kind of four primary use cases which we've seen customers adopt this for. The first is form filling. Uh, so if you have to do that at scale across a large number of different websites, and you don't wanna write a prompt for each single website or write like a, like a, like a, a deterministic script for each single site, you can use that, uh, Nova Act for that. Uh, search and extract if you wanna, uh, do research and get

data from a lot of different websites, shopping and QA. So a bit more on what we launched. Uh, the-- Let me just get through all these. Uh, so it's a first-class AWS service. So if you go into the AWS console, type in Nova Act, you will see Nova Act there. And from the console itself, you can see all of the, all of the trajectories. So it's a very easy way to see everything which your agent has done, the bounding boxes around all the elements which it's looking at, and it's trying to take

actions on there. And then behind the scenes, we have a custom computer use model, which is, uh, doing all of the, all of the inference there. This year, we launched, uh, an MCP server as well as a skill, and you're gonna be using those in the workshop, and that's for your agents to be able to use, uh, Nova Act very easily. And then we've got human in the loop and tool use. And we also have a playground. So if you go to the Nova Act homepage, um, it's a no-code way to just see Nova Act in action there. Now,

looking ahead, and the topic of today's workshop is Perception Agents. Now, why do we feel there's a need for Perception Agents? So, uh, for those of you who are not familiar with the term of computer use, uh, that's basically the way for an agent to take an action on a computer screen, right? So you see a blue button in the right corner, and you tell your agent, "Hey, go click this blue button over here." Very important, very powerful. It's very important for, uh, automating workflows. But we feel that there's another half of that, which is the

agent actually understanding what it's seeing on the screen itself. Not just using that as a sort of blind, uh, screen that it can just go and click on elements there. And we really want this agent to be able to understand screen just like a human would, right? So if I showed someone a, uh, email client from, like, the late nineties that was, like, a Yahoo email client or Hotmail client or something which you'd never seen before, you would probably be able to figure out how to send an email from that even though you've never used

that client before. 'Cause we've got this collective understanding of thousands and thousands of hours of using these screens that we kinda intuitively understand how to use these things, and that's really where we're trying to get to with these, with these agents. Uh, and the second one is these agents don't require APIs in order to work, right? So if you think about a lot of knowledge work, um, it's a lot of humans using screens that just don't have APIs, and that's why there are people who are doing these things.

And we want these agents to be able to carry out the same work across a variety of sites as well. And we want the models to see the screens the same way that we do because that really facilitates the collaboration, right? So rather than typing up paragraphs for what you want the agent to do, uh, on this screen and describing the screen, uh, you can just point, click, and describe changes that you want, and that's what you're gonna do in the workshop today is you're gonna click on a couple elements and say, "Hey, I wanna change the background to red," and that's it, and

then you just have the agent go, go and do that.

And, uh, to, to start with, we've open sourced two perception agent primitives, and you're gonna be using these in depth in the workshop today. Uh, the first is visual annotation, and so what this lets you do is on any webpage, you can point, click, and just describe changes that you wanna make. And when you're done with that, you click Export, and it exports out a JSON file with all the changes that you want and a prompt, which you can give to your coding agent along with that JSON file to say, "Hey, go and make these

changes and do that." So importance, uh, to, to note, this has nothing to do with inference. This is purely deterministic. It just outputs JSON, doesn't have-- it's not tied to any model or anything else like that. The second part, uh, the second primitive, which we've launched, is visual verification. And if you think about a, a site that you have, you've got certain CSS styles and designs which you want your, your site to adhere to. Uh, you've got different workflows which you want your site to support. Maybe you have, like, a checkout

flow or a login flow or something else, and you can describe those as Gherkin files. And so you can have visual verification, this, this skill, um, run by your coding agent against these design artifacts that you have, against these Gherkin files, and make sure that your site is adhering to those. And again, this is something which you're gonna use, uh, in the workshop today. And again, these are all open source, available today, and, uh, yeah, that's what we're gonna do in the workshop. So I think at this point

I've spoken enough, and I am going to hand it off to Shruti, who is gonna kick us off on the workshop.

Do just parts of that.

Okay. Thank you. All right. Can you all hear me okay?

Yes.

Okay. Welcome. Uh, thanks, Emile, for, uh, that wonderful introduction to our, uh, Amazon AGI Labs and the products that we launched, we released, uh, earlier in May this month-- this year. So as he mentioned, we are going to be doing hands-on workshop, but hands-on on, like, whatever we release, the UI and visual annotator and the visual verification. I

have a short demo here, uh, but before we dive into the demo, I just want all of you to, like, step into this role play here. Uh, just, just bear with me. Um, so imagine you are a front-end engineer, and you are building a web application. You have, uh, a design team, and you are taking inspirations. They have these Figma files that you are, uh, building against. You are also collaborating with a sister team

that they want to put in a piece of their, uh, information on the webpage that you are building. So if you are-- If you think about it, that you have the designs, you'll probably take, like, uh, four to five, six weeks to, like, build the designs and build everything code base and actually build that website. Now, in the third week or, like, fourth week around that time of your building process, you actually think, "Oh, maybe I should, like, go back and, like, verify if this looks

fine, if everything looks okay with the rest of the team," and you are constantly collaborating, right? So you are checking if, if the work that you have done is something that is, that is what you want to ship. Um, so you send your code base to-- You share the GitHub or however you are doing your, um, GitLab or whatever, uh, you have your code in. And you-- They run that, this, this web server locally, and they open up the website, and they say, "Oh, this, this

feels off. You know, the header, heading should be probably not in red. I sh- I want to switch it back to something." Um, and they provide these feedbacks. Probably they will do it in, like, a Slack message or in a shared doc, for example. Uh, and then once you get those feedback on that, uh, on your website, you would, um, then, like, go over one of them. You'll disagree, agree, uh, to some of them, come to a middle ground, and finalize

this is the feedback you want to incorporate. You have to go through all of them and one by one then, and start making the changes. Uh, now with coding agents, I guess it'll take maybe over- Which used to take maybe a week or it can like reduce to like maybe a few, few days. Now think about if you had the ability to take their feedback directly and pass it to the coding agent without you, your supervision, and just like once the changes has been done,

that's when you interfere and say, "Oh, this looks good. All the changes have been incorporated." How would that reduce the time? Maybe few hours, you just start, um, collect the feedback, send it to your agent and go for a coffee break, come back and all the changes have been merged. So this is exactly what, uh, Emile was mentioning, uh, that we have a Chrome extension that we are calling as visual annotator. Um, there are three ways to annotate. You can

see here, it's either you draw on the page or you pick the element, or you drop a pin as you would drop a pin on a Google Map, say, and provide your annotations. And once you have your annotations, you can download these and send it to your team who needs to make the changes. Or even if it's you who's make the changes, who's making the changes, then we provide a AI-friendly prompt that you can send it to your agent and it's going to change automat-- like, well, you have to instruct the agent to like

change the code base.

So this was the visual annotator. The next, um... Oh. The next, uh, primitive that we launched is the visual verification skill, and when I say skill, it is actually a skill that you can install on Claude or Kiro P- Kiro Power. And, um, you would see these verifications, as Emile mentioned, these are these design files that have, against which you will be running your

visual like, uh, verification against. Like if, does the design colors look fine? Is the font size right? Is the user flow right? But one thing that will come to your mind, like these design specs, these flows, how are these generated? How are, how, how are you gonna build that? So not to worry, we have a solution for that as well. When you run UI verification for the first time, the, there will be a, a deep, um,

uh, one level, uh, surface crawl that will be done by the UI verification tool on your website to go and check what are the links, what are the content, and based on that, it's going to create these user flows that are in terms of, in Gherkin f- uh, for syntax that the UI, UI verification is going to use when it's going to run the verification next. So let me just play that, play the demo here. Um,

so you can see, you just like call the tool. It's going to go through the design docs, specs, uh, spe- specs, and the, uh, user flows. Uh, go through it. It's going to open up the browser, run through the design, uh, flow, user flows, and then generate a report. So this, these are the two tools that you will get hands-on in this workshop. Uh, so I'm gonna jump to what we have, how the workshop has been

laid out as of now. First, as I mentioned, two, uh, hands-on. The second thing that's what we are trying to do is, these are the two primitives, of course. Perception is not just these two primitives. How would you wire these two primitives up for a particular use case? There is a scenario that has been, um, given in the workshop. We are trying to make this fun, so please try it out yourself, um, and see, like it's like a game day.

You would see there are like certain hints for the, to solve the problem, what the problem is, what the end goal is. So everything is laid out in the workshop. Uh, it's a GitHub link. We will share that with you, uh, in a bit. Uh, so how will you basically wire these two things up? Uh, the sec- the next thing we have are, um, sponsors for the prizes. We have certain contests around at the end and, um, thank you the Bee team for sponsoring the prizes for the, uh, for

the, uh, event. And, um, we will have a way how you can, instead of typing or like annotating on the screen, what are some other ways, cool ways of making sure that there are some annotations that you want to do or like feedback that you want to provide about a webpage or a designs. So Bee team will show you how you can do that beyond just typing in your feedback. Finally, there are...

Well, before that, we have, after you finish the workshop, we also have like three different options for bonus challenge. We will have plenty of time for you to try out these tools in a particular use case that you want to. Um, maybe some, uh, you want to try wire it up with Bee, um, and see how that run, how you would probably wire it with any other, uh, form of, um, feedback that you would essentially get from your team. Or if you think that, hey, this is really cool,

I want to build something that I want to show to my team on Monday, like the following Monday. So go for it and show, if you're brave enough, come in front, come at the stage, show your use case, demo your use case, and you'll get a Bee device. The final part is, uh, we are gonna do a little fun exercise where, uh, once you have a good understanding of these two primitives, uh, you can think of like multiple ways of how you would use it on a

daily, on your day, day-to-day. And we will s- you will submit these ideas of like, "Hey, I think this will be really cool for me to use this," and we will have people vote on for these ideas. And the top three ideas Will get a Bee device, and I think we also have one for, uh, crowd favorite. So with that, I'm gonna jump into the workshop. This is the link for the workshop.

Uh, you can type in, it's pretty, uh, it's, it's available on GitHub, so you can type in Perception Agents Workshop. You will probably get the first GitHub link. Um, in the workshop, we also need you to pl- um, have, use a No- Nova Act API key. So there is a piece where in the workshop we have obfuscated the first three letters. So that is on the screen here. So you can

like copy-paste these three letters, A-C-E, that stands for ACE, and, uh, you're good to go with the Nova Act API key.

So let me just

pause this. So in the workshop, you would-- the README will basic- point you to the WORKSHOP.md file. And here is the scenario that I was talking about. I'm gonna read it out from here. "You are a front-end, uh, developer preparing to ship a landing page for a podcast. Uh, yesterday you demoed the page, uh, to a few peers, and they provided some feedback. The hero title feels too big. The accent color doesn't pop up the way you... it should. Some buttons are

oversized, um, and a few small things that you s- are slightly just off. So you are on board with those feedback, and today you come in fresh and are ready to incorporate those changes." So what do you have at your disposal? The Nova Act Chrome extension, the UR- UI verification skill. You have the Nova Act SDK and the Nova Act MCP server. These are the tools that you have at your disposal to basically, uh, implement the

solution, uh, where you would essentially, uh, build, build this entire loop by itself. Like, you just annotate, and automatically you would just hit a button on, in the Chrome extension, and, uh, all the changes will be applied. The UI verification will run, and you have a report by the end of it. Um, so this is the challenge, and there are a few

prerequisites. So, um, the, you need to have an, uh, macOS or a Linux too, because Nova Act MCP server runs on macOS and Linux. Uh, you should have Node.js installed. If not, you can install them. Uh, npm, Python, uh, Git, Chrome browser to basically test it out because it's a Chrome extension, and then, um, an AI coding agent. So if you have Claude, that works great. We also have Kiro integration, so if you don't have any co- coding,

um, agent, you can install Kiro and start playing around with it. Um, and lastly, we have the uv, um, that's the uvx. That's basically to install. If you don't want to install uv or uvx, then please raise your hand and we can help you install using pip. Um, so that's, that's that. And the workshop is laid out for, I think the setup should be about five to seven minutes. Uh, then you can

like start playing around with those skills, like gets a good understanding of how would you use the Chrome extension, how would you use the, uh, UI verification skill. That's what has been explained here. Uh, to solve the challenge, you have, um, about fifteen minutes in. If you want to go that route, like I want to go step by step, and I'm just gonna go scroll down really quick. Yeah. So choose your path. I have this workshop designed in two levels.

Uh, b- you remember I said like I want you to have fun. So two levels. If you want to go for a challenge, go for the medium route. If you feel that, uh, I just gonna like, uh, follow the instructions and get the workshop done, feel good about it. So yeah, we have an easy mode where you'll have exact instruction what you, what should be done. And you will have these two primitives wired up by the end of the hands-on. Um, so fifteen minutes we are expecting the easy path, um, and then

thirty to thirty-five minutes for the medium. Once we have allocated the fifteen minutes time, I will come up and show you where you would like essentially, um, add your code so that, uh, you can wire these two primitives up, uh, and you can see the entire thing come into live in action.

And we also have a bonus challenge. If you finish early, you can start with the bonus challenge. Uh, but before that, we will have the Bee team, Giovanni, um, in some time to like present.

First-

After, yeah. Yeah, sure thing.

So, um, I saw that you had, as I was installing these, um-

Mm-hmm.

What is it? The Nova Act UI verification skills. Um, it, it said that there were some default clients that are always supported, and they include like Codex and Claude Code. But then below that, there are these additional skills, and one of them is also Claude Code. So I'm just curious, like are there two different things going on there, um, or are those the same? The, the, the fact that there are two different Claude Code skills.

Yeah. So there are two skills. One is the Nova Act, uh, skill, one is the Nova A- UI verification skill. So that's why you must be seeing both of them. Yeah.

So I'm gonna give you the time here to

I'm gonna bring up the slide back. I'm gonna ask Emile to pop up here. Uh, slide back on the GitHub repo, and so that you have the, uh, API key handy if you have to change. Feel free to, like, raise your hand if you have any questions. We have our great, awesome team supporting the workshop. There you find them, see them waving. So, uh, reach out to either of us and, uh, we will pop by you to help. Have fun.

Okay, I'm just gonna try to walk through the solution so that...

Okay.

So, uh...

Okay.

So this is the step-by-step instructions. You can see that we have the Chrome extension, we have your agent s- uh, AI agent running. Uh, if you want your coding agent to, like, automatically apply the changes from the extension, you need a bridge, because there's no bridge between them. How would the extension, like, send the prompt to the coding, uh, coding agent? So for that, we have built, uh, something called Agent Bridge

that will automatically, uh, take the annotations and appl-- give, prompt it to the coding agent. That's where it'll start applying those changes and then running the UI verification skill. So, um, essentially in the extension, we don't have the Apply Changes button in the, in the primitive mode, but I, for this workshop, I have built in-- we have built this, um, Apply Changes

button in the Chrome extension. Um, I'll walk you through that in a bit. Uh, so for the Chrome-- for, for this entire thing to work, you need to make changes in, like, two files. The first file is, um, the manifest.json file in the Chrome extension, where you essentially add the, this line for the host permissions to start accepting requests from the local host.

Uh, and then you add this, this piece of code. It's a huge, like, piece of, chunk of code here. Um, and you can... What this part, uh, does in the extension is creates an Apply Changes button, wires it up with the extension, uh, makes sure-- makes, makes the UI verification run, uh, in using the Nova Act SDK instead of the skill, uh, and starts, uh,

generates a report that you can click on and then view the report. Um, so this piece of code is, goes in this file. That is the, um, let's see, content.js in the extension, and it needs to be added around, like, line two sixty-one, uh, between this append child export button, just before. And I think I have hashed, uh, there's a comment there, add code

here, in that file, and that's where you would, like, paste this code. So once you have that running,

now you would see this Apply Change button here in the extension. Um, I think, uh, Emile was testing this, so he changed this to, like, FIFA World Cup. Why not? Um, so I'm gonna select one of these and I'm gonna refresh the extension again.

Let's see.

Annotate current page. Uh, I'm gonna try to change the published date.

Change

to, today's June twenty-ninth. Okay. And save it. And this is the extension. Apply changes. So in the background, it's gonna open up your coding agent. Uh, it's been tested on Claude Code and Kiro, as I mentioned. So if you are trying Cursor or, or Codex, the Agent Bridge is something that

you would have to edit so that it can accept these Apply Changes button, uh, like, requests. Um, so in a bit it'll start applying the changes, and you will see certain logs that pop up as well in the repository. Um, that is the key, the CLIs logs, as well as when the verification runs, Nova Act has these logs, uh, that gives you exactly what the, uh, model was thinking and

what part of the screen it was trying to make a decision on. And you can see that the button changed to Run Verification now, so it's running the verification step.

And in a bit, it'll also generate a report that you can see. So this is where you basically apply the changes and go grab a lunch or, like, a break, and then come back and just see, uh, the verification report once you're back.

Gonna give it a minute because it's going to go through all the design flows, all the user flows, and then generate a report, so it takes a bit of time.

Let's see if I can pull up the repo.

Oh, shoot.

I think

the Wi-Fi is super slow, so...

But, uh, I think maybe in the meantime, I can have Giovanni come over on the stage and show you a cool way to annotate your webs- web pages instead of, like, just going and typing stuff on the web, on the web page that you're working on.

Hello.

Oh, the prime.

Oh.

Can we have... Okay.

Yeah, on the... Can I see the slides?

The slides? Yes.

Yeah, just give me one second. Hello. Hi, everyone. I'm Giovanni. Uh, I was thinking to... Imagine you are in to the, your design meeting just talking about the design modification that you wanna do to that website, and then automatically your perception agent, agent is able to modify the website. This is something that you can do right now with the workshop. And this, why? How? Using the Bee wearable devices. So

Bee is a device we have in Amazon, uh, you can buy. It's like in some bee.computer website. It's a wearable, it's your AI assistant. So it just have all... It has a microphone, so you can activate this with the button, and then it's gonna listen for your conversation. No worries, there is a L- L, uh, L- ID that's will be on if it's listen to your conversation, and then all your conversation are in your private cloud.

After the AI agent that is running Bee is able to gives you suggestion about whatever conversation you had. So what I did is like integrating with this, this agent, the Bee agent, with the perception agent that we had for the workshop. So getting the input, listening from a AI convers- from a design conversation. Uh, I don't see the video.

Yeah.

I'm gonna... Ah, this one. Yeah. Thank you.

You can achieve this with three simple step. So first of all, yes, you have to have a Bee. And I have one with me that you can use it right now, or we can have a test, a test account to download, uh, that you can use to get the conversation, a simulated conversation. Then you need to install the Bee CLI. It's a CLI that is able the... It's gonna allow you to get the data for your conversation, and it's in the workshop, the instruction to download the Bee CLI by just a npm command. Then you start the server

of the workshop, and then you start the bridge, listening from Bee conversation and applying that to the perception agent. Then you just enable your Bee device. The LED is on. You say something like, "Oh, I wanna change the title color to red. I wanna change the emoji. Instead of a brain emoji, I want a robot emoji." And on the right side, you will see your conversation, your transcript of the conversation, and even the most important points. Here you can see the

response API, the, the, the API response coming from the Bee. You can open the panel and check. And then you can just, like, select Apply, and here is gonna be the normal flow of the perception agent workshop. So it's gonna change the color with the color that you just said. It's gonna change the emoji. And after the modification of the website are applied, it's gonna run even for you automatically the

verification phase on Nova Act. So we have a test account that you can use with a preloaded conversation, preloaded design conversation, that is like just talking about, "Oh, we change the color, we change the emoji," something like that. And you can, after you install the CLI, and you log it in with a Bee account, with a test device, with a test account, you just click on the bottom button, the bot- the button at the bottom. Uh, this is the verification page you can check.

But otherwise, if you wanna test the Bee device, we can, like, you do... We can, uh, just use mine, and we show you around if you want it. Like, we can actually sim- like have a conversation about the design, and you can see that it's actually applying this after, like, two or three minutes of elaborating the conversation. The Wi-Fi is a little bit slow, so that's why I didn't do the live demo. Uh, yeah, and then in the end, there it, it, there is the verification report that you can check. The Bee devices are also our prizes for the, for the challenge, so yeah.

You're gonna have one if you win the challenge. And we also are at the booth today and all the week. So yeah, let me know if you wanna integrate Bee, and I'm here for you.

So Bee device, if you want to come Up on the stage. Um, we have 10 minutes for you to, like, explore these two primitives. Think of an idea that you want to probably use it, and if you wanna come up, you'll get one of these devices. Um, so yeah. The, if ... Just get creative, I guess. Hey, everyone. I have Jed with me, who built something really cool, and he

would like to show it to everyone. Jed, come up on the sta- stage.

Yeah, hi.

Yeah.

Hi.

There you go. Jed.

All right.

Oh.

Yeah, yeah.

Yeah.

What am I doing? Where am I going?

Oh, I called Jed, but yeah.

Oh, I, I-

Yeah

... my name is Jeff, so I thought I was-

You are... Sorry. I'll have Jeffrey next-

Okay

... to show something cool as well.

Oh, okay. All right, this, this was just goofy, but it was actually an, an, an interesting, uh, uh, little funnel here is, is, uh, with this annotators going through and-

Can you put that in the

Oh, you can't see it.

Change it to a duck podcast.

There you go.

There we go.

So just turn... asked, asked it to turn it retro, NES, uh, do some duck jokes, and, uh, went back and did the verification funnel. Um, it's probably, uh, as what I was telling them, is probably r- r- really interesting to me here is to watch this, this, uh, great AI said, "I made it a duck." Uh, you know, "I, I made a duck icon," but, uh, y- often, as everyone here likely knows, is you don't

actually see that happen. But then it, it went through and did the, the entire verification funnel, came back with the reports saying, "Yep, it's done. I did it. I actually did it." So that was cool.

Can you scroll through the history so we can-

Oh, yeah

... what else

it did?

I had it, I had it just do, asked it to do some jokes.

Which it's pretty bad at.

That's amazing.

There you go.

Oh, because you came up with a cool demo. Would you like to step forward?

Oh.

As promised, you win this-

Oh, thank you

... Bee device.

Up next, we have Jeffrey, who wants to come and present what he did.

You can just...

I have a, I don't-

Yeah, so I'm still building a test harness to actually be able to show this, 'cause the default Bee account has a, a sample conversation that isn't relevant to the thing. So this is just, this is just something trying to, uh, bring the Bee to life. So the idea was to have access to the Bee CLI, um, and collect the summary of a, of a conversation, which was, uh, a discussion of, of, uh, backlog features that would be added to a, uh, repo that you're working on, right? So you're working on a

specific, uh, GitHub repository that's, that's, that, that's cloned, and then, um, uh, what's gonna happen is your conversation is gonna be used in context with, uh, all the code that's, that's there. It's gonna build a backlog based on the summary, and it's going to use the Atlassian MCP to put that into Jira. So that's the, uh, that's the, uh, idea here. And so the key thing that I'm just building right now in the background is, uh, the, uh, the test harness to insert a new

conversation that's relevant to the, the repo. So that's it.

Pretty simple and, uh, yeah. Anyway.

That's awesome. Thank you so much.

Yeah, thanks. Thanks. Great. Thanks very much.

So here you win your Bee device.

Great.

I know you wanted to go and get it, but-

Yeah, yeah

... you won it.

Thank you very much.

So, oh.

You have a question?

Giovanni told me-

Ryan, who's gonna present next.

Hello, everyone. Excited to be here.

Let me see. Start mirroring. So I was, I was late to this session, uh, because the other session ended at 3:00. So, um, I came up to, uh, Giovanni. Hold on me me, Shift + B key right here.

Oh, perfect. That's fine. So I came to Giovanni. Um, currently wearing a Bee hat. I currently attend, uh, Wharton, uh, Business School. My cluster's mascot is a bee. I am one of three people who care that the mascot's a bee. Um, so you can imagine my excitement when I came to this conference and saw we were presenting about a Bee device. So naturally, being late, being a Wharton grad, I went and asked if I could have a free Bee device. Uh, Giovanni said, "Do you have an

idea to share?" So I, I'm currently working at an AI upskilling startup Um, I am not a huge fan of Linear. Apologies to anyone who works at Linear. I just need a better way of understanding what should I be doing any time during the day, going from capturing an idea, discussing with my co-founder, and then updating a project repository. So this is a natural use case for the Bee device. So we have a internal tool called Sidekick. Emphasis on internal because there is no

design. Um, in my co-founder's defense, he did tell me to, uh, tell Claude to make it look pretty, no mistakes, and this is the best it did, so, um. But how it works is we have a, a joint inbox. Um, you can ignore the Go Touch Crash SF. I just see all these billboards and, um, it gets me concerned. Uh, that we send emails jointly to, uh, [REDACTED], and these are all the different

inputs we have, um, thoughts throughout the week, newsletters. I don't know if anyone here s-subscribes to an AI newsletter. Um, see something interesting, we send an inbox. And we are big believers in getting things done, which was a, uh, mode for knowledge workers organizing their projects, tasks, developed by David Allen in twenty ten. And oh, if the Wi-Fi works, you could be able to see it. Um, sorry, that did just work off the screen.

One second. So the getting, the getting things done tree shows all the projects, and under each project it has, uh, the strategy, has what the, what completion looks like, tasks, next tasks, references. And we've built an MCP so we can sit and talk to Claude and strategize on, "Hey, this is what we're thinking on this project. What do you think we should be doing?" All to solve the need of when you're working after you have a meeting of like, "What should I be doing right now to tie to my ultimate

strategy?" So how we imagine this to work with the, uh, Bee, uh, device, as Giovanni, uh, informed me, there's an MCP, would be capture our different meetings or different ideas, have it sent to our capture inbox, and then use that to update our strategy. Um, I wish this was, was loading because... Let me see. Oh, that's why. My phone's... See, that worked. Um,

yes. You know, I'm now O for eleven for live demos. Um, but that is, that is my concept of the Bee device. I'm looking forward to using it and sharing on LinkedIn how much I love the Bees. Uh, go Bees. Buzz buzz.

Thank you.

How would you, uh, go and integrate the perception agents into your, uh-

Oh, I-- actually, I have an answer to that too.

Okay.

So as I've actually emailed myself and Sidekick to discuss later, my co-founder, is, uh, we developed a lot of product offerings b- such as Sidekick, um, different tools for learning, um, the whole mission of making sure everyone knows how to adjust to the upcoming workforce displacement. But the bottleneck right now is product design, of building something that users can understand exactly what to do. So we would use the annotation feature of the perception agent to then, um,

send that description either through Sidekick, through our own, like, agent we have running to, to make improvements, but then to close that loop of development.

Oh.

So yeah. So if AGI would love to sponsor Ossian- ... um, I will give you my card, and we can talk later. So thank you.

Awesome. Thank you so much for coming up and demoing. And here, as promised, a Bee device.

Thank you. I appreciate it.

So as we are wrapping this up, um, just, Emile, I'm just gonna...

We are gonna put up a survey on the bo- on the screen. Uh, it's something that we...

Yes?

So the time's up.

You want to-- Oh, okay.

Like one minute.

Okay. There is, uh, Jonathan who wants to present as well.

Uh, so mine's more of like a concept than... I didn't have time to build everything, um, but I'll just kind of walk through it.

All right, so I was kind of looking at the, uh, the proxy for the Bee and trying to figure out how are we getting the information from the CLI. Oh, it's not, it's not sharing. Oh, there we go. All right. So I was trying to figure out, uh, how the proxy actually works and how we're getting information from the Bee, and, uh, besides just doing transcripts, uh, transcription, could you pull different data from different sources? Could you pull it from an IoT device for whatever reason? Um,

and what I started to come up with is, uh, at the company I'm at, which is a finance company, we have a lot of non-technical people, and not only do we have to get our, uh, criteria in, but we also have like a l- lot of UI, UX, um, that we have to try and capture, and we don't always get it in like the best details, uh, for acceptance criteria. So I've been pushing the team to use, uh, Jam MCP, which basically allows you to do a recording, captures all the

console logs, captures the network requests, and then, uh, feed that information. Um, we also have-- I built an AR- AI harness, so we can pipe this information to it. So really the architecture kind of looks like this, where very similar to like the, um- The CLI, you could actually take the Jam MCP, pipe that information into the proxy, have the LLM go out to, um, Nova, and then actually take the UI, uh, implementation.

So if you're saying things like, "Oh, I wish this, uh, would say, you know, uh, this investor schedule needs to go like this, um, or this score should be, uh, a research level score and be in a different location," uh, we'd be able to actually pull that information from the MCP, send it over to Nova, and actually make the changes on the AI harness that is running, uh, locally. So you could do the proxy the same way, um, but through a different device.

Um, did some initial tests. It looks like it works even through just some mocking, uh, but wasn't able, with the Wi-Fi, to get, uh, the fig-- or the Jam MCP up and running. But that'll be the test when I get home. So, thanks.

Thank you so much, and-

Appreciate it.

There you go.

So right now, we have

fifteen more minutes that will close the workshop. Um, we will put up a QR code on screen. If you feel that you have a great idea and you don't wanna come up and talk to us about it, you can submit these ideas here, and we'll do a... Oh, there it is. We will sub-- we will vote on those ideas, and the top three ideas will get a Bee

device. So if you have an idea, please put it in those, the poll.

Just, uh, give me a second.

There you go.

You guys see that? Yeah.

So we want them to scan the QR code and enter-

Yes

... an innovative idea using what we've learned today.

What would you use this perception be, all the things that you saw today, all the things you played around with today? What do you think would be a good use case for you that will help you make your life easier on a everyday basis?

We saw a bunch of ideas now, s- and if you think you had one, you're feeling a little shy, there's a chance to win another device.

So what do you think? Five minutes, and then we'll-

Five minutes, and then we will-

This site.

Yep.

Cool. Voting's closed. Ah.

Let me fix that.

Okay, I stopped the poll now. So all of them, great ideas. I was reading through them, and I was discussing with Emile there, like, so hard to pick. Is, isn't it, Emile?

So, we have

the one with pixel judge. Can someone raise their hand who submitted that idea?

No one? Pixel judge... Uh, what's that?

That one.

Okay. Um, the next one was the agentic end-to-end testing for a product. Okay. Come on up here.

Do you want to have it?

Here you go.

Thank you. I'm very excited to, to use this. Yeah.

Sweet.

Awesome. Awesome. Awesome to hear that. Thank you.

The reframe a l- a life thinking canvas for brainstorming and planning.

Yep.

That's you. Come on up here.

Come on down.

So we had decided we'll give away three, and

one was not claimed, so we have -- we did actually circle down to four. The last one is build a canvas app where a human perception agent work on a UI design together in real time. Come on up here.

Thank you, everyone.

A huge round of applause for all the winners here.

Can we go to the last slide?

Like, the thank you slide.

Thank you slide.

Okay, so I just want to wrap the session up with, uh, something like a key takeaway. The first, the most important thing of this workshop, you can all imagine, is the way human and agent collaboration is happening is changing. So we need tools that will make the visual piece of how the, um, human and agents are collaborating, and this is what a step next to building

perception, perception agent is. So you can get started with Nova Act, if you haven't already s- you know, looked at the GitHub repo. It's available. You can scan this QR code. It's an SDK. And also we have a blog post by our team that you can look it up explaining how, what we are thinking about perception agent at large. So with that, thank you everyone, and we also have a feedback, um,

QR code up. If you think you like the session, you like Emile, you liked me, you like Ji- Giovanni here, who presented, please leave us a feedback. We appreciate all your presence here. Thank you so much.