Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab

“If my agent works fifty percent of the time, it doesn't work at all.” Emile Baizel opens the technical case for perception agents with a customer's practical objection: debugging unreliable automation can consume the time it was supposed to save. At Amazon AGI Lab, where Baizel leads a solutions architecture team, customer engagements feed both product feedback and evaluations used in model training. Shruti Arora turns that concern into the workshop's concrete task: make a website change, then check the rendered result.

Three priorities frame the work:

• Collaboration: an agent should understand the user's objective and communicate how it plans to achieve it.
• Perception: it needs to understand the environment where the work happens, beginning with browsers and extending toward other visual screens.
• Reliability: successful automation must survive real tasks without handing its savings back through debugging. Baizel describes reinforcement learning in simulated environments informed by external customers and internal Amazon teams.

Amazon Nova Act supplies the browser automation layer. Natural-language instructions drive navigation, search and form filling, with human involvement available for approvals or situations where the agent gets stuck. The use cases presented include filling forms across different websites, searching and extracting information, shopping and QA. The attraction of form filling is avoiding a separate deterministic script for every site's layout.

The AWS console exposes execution trajectories and bounding boxes around elements the agent examines or acts on. A custom computer-use model performs inference; an MCP server and a skill make Nova Act available to other agents. These interfaces serve different needs: the browser performs the work, while trajectories make its actions inspectable when the outcome needs explanation.

Computer use can start with an instruction as simple as clicking a blue button in the right corner. Perception adds a harder ambition: understand what the screen means well enough to work in an unfamiliar interface. Baizel's analogy is an old Yahoo or Hotmail email client. Someone who has never used that particular client can often work out how to send an email because years of screen use supply transferable expectations about controls and workflows.

That ambition has two practical consequences. First, screen-based work can reach applications without an API for the task. Second, the screen becomes a shared reference for collaboration. A person can point at an element and ask for a red background instead of writing paragraphs to identify its location. Human-like understanding remains the direction of travel here; the workshop demonstrates a narrower combination of visual feedback capture and browser checking.

Two open-source primitives divide that work. Visual annotation captures requested changes and exports a JSON file plus a prompt for a coding agent. The export is deterministic: it performs no model inference and is not tied to a particular model. Visual verification gives the coding agent a way to check the site against design artifacts and workflows described in Gherkin files, such as login or checkout flows. Capturing an instruction and judging its implementation are separate jobs.

Arora grounds the primitives in a familiar frontend review. An engineer builds from Figma designs while coordinating with another team. Reviewers run the site locally, notice that a heading's color feels wrong and leave comments in Slack or a shared document. The team still has to agree which feedback to accept; afterward, someone must translate those decisions into code changes. Passing the accepted feedback directly to an agent aims to reduce that translation work. The suggested reduction from days to hours is a workshop scenario, rather than a measured productivity result.

How can a reviewer identify what should change without describing the whole page? The Chrome extension offers three parallel annotation methods:

• Draw on the page: mark the area associated with the feedback.
• Pick an element: select the element being discussed.
• Drop a pin: place a location marker, like a pin on a map, and attach the annotation.

The annotations can be downloaded for a teammate or supplied with the generated agent-friendly prompt. Exporting the feedback alone does not edit the codebase; the coding agent must still be instructed to act.

Verification needs a reference for what the page should do and look like. Arora describes checking colors, font sizes and user flows against design files. On its first run, the UI verification tool performs a one-level crawl, examines links and content, and generates user flows in Gherkin syntax for subsequent verification. That supplies a starting set of flows from the existing site; its scope is limited by what the crawl discovers.

A verification run reads the design documents, specifications and flows, opens a browser, exercises those flows and generates a report. This gives the feedback process an observable ending: a person can inspect a report about the rendered application after the code changes. The workshop's next challenge is to connect those two primitives so that collecting feedback, applying it and checking it no longer require separate manual handoffs.

The hands-on scenario starts with a podcast landing page and accepted peer feedback: the hero title is too large, the accent color lacks impact and some buttons are oversized. The intended result is a connected workflow: annotate the page, press a button in the extension, let the coding agent apply the changes, run UI verification and receive a report. The available pieces are the Chrome extension, UI verification skill, Nova Act SDK and Nova Act MCP server.

The workshop setup requires a Nova Act API key, Chrome, an AI coding agent, Node.js, npm, Python and Git. The demonstrated Nova Act MCP server runs on macOS or Linux. Arora also identifies uv/uvx as part of installation, with pip offered as an alternative. These requirements belong to the workshop environment described in the recording.

Participants choose between an easy path with explicit wiring instructions, estimated at fifteen minutes, and a medium path with more integration work left to solve, estimated at thirty to thirty-five minutes. An installation question also clarifies an easy source of confusion: the Nova Act skill and the UI verification skill are two distinct skills, even when both are listed for the same coding client.

The missing connection is straightforward: a browser extension and a running coding agent do not automatically know how to send work to one another. Agent Bridge takes the annotations and prompts the coding agent to apply them. The workshop adds an Apply Changes button to initiate that handoff; the standalone annotation primitive does not include it.

The extension changes are concentrated in two files:

• manifest.json: add host permissions for the localhost connection used by the integration.
• content.js: add the button and its wiring, start verification and expose the generated report.

There is a consequential implementation distinction in the walkthrough: although the earlier explanation describes invoking the verification skill, this button integration runs verification through the Nova Act SDK.

Where does the new control enter the existing extension? Arora points to the marked insertion location in content.js, around line 261 in the workshop copy, near the code that appends the Export button. The supplied block creates the Apply Changes control and connects it to verification and a clickable report. The useful distinction is between exporting feedback for later use and adding an action that carries the feedback into the editing workflow.

What crosses each connection, and what supplies the verification criteria? The diagram separates the annotation handoff from the design documents and user flows used to judge the resulting site. A code change starts the next stage; the specifications supply information to that stage.

For the live example, Arora selects a published-date element, asks to change it to June 29, saves the annotation and presses Apply Changes. The integration opens the coding agent in the background. Agent Bridge had been tested with Claude Code and Kiro; using Cursor or Codex would require editing the bridge to accept and route the button's requests. Support for a skill in a client therefore does not by itself establish support for this particular bridge.

The workflow exposes CLI logs and Nova Act execution logs, including the screen regions involved in decisions. Arora observes the button change to Run Verification and explains that the run traverses design and user flows before producing a report. Slow Wi-Fi interrupts the walkthrough before a completed result for the date edit is established, and the session moves to another feedback source.

Giovanni's Bee integration changes how feedback enters the workflow. Instead of selecting an element and typing a request, a team discusses design changes aloud. Bee is presented as a wearable assistant with a microphone activated by a button and an LED indicating listening. The integration retrieves conversation data and feeds it into the workshop's perception-agent workflow.

The setup uses a Bee device or a test account with a simulated conversation, the Bee CLI to retrieve conversation data, the workshop server and a bridge listening for Bee conversations. This preserves the downstream editing machinery while replacing the feedback input. A preloaded design conversation lets participants try the connection without first recording their own meeting.

What can a person inspect before those words become edits? In the demonstrated flow, a panel presents the conversation transcript, important points and Bee's API response. The example requests two concrete changes: make the title red and replace a brain emoji with a robot emoji. Selecting Apply sends those requests through the normal workshop flow, changes the site and then runs Nova Act verification.

The Apply step remains a human handoff between captured conversation and code changes. Giovanni describes roughly two or three minutes of conversation processing, but uses the prepared demonstration because of slow Wi-Fi rather than completing a live wearable run. The example shows how a new feedback channel can feed the same edit-and-check sequence; it does not establish unattended interpretation of an arbitrary design meeting.

An audience example turns the podcast page into something deliberately goofy: a duck podcast with retro NES styling and duck jokes. The participant sends the requests through the annotator and then runs the verification workflow. The joke quality is debatable—“pretty bad,” in the participant's assessment—but the icon request exposes a useful failure mode in coding-agent work.

What changed after the agent said it had made a duck icon? The participant distinguishes the coding agent's completion claim from the subsequent verification report, which he says confirmed the change. His familiar frustration is that an agent can announce an edit that never visibly appears. Here, checking the rendered site supplied a second observation after the code edit. The account supports a successful workshop example, without establishing a general detection rate or the report's individual assertions.

The next participant, introduced as Jeffrey, proposes using a Bee conversation summary to build a development backlog. The intended input combines a discussion of requested features with the code in a locally cloned GitHub repository. That repository context would help turn the discussion into relevant backlog items, and the Atlassian MCP would put them into Jira. The test harness was still being built: the default sample conversation did not match the repository, so a relevant conversation first had to be inserted.

Ryan's existing internal tool, Sidekick, addresses a related question: after collecting ideas and discussing projects, what should someone work on now? A shared inbox captures thoughts, emails and newsletter material. A Getting Things Done-style project tree organizes strategy, what completion looks like, tasks, next tasks and references. An MCP connection lets the founders discuss that project context with Claude and ask how their next actions should serve the strategy.

The proposed Bee connection would send meetings and ideas into Sidekick's capture inbox and use them to update strategy. Loading problems prevent a complete live demonstration. Arora then asks the important integration question: where do perception agents fit? Ryan identifies product design as the bottleneck and proposes using visual annotations to send descriptions through Sidekick or the team's own agent to improve their products. Conversation capture would supply planning context; visual annotation would supply concrete UI feedback.

Jonathan's concept returns to a specific frontend problem: nontechnical colleagues can identify a UI problem without supplying detailed acceptance criteria. A recording can preserve what they were looking at and saying. He describes Jam MCP as a source of recordings, console logs and network requests, with an existing local AI harness available to consume that information.

The proposed path sends Jam MCP information into a proxy, has an LLM call Nova and routes the requested UI implementation through the local harness. His examples concern changing an investor schedule or relocating a score. Compared with a bare text request, the recording and diagnostics could carry more of the circumstances surrounding the feedback. This remained a concept with initial mocked tests; Wi-Fi prevented getting the actual Jam MCP connection running during the workshop.

The final audience exercise broadens the possibilities again. Selected ideas include agentic end-to-end product testing, a thinking canvas for brainstorming and planning, and a canvas where a human and a perception agent work on UI design together in real time. These are proposed uses, rather than additional demonstrated integrations. The shared canvas idea makes the collaboration ambition especially concrete: the interface itself becomes a place where both parties can contribute.

Arora closes on the need for tools that support the visual part of human-agent collaboration. The workshop gives that direction a small, usable shape: identify something on the page, express the desired change, carry it to a coding agent and inspect the rendered outcome. Annotation makes the request easier to convey. Browser verification gives the team something to examine after the agent says the work is done.