Tell the Robot What You Want — Sandhya Subramani, AWS

Sandhya Subramani introduces Scout as her rover co-host, then moves from the presentation to its interface. Her opening request combines two actions in ordinary language: turn on the headlights and greet the audience. Scout takes a few seconds to respond. She describes a setup involving a Raspberry Pi, a SIM card, and a 4G connection. The demonstration depends on a networked command path.

Scout describes a presentation screen, stage lights, and audience seating. Subramani then asks how many people it sees, a question she says she never trained it to answer. Scout responds that it will inspect the front camera and reports two people: one near the speaker monitor at the back of the stage and another farther right. This demonstrates a conversational use of camera information beyond a specifically trained question-and-answer task. The count remains Scout’s reported interpretation, rather than an independently established measurement.

A request to spin 360 degrees adds movement to the exchange. After a delay, Subramani reports that Scout is turning and anticipates another description of its surroundings. The interaction brings together language, observation, and an existing movement capability, with a visible wait between the request and the reported action.

The mechanism behind Scout’s conversational behavior is an agent layer built with Strands Agents, which Subramani identifies as an open-source AWS framework. The robot already has actions it can perform. The added layer interprets a request and orchestrates the tools needed to carry it out. Her analogy is direct: just as a software agent receives software tools, it can receive a robot as a hardware tool exposing preset functions or programmable policies.

The agent’s decision is which policy to invoke and when. That makes a fixed collection of capabilities accessible through a wider range of natural-language requests. It does not itself supply missing physical skills. Subramani makes that boundary explicit in her setup example: a request to pick up a red cube should work only if the robot already has that capability.

She presents the agent wiring as five lines of code: import the Strands agent, make the robot tool available, assign it to the agent’s tools, and issue the task. The small integration surface depends on the robot tool and its underlying capabilities already existing. The red-cube example illustrates that interface; it is not a demonstrated grasping task.

Scout’s design includes three Strands agents. A thinker agent continually assesses the environment and considers what to do next. A communication agent connects to Telegram and a web app, allowing Subramani to converse with Scout and request actions. This separates ongoing environmental assessment from the channel through which a person gives instructions.

The third agent handles voice interaction, but Subramani has disabled it for the presentation. It would otherwise interpret her stage speech as speech directed at Scout and repeatedly interrupt her. Although she describes the three agents as designed to work together, the live configuration deliberately reduces that interaction. The practical issue is deciding when a person is addressing the robot, not simply whether the robot can understand speech.

Subramani says the package supports more than 40 robots across eight categories through robot tool calls. She organizes the architecture into four layers: the agent, the policy provider, the execution backend, and the physical hardware. The flow is bidirectional. Instructions descend toward the robot, while observations return upward to inform the agent.

At the top, the Strands agent receives a request and selects a policy. The policy provider supplies the learned behavior, and Subramani describes a training process that collects data, trains on it, and creates additional simulation data to produce a VLA model. The agent invokes the relevant policy for the question or command. This keeps task selection separate from the training that gives the robot an executable behavior.

The backend provides the environment or interface on which the policy runs, whether a simulation environment or a hardware environment. Physical output occurs at the robot. During this explanation, Subramani interrupts herself to discuss Scout falling and asks it to move back. She expects it to tolerate a fall and resume, but those expectations are not a demonstrated recovery guarantee. The interruption makes the physical endpoint of the architecture tangible: selecting a policy still leaves the robot to execute it under real conditions.

The proposed deployment places Strands Agents on both the cloud and the edge. Subramani describes cloud-side training of VLA models and policies using AgentCore, with direct edge-side invocation to make robot execution faster. Strands can decide which part to call. The motivation is to combine the capacity needed for substantial training with a shorter runtime path for physical actions; the talk does not provide latency measurements or a detailed placement rule.

She connects this arrangement to a learning loop: the robot collects information, that information can support further training, and the resulting system should still execute quickly at runtime. Her central division of responsibility is that the agent decides what to do and the policy decides how to do it. As she again checks whether Scout can recover its posture, that distinction also marks a limit: competent task selection does not establish reliable balance or movement.

Subramani contrasts robots programmed for a particular set of autonomous tasks with a possible future in which VLA models become as broad and capable as large language models. In that future, she suggests, a single model might provide enough capability to avoid repeated task-specific adaptation. This is a forecast about what policies could become, rather than a capability established by Scout.

Until then, she presents agents over existing policies as a useful intermediate step. Existing VLA models and policies still need adaptation, but an agent can make their capabilities accessible through a broader set of requests. Scout falls again during the discussion, and Subramani tries to get it upright and stopped before returning to the interface. The proposed future and the present robot’s limitations remain visibly separate.

Back in the demonstration, Subramani asks Scout to do something complex. Scout produces speech about a completed 360-degree turn and being safely onstage, then announces a signature performance. Subramani observes that it is not doing anything and asks it to act. The concrete change she reports is that it turns off its headlights. Fluent narration does not, in this exchange, establish completion of the requested performance.

The interface exposes a tool invocation labeled “rover speak,” which Subramani identifies as the function chosen in response to the request. She thinks Scout should also have attempted an action and attributes a fall to that attempt, though the precise action is not established. She has seen it perform a dance before, but that earlier behavior does not materialize reliably here. The useful distinction is between a selected tool, its verbal output, and a successfully completed physical task.

Scout also serves as a data-collection platform. Subramani can move it manually, direct its navigation, and create training episodes. These episodes can capture how it responds to questions and the reasoning it reports, providing material for subsequent improvement. Autonomous behavior and manual guidance both contribute to the experiment. Collecting those records is a concrete step toward training; it does not by itself demonstrate that the robot improves during the interaction.

In the configuration, Strands Agents supplies the harness SDK and an Anthropic Claude model supplies the language-model component; the exact model version is not reliably established. A system prompt describes what Scout should do, its rules, and the purposes of the available capabilities. Those descriptions help the agent decide which tool to invoke for a request. The voice component uses OpenAI Realtime, and Subramani says she has added safety and guardrail information, without detailing the enforcement mechanisms.

The final demonstration uses Telegram, which Subramani also uses to communicate with Scout when she is away from home. She asks it to spin, analyze the room, and identify the best-looking person. The playful request combines movement, perception, and a subjective judgment. She warns that the answer is unpredictable and may be biased. Scout acknowledges the request and appears to turn, but a follow-up request to spin farther does not produce the additional movement she expects.

Scout then reports that its spin is complete and estimates six to seven people, describing people near the front, others in the audience, and someone onstage. These are the robot’s scene descriptions, not a verified count or proof that the entire room was inspected. Its response to the appearance question includes a useful perceptual limitation: from its low, ground-level viewpoint, faces are difficult to judge. It instead awards style points to the person onstage for their stride and the blue curtain backdrop.

Subramani closes by joking that Scout can provide a confidence boost when she is the only person in the room, then thanks the audience. The final exchange demonstrates a remote conversational interface that can request physical action and return an interpretation of camera input, while the answer itself acknowledges that the robot’s viewpoint limits what it can assess.