Robotics

Embodiment, tasks and physical feedback

A robot is a powered physical system that senses and acts to perform assigned tasks. It need not operate without people: teleoperation uses continuing human input, while autonomy delegates some perception, decisions, and execution to the machine. Autonomy therefore depends on the mission, environment, and human involvement, as described in the NIST autonomy framework. [4]

Embodiment is the way a robot’s body shapes what it can observe and change. An actuator converts supplied energy into physical force or motion; a sensor measures a physical quantity. An arm manipulates through its moving appendages, while a wheeled base changes location. A movement instruction usable by one body may be impossible for another. [5]

A control loop repeatedly measures and adjusts physical behavior. A policy selects actions from available information, using programmed rules or learned behavior. Requested motion, actual motion, and task progress are distinct: commanding closure does not establish that the block was grasped. Renewed observations determine the next decision. [9]

Grasping and transport are subgoals with physical preconditions: transport requires a retained load and an executable route. Agent Engineering explains the general subgoal vocabulary. In robotics, an action can also establish missing information—for example, opening a drawer before selecting an object—so planning must respond to observations produced during execution. [10]

Coordinates, frames and physical measurements

A point denotes a location; a displacement vector denotes a change in location. Vector components describe amounts along chosen unit directions. For example, a displacement of (3,4) metres has length 5 metres. Adding components requires the same frame and units. Translating the arrow does not change its displacement, whereas changing the origin changes a point’s coordinates. [11]

A coordinate frame has an origin and orthogonal unit axes. Orientation describes its axes relative to another frame; pose combines orientation and position. In robot spatial algebra, $R_{AB}$ converts B-components into A-components, and $t_{AB}$ locates B’s origin in A. Re-expressing coordinates does not move the physical object. [1]

Camera intrinsics describe projection into pixels, including focal lengths and principal point. Extrinsics relate physical frames. A pixel identifies a viewing ray, not a unique metric point: depth or a justified geometric constraint is still needed. Hand-eye calibration estimates camera-to-gripper geometry; robot kinematics then connects a measured camera-frame point to the base. [12]

Image resizing, cropping, and orientation must remain consistent with that geometry. Vision AI covers preprocessing coordinates. Calibration does not supply missing depth, and changing image resolution requires corresponding changes to focal lengths and principal point. [12]

Robot configuration, reachability and mechanical limits

A revolute joint rotates; a prismatic joint slides. Each independent coordinate contributes a degree of freedom. Linkage constraints can make joint coordinates dependent. A configuration specifies the whole posture; configuration space contains these possibilities. Workspace describes physical positions or poses, often of the end effector—the tool or gripper interacting with objects. Robot kinematics distinguishes these spaces. [2]

Forward kinematics computes tool pose from configuration; inverse kinematics seeks configurations for a requested pose. There may be multiple solutions or none. A path describes geometry; a trajectory adds timing. Reachability therefore leaves unanswered whether the whole robot clears obstacles and whether its actuators can execute the requested timing. [13]

Velocity is position’s rate of change; acceleration is velocity’s. Force changes translational motion, while torque is a turning effect. Mass and rotational inertia resist acceleration; rotational inertia also depends on mass distribution. Gravity and coupled links affect required input. A reachable movement can demand more torque than the actuators can supply. [13]

The Jacobian maps joint velocities to instantaneous tool velocity. Near a singularity, some tool directions require large joint velocities. At rank loss, directions outside its range become unavailable. This is directional loss of capability, not necessarily loss of all motion. [14]

Timed sensor observations

Proprioception measures the robot’s own condition; exteroception measures its surroundings. Operating range, resolution, and update frequency constrain useful sensing. Noise varies between samples; bias is a persistent offset. Occlusion hides a surface, while dropout leaves a measurement unavailable. Neither absence of a detection nor agreement between imperfect measurements establishes ground truth. [8][15]

An accelerometer does not directly provide gravity-free translational acceleration. Inertial navigation rotates body-frame readings into global coordinates and removes the gravity contribution under the sensor’s convention. Orientation error contaminates this correction. In an isolated constant-bias model, acceleration bias $b$ produces velocity error $bt$ and position error $bt^2/2$; fused estimators need not follow that isolated growth law. [15]

Acquisition time identifies when a measurement was taken; arrival time identifies when software received it. Joint values within one JointState message must describe the same measurement time. Camera, joint, and force messages can arrive together while describing different moments. Spatial calibration and temporal alignment are separate requirements. [17][19]

A moving camera needs the transform appropriate to its measurement timestamp. A tf2 lookup requests a source-to-target relationship at a specified time. The latest available transform can answer a different temporal question. Waiting may obtain delayed history; it cannot reconstruct discarded history. Treat an unavailable transform as missing evidence rather than silently substituting another time. [20]

State estimation and uncertainty

State estimation infers quantities needed for action—pose, velocity, object location, or contact—from measurements over time. A belief represents uncertainty over possible states. Prediction advances that belief through a motion model; correction weights possibilities by how well they explain a new measurement, then normalizes. World Models explains why observations may reveal only part of the state. [21]

The Kalman filter is exact for a specified linear Gaussian state-space model with its conditional-independence assumptions. It represents belief by a mean and covariance. Prediction propagates uncertainty and adds process noise; the measurement residual—the observed value minus its prediction—corrects the mean through a gain that accounts for measurement uncertainty. Nonlinear extensions introduce additional approximations. [22]

One Gaussian cannot represent separated pose hypotheses. Weighted particles can retain alternatives, although finite sampling can miss a plausible one. Localization estimates pose relative to a map; mapping estimates the environment; simultaneous localization and mapping, or SLAM, estimates both. Drift is accumulated estimation error, not necessarily an obvious sensor failure. [21][22]

Observability concerns which state or calibration differences measurements can distinguish. More samples cannot resolve differences that the chosen motion never exposes. Visual-inertial calibration can require motion exciting multiple axes; degenerate motion leaves parameters unresolved. Agreeing sensors can also share an erroneous calibration assumption, so agreement alone is insufficient. [23]

Calibration estimates relationships such as camera-to-IMU geometry and timing offset. Kalibr uses informative rotational and translational motion and examines fitted residuals. Repeating nearly identical poses cannot replace informative motion. [19]

An action can improve knowledge before advancing the task. Opening a drawer reveals contents; moving an occluding object exposes hidden space. Such information-gathering actions have physical costs and preconditions. Their observations can justify a revised plan when acting immediately on the current estimate would be poorly supported. [10]

Executable action contracts

A setpoint is a desired value for a controller to track. Action vectors require more than dimensions: their components need units, reference conventions, timing, limits, and result semantics. A controller may track requested joint values without establishing that a grasp survived or an object reached its destination. [6]

Absolute and relative actions are not interchangeable. ALOHA’s ACT policy predicts absolute joint-position targets for lower-level controllers. The dynamics-randomization puck-pushing study instead supplies offsets from current joint angles. Sending the same numeric vector through these interfaces requests different motion. [24][25]

Command authority also needs an owner. In ros2_control, command interfaces are exclusively claimed, while state interfaces can be shared. Hardware can reject a proposed mode switch. Reading a measurement therefore differs from having authority to command a motor; unrelated hardware access paths require their own controls. [26]

Trajectory actions can report progress and enforce path or goal tolerances. A tolerance violation can abort execution and request holding position. Fire-and-forget topic submission lacks equivalent result notification. Neither a reported hold nor successful tracking independently verifies load support or object placement. [6]

Task planning and feasible motion

Task planning selects actions and preconditions. Sampling-based motion planning samples robot configurations and collision-checks connections; joining valid connections yields a route. Rapidly-exploring random trees (RRTs) extend existing branches toward samples for individual queries; roadmaps retain a connection network for reuse in an unchanged environment. Checks must cover the robot’s intervening geometry—the swept volume—not just endpoints: coarse checks can miss thin obstacles. [27]

Sampling-based search does not guarantee a solution within a finite budget. Trajectory optimization instead chooses states and controls while imposing dynamics and other constraints. Direct transcription makes intermediate states optimization variables; shooting obtains them by simulation. Obstacles can make the problem nonconvex, so finding one useful solution does not establish a globally best trajectory. [27][28]

A grasp can reach the block yet obstruct transport. That failure can require a different grasp, a changed task order, or new observations. The task decision and motion problem constrain each other. The diagram shows a proposed decision structure; its accepted-plan exit still precedes execution and outcome verification. [10]

Feedback tracking and actuator saturation

A reference is the desired behavior; tracking error is its difference from measured behavior. Disturbances push motion away from that reference. Overshoot passes the target, while stability concerns whether deviations remain controlled. PID combines immediate proportional correction, accumulated integral error, and derivative response to changing error; derivative action is commonly filtered because differentiation amplifies noise. [29]

Saturation caps applied actuation even when requested input grows. Continued integration then creates windup, prolonging recovery or overshoot. Anti-windup limits accumulation or uses the requested/applied discrepancy. Gains and tolerable delay depend on the closed loop; low average latency establishes neither stability nor a universal tuning rule. [29]

Predictive control and renewed measurements

Model predictive control, or MPC, optimizes a finite future sequence, applies its first input, obtains a new state estimate, and solves again. The horizon is the number of future steps planned. Feedback enters through renewed measurements even though each nominal optimization proposes an open-loop sequence. [30]

MPC requires an actionable estimate, suitable dynamics, meaningful objectives and constraints, and a feasible solution obtained in time. Nominal feasibility does not automatically establish continuing feasibility, stability, or robustness to model error. Replanning can replace an unexecuted future; it cannot undo an input already applied. [28]

Dynamics may come from physical equations or learning. A learned model predicts action consequences rather than selecting actions directly. Recursive predictions can compound error, and longer horizons need not improve decisions. Learning transitions and predicted rollouts explain those mechanisms; they remain inputs to physical control, not substitutes for renewed measurements. [31]

Contact, grasping and compliant motion

Contact introduces forces and motion constraints. A contact mode describes which contacts are active; transitions such as impact can change velocity and the applicable dynamics. Hybrid models combine continuous motion within a mode with guarded transitions and state resets. A free-space controller cannot simply assume its previous dynamics still apply after contact. [32]

Compliance means yielding under force. Impedance control shapes the relationship between motion and interaction forces, including stiffness, damping, and effective inertia. Stiffness relates displacement to force; damping opposes relative velocity. Force feedback regulates measured interaction. Hybrid control can separate motion along a surface from force normal to it. None supplies universally safe force or stiffness settings. [3]

Approach, contact, lift, retained grasp, and release need different evidence. In the GelSight experiment, deformation and relative marker/texture movement supplied slip clues after lifting. Detected slip prompted stopping, lowering, releasing, and another grasp with a higher contact threshold. This demonstrates contact evidence changing action; it does not establish universally safe retries or a sufficient stopping policy. [33]

Execution timing and concurrent control

Real-time correctness includes meeting a deadline, not merely computing a correct result quickly on average. Jitter is variation in timing. Observation age is elapsed time since acquisition. Multirate systems run sensing, task decisions, and actuator updates at different frequencies, so their handoffs need explicit validity conditions. Scheduling, blocking, allocation, and page faults can all disturb timing. [34]

Assume ten 100-bit messages on a 1 Mbit/s Controller Area Network (CAN) bus. Payload transmission takes 1 ms; serial execution with a 2 ms policy takes 3 ms. This arithmetic omits protocol and scheduling overhead. Pipelining can improve initiation cadence without removing that observation-to-command delay. [35]

Synchronization coordinates stage handoffs. Padding allows scheduling cushion but does not prove deadlines. Diagnosing motion therefore requires sample identities and timestamps alongside command timing. [38]

Quality of service, or QoS, controls communication behavior. Keep-last depth bounds retained samples, not their freshness. Lifespan expires samples; deadline reports expected message intervals; liveliness concerns publisher activity. Compatible settings enable communication. Missed-deadline callbacks report events but do not choose a physical response. Frame checks, timestamps, bounded queues, and late-result handling remain application responsibilities. [39]

def admit_proposal(proposal, observation, now, authority):
    # Application policy; this function does not stop the robot.
    if not observation.valid or not observation.frame_verified:
        return request_supervisory_response("invalid observation")
    if now > observation.valid_until or now > proposal.expires_at:
        return request_supervisory_response("expired evidence or command")
    if not authority.owns(proposal.command_interface):
        return reject("command interface not owned")
    return dispatch_for_execution_monitoring(proposal)

A watchdog detects missing expected updates and triggers a defined response. Nav2’s collision monitor can request stopping when sensor data expire, but command limiting still needs verified actuator response. Remote-service loss must leave an executable local response; sending cancellation is not that response. Local and On-Device AI covers sustained compute and thermal constraints. [40][34]

Learning actions from demonstrations

Teleoperation lets a person control the robot while observations and commands are recorded. Reusable episodes need task and operator identity, aligned observations, and interpretable action labels. The data engine described in Robotics: why now? tracks collection episodes and uses annotations to select training data; collecting motions alone is not the entire pipeline. [41]

Action-label semantics matter physically. ALOHA records leader-arm targets as actions and follower positions as observations; their displacement influences force through the motor controller. Recording follower positions as targets changes the supervision. A proposed deployment check should also verify that each policy input is actually available onboard, with matching frames, units, timestamps, and action intervals. [24]

Behavioral cloning fits a policy to expert observation–action examples. During execution, a mistaken action changes later observations, potentially leaving demonstrated states. DAgger addresses that mismatch by collecting learner-visited states, obtaining expert action labels, aggregating them with prior data, and retraining. Its supervision remains the expert action; its learning procedure does not supply protective hardware execution. [42]

Low held-out action error measures imitation on sampled examples, not complete task execution. Keep independent data boundaries and evaluate closed-loop behavior. Training a policy to choose actions also differs from learning transitions, where the target describes what happens after an action. [42][31]

Learning from interaction and physical outcomes

Reinforcement learning improves action selection using interaction rewards: numerical assessments of outcomes. Return accumulates rewards over a trajectory. Machine Learning Fundamentals explains the general mechanism. In robotics, substituting simulated trials for expert demonstrations exchanges some collection burden for uncertainty about transfer to the physical machine. [43]

A task reward must distinguish useful completion from shortcuts. A gripper-location reward assumes that the object remains held; a later drop can invalidate that proxy. SERL describes image-based success classifiers that policies can fool, and an option that updates the classifier using policy-visited states. Optimizing a judge is not independent verification of the physical outcome. [45]

Resetting object relocation means physically returning the object. Separate task and reset policies still perform that work. Collection records should distinguish completion, failure, and a time limit without treating every stopped rollout as success. Human assistance and repeated handling remain operational costs even when policy updates are automated. [45]

Offline learning avoids additional training interactions but cannot explore to fill missing high-value experience. Improving beyond recorded behavior requires judging actions and states that may be poorly represented. This distribution shift affects learned values, policies, and dynamics; the resulting policy still needs physical assessment. [44]

Language-conditioned policies and action chunks

A vision-language-action model, or VLA, uses visual observations, language instructions, and robot state to produce actions. The architecture described in Robotics: why now? connects semantic task decomposition to a faster continuous-action expert. Multimodal correspondence explains how input modalities relate; the physical action contract remains an additional obligation. [46][47]

Semantic transfer and motor skill are separate capabilities. RT-2 uses discretized action tokens and joint training on robot trajectories and vision-language tasks, but its web knowledge did not supply motions absent from robot training. Correct instruction interpretation therefore cannot establish the ability to perform the requested manipulation. [48]

Cross-embodiment training combines experience from different bodies. Open X-Embodiment maps heterogeneous observations and actions into usable training interfaces; a common format does not make mechanics equivalent. Dataset breadth, the mixture actually trained on, and evaluated robots are different quantities. GR00T N1’s embodiment-specific decoder illustrates adaptation at the output boundary, not proof that changing a decoder alone enables transfer. [49][50]

An action chunk predicts several future actions together. Diffusion Policy predicts a horizon $T_p$, executes a prefix $T_a$, then observes and replaces the remaining future. Longer prefixes reduce correction opportunities; shorter ones can reduce sequence consistency. Unlike MPC, it samples a learned action distribution rather than solving an explicit dynamics-constrained task optimization at every update. [51]

Even a conversational robot requires an execution layer beyond language generation. The Reachy Mini application dispatches movement and camera requests and suppresses speaker echo in microphone input. These integration responsibilities are distinct from the model’s ability to produce a reply. [52]

Physical protection and command authority

A hazard can cause harm; risk assessment also considers exposure, severity, likelihood, and avoidance. Safeguarding uses protective measures such as guards or presence sensing. An interlock connects a condition, such as access, to permission for operation. Assessment must include startup, maintenance, faults, and stopping performance—not only normal motion. Detection delay, braking, and load behavior determine the physical result. [7]

A routine pause, application velocity limit, and protective stop have different contracts. Nav2’s collision monitor can limit commands separately from trajectory planning. A warning outside the command path cannot perform that intervention. Software limiting is not a safety-rated emergency stop; sensing coverage, delivery, braking, and actual response need separate validation. [40]

In the documented Universal Robots implementation, safeguard stopping can retain power for monitored standstill, while emergency stopping removes joint power after stopping and engages brakes. Their reset and restart requirements differ. These are product-specific behaviors. Publishing zero velocity implements neither the safety inputs nor their monitoring and restart interlocks. [53]

Emergency stopping complements hazard prevention. The UR16e procedure requires identifying and resolving the reason for activation before manual reset and resumption. Power removal is not a universal safe response for gravity-loaded mechanisms: load support and the actual brake behavior must remain part of the assessed application. [54]

Runtime assurance separates advanced control from a decision mechanism governing permitted actuation. Black-Box Simplex retains an accepted backup sequence rather than assuming every new fallback proposal is safe. Its guarantee requires an initially safe backup, correct decision module, and modeled disturbance bounds. Switching must preserve an executable continuation before recoverability is lost; maintaining safety does not imply task completion. [55]

Fallback activation can fail through resource conflicts. ros2_control attempts configured fallbacks after controller errors, but required interfaces must be available. Deactivating a controller does not specify braking distance, holding force, or gravity response. A handoff that removes obsolete commands while leaving no controller able to support the load is incomplete. [56]

Recovery and authorized resumption

Recovery chooses a new action from the present physical situation. Waiting, retreating, and regrasping have different consequences and preconditions. Nav2 exposes distinct recovery behaviors with collision checks; it does not make any behavior universally appropriate. The proposed recovery flow below combines such explicit selection with fresh evidence and command authority; it is not a validated workcell procedure. [57]

Clearing a sensor, resetting a stop, and restarting motion are separate events. Before a new attempt, the proposed design discards obsolete goals and re-establishes relevant object, load, and human state. It bounds reattempts and distinguishes requesting operator help from an operator accepting responsibility. Unknown readiness keeps resumption blocked. [53][57]

Simulation-to-real transfer

Sim-to-real transfers behavior developed or assessed in simulation to physical execution. The reality gap consists of consequential differences in sensing, dynamics, timing, or environment. A shared interface can simplify reuse: the K-Scale presentation describes switching simulation and hardware endpoints through the same control interface. Interface parity alone does not establish matching physical behavior. [58]

System identification estimates model parameters from measurements. For known robot structure, a workflow records joint positions and torques, estimates derivatives with filtering, stacks dynamics equations, and fits parameter combinations. Informative excitation distinguishes competing parameter effects; additional samples cannot identify an inherently unresolved combination. Physical constraints and prior estimates can restrict implausible fits. [59]

Fitting and validation need separate measurements. Compare predicted or simulated outputs against an independent dataset with relevant frequency content and consistent preprocessing. A model that fits its estimation records can overfit those records. Held-out agreement supports the assessed input–output conditions, not complete robot operation. [60]

Residuals are measured outputs minus one-step predictions. Temporal correlation can expose unexplained dynamics; correlation with past inputs can expose omitted input-driven behavior. Interpretation depends on the model and feedback conditions. Even good one-step residuals do not establish accurate recursive simulation, where predictions become later inputs. [61][59]

Domain randomization varies simulated conditions rather than fitting one setting. Visual randomization can vary lighting, textures, camera positions, and distractors. Its demonstrated perception-and-grasp transfer does not establish contact-dynamics coverage. Identification and randomization are complementary: measurements constrain plausible conditions, while varied training can expose the policy to differences within those conditions. [62]

Dynamics randomization can vary masses, damping, friction, controller gains, observation noise, and action intervals. The Fetch puck-pushing study used these variations and physical tests with motion-capture observations. That evidence concerns its task and sensing setup. A simplified variation in action intervals does not cover every communication fault, contact condition, or hardware failure. [25]

Mocks can still test valuable failures, including injected sensor states and missing command feedback. Their coverage is whatever the test implements and exercises. Host timing characterizes that host and load; omitted backlash, saturation, contact, or gravity response remain untested. Learned environment models add another approximation, explained in World Models. [64]

Evidence of complete robot performance

Selected demonstrations show possible behavior, not its outcome distribution. The GR00T presentation acknowledges failed attempts behind a showcased scene. A proposed evaluation progression therefore moves from component checks and simulation to controlled hardware trials and supervised operation. Each stage must state the robots, objects, starting conditions, and disturbances it actually covered. [65]

Intermediate and final outcomes answer different questions. CloudChef describes evaluating cooking-state estimates, motion speed, and final food quality separately, including blind taste tests. Accurate perception and fast movement cannot substitute for acceptable output. Its line-cooking workflow also retains human replenishment and excludes other kitchen tasks, making operational scope part of the usefulness claim. [66][67]

Compare a candidate with the existing controller or operating process under matched conditions. Predefine the response and measurement window, repeat fresh trials, and randomize order to avoid aligning treatments with warming or wear. Repeated samples from one unchanged trial are not independent replications. Evals develops coverage; metric definitions preserve denominators. [68]

Timing and contact can interact: added delay may have a different effect under different stiffness settings. Crossing two settings of each factor requires four combinations. Comparing the delay effect within each stiffness setting reveals that interaction. Four single observations alone cannot estimate trial variability; replication and controlled assignment remain necessary. [70][68]

A lower confidence bound on completion rate and an upper bound on deadline-miss rate address population quantities under stated sampling assumptions. Confidence describes repeated-sampling coverage, not a guarantee for each run. Finite successful trials cannot establish that rare hazards are absent. Uncertainty in scores explains interpretation and comparison. [69]

Bind each proposed evaluation record to hardware, calibration, controller, policy, and software versions. Link measurements to the estimates and commands that used them, then to observed physical effects. Categorized deployment failures can guide revised simulation cases. Failure analysis helps separate the observed fault from competing explanations before changing the system. [71]

Open questions

1. 01

   Reliable transfer needs to separate instruction understanding, object variation, and changes of body. Shared datasets mix observation and action conventions, making the source of improvement difficult to isolate. Progress would include held-out embodiment tests with explicit adaptation data, interface changes, and repeated physical outcomes. [49]

2. 02

   Uncertainty must remain useful when sensors share calibration errors or motion fails to expose an unknown parameter. This matters because confident estimates drive consequential actions. Progress would show that information-gathering motions resolve identified ambiguities and improve subsequent execution under independently assessed calibration shifts. [23]

3. 03

   Runtime assurance remains difficult when perception becomes stale while load support requires uninterrupted actuation. Progress requires demonstrating executable fallback access, bounded switching delay, and measured physical response under sensor, solver, and controller faults—not merely an accepted fallback configuration. [55]

4. 04

   Contact recovery must handle objects whose friction, deformation, or grasp geometry changes during execution. A detected slip does not uniquely determine the next safe action. Progress would compare bounded recovery policies across held-out objects while separately recording retained loads, damage, interventions, and eventual task completion. [33]