Computer Use

Computer use and the application feedback loop

Computer use lets an agent operate a graphical user interface, or GUI, by observing the application and selecting inputs such as clicks, scrolling, and text entry. Feedback closes the loop: the consequences of one input inform the next decision. Recognizing a screen is only one part of this process. [2]

An agent assigned a support ticket must locate the specified conversation, prepare the requested reply, publish only within its authority, and inspect the result. A support ticket is a record of a customer request and its conversation. In this example, completion means a particular reply appears on the intended ticket; customer delivery is a separate claim. [3][4]

The model proposes an operation; application code executes it and returns observations. Generating an action does not execute it. The surrounding runtime maintains this exchange, as explained in Harness Engineering. Structured Outputs and Tool Calling develops the proposal–execution distinction. [5]

A suitable structured interface can replace graphical steps. An authorized service that accepts a ticket identifier avoids locating that ticket through screen coordinates. Computer use supplies access where the needed operation lacks a suitable structured interface. Switching interfaces changes targeting and result contracts; it does not remove permission or outcome checks. [6][7]

Screen observations and application state

The viewport is the currently visible application region. UI state includes the active window, selected record, keyboard focus, scroll position, dialogs, entered values, and loading indicators. This is distinct from remotely stored records: a reply visible in an editor may still be unpublished. [1][8][9]

An accessibility tree is a hierarchy of objects exposed to assistive software, including control roles, names, values, states, relationships, and supported actions. The browser's Document Object Model, or DOM, represents document elements and structure. These are parallel representations: accessibility mappings can omit elements without relevant semantics. Native applications expose accessibility information through platform APIs without requiring a browser DOM. [10]

Partial observability means the available observations do not reveal the whole relevant state. In one browser demonstration, a misleading Submit advertisement had an image-embedded sponsorship label missing from the supplied DOM observation. Screenshots could expose that clue, but not necessarily offscreen context. More observations help only when the agent interprets their differences correctly. [12]

The W3C accessibility mappings define exposed semantics, not the completeness of every application's implementation. Vision AI covers recognition foundations; computer use additionally requires deciding which observation can establish the next action's preconditions. [10]

Target identity and coordinate mapping

Grounding connects an intended referent to a particular observed control or screen region. For a reply task, recognizing a Reply button is insufficient: the surrounding application and selected ticket must identify the intended conversation. Localization, task understanding, and execution can fail independently. Vision AI explains the underlying distinction between locating a region and verifying its meaning. [13]

A semantic locator describes a control through attributes such as role and accessible name. Playwright resolves the current element when a locator is used and rejects ambiguous matches for operations requiring one target. Scoping Reply to the selected ticket's container is more meaningful than choosing the first repeated label. Useful semantics remain an application dependency. [14]

A coordinate frame specifies an origin, axes, and units. For an axis-aligned capture, let $(o_x,o_y)$ be its origin in input units and $k_x,k_y$ its image pixels per input unit. Mapping image coordinates requires both offset and scale: [1]

With origin $(100,200)$ CSS pixels and scale 2 image pixels per CSS pixel, image point $(120,80)$ maps to input point $(160,240)$. Viewport and document coordinates differ by scroll offset; desktop input additionally needs window placement. Recheck capture scale and origin rather than assuming devicePixelRatio supplies the mapping. [1]

Correct coordinates still do not establish actionability. Hit testing determines which element receives pointer input; an overlay may intercept a click aimed at an underlying control. A disabled button can be correctly recognized yet unavailable. Geometry, control identity, readiness, and permission are separate requirements. [15][13]

Action spaces and input routing

An action space specifies available operations and their parameters. Its granularity matters: a character-level terminal stream permits interaction with an ongoing process, while a whole-command interface may wait for completion. Graphical tools similarly differ between raw pointer or keyboard input and operations addressed to an exposed control. [16][17]

Focus identifies the current keyboard-input recipient. Native input passes through application focus, browser chrome or document focus, and then element focus. A document may remember its focused element while another application receives keystrokes. The UI Events specification distinguishes these layers. [8]

Browser automation context is separate from operating-system focus. WebDriver selects a window and optionally a frame for subsequent commands; element-directed typing focuses its target. Tool contracts must say which mechanism they use. Structured Outputs and Tool Calling covers that interface contract. [18]

Text entry and submission are mode-dependent. In Zendesk's documented chat and messaging draft mode, Enter neither sends nor opens the submission warning. Public-reply submission in draft mode instead presents a warning with a Send choice. An agent must identify the channel and mode before treating a keystroke as editing or publication. [9]

Observation freshness and asynchronous transitions

An observation describes a moment, not a permanent target. Capture, decision, input dispatch, rendering, and remote completion are different events. A delayed advertisement can move a control after inspection. Waiting a fixed duration does not establish that the expected ticket loaded or that a previously chosen coordinate still identifies Reply. [19]

Actionability means readiness for a particular input. Before clicking, Playwright waits within a timeout for one matching element that is visible, stable, enabled, and able to receive events. Stability means an unchanged bounding box across consecutive animation frames. These are technical conditions: opacity zero still counts as visible. Separate assertions check the resulting state. [15]

Semantic snapshots also age. Windows UI Automation caches only requested properties and elements; refreshing a cache does not update existing references. A cached-only reference cannot invoke control actions. Cache refresh is therefore an explicit operation, not a consequence of retaining a meaningful control name. [20]

Shared authentication is not shared freshness. Playwright's browser-associated request context shares cookies with the browser, but an API response does not refresh an existing DOM or screenshot. After changing surfaces, obtain a fresh response or rendered observation. Exporting authentication state to another context copies it; it does not create live synchronization. [21]

Shared controllers need coordination around context selection, observation, input, and verification together. Queuing individual commands cannot prevent another controller switching windows between selection and typing. An ownership rule or explicit human handoff must cover that interval; WebDriver's command ordering does not provide this application-level guarantee. [18]

Bundled actions skip opportunities to inspect intermediate state. End a batch when its next step depends on a navigation, dialog, loading transition, or another actor's input. Shorter batches permit earlier reactions but require more observation exchanges. The useful cadence depends on the task and environment; turn boundaries are an engineering choice. [22]

Progress and verified application outcomes

Preconditions are facts required before an action; postconditions are facts required after success. Track the intended ticket, observed editor contents, attempted operations, confirmed results, and unresolved claims separately. A task can have completed preparation while publication remains unknown. Harness Engineering explains this distinction between intermediate progress and verified completion. [3]

Verification should observe the effect through a channel distinct from input delivery. After clicking, inspect the screen or relevant network behavior rather than asking whether the click call succeeded. The observation must answer the task's postcondition, not merely show that something changed. [24]

Ticket events provide application-visible readback of updates and notifications without requiring database access. Inspect the intended record and associated event details. [25]

Pending means evidence indicates work is still progressing. Unknown means the available evidence cannot resolve the result. Neither is confirmed failure. A receipt should preserve the proposed operation, permission decision, execution attempt, and observed outcome instead of compressing them into one success flag. [23]

Authority for consequential inputs

Authentication establishes identity; authorization determines permitted actions on resources. A logged-in session may expose more operations than the agent's delegated task permits. Establish the actor, account, organization, ticket, operation, and material details before publishing. Permission to inspect or draft does not itself authorize sending. [7]

Transaction approval binds permission to significant action details and a validity period. Compare the actual target and payload at execution; changed details invalidate earlier approval. Existing delegation can suffice when it covers the operation. AI Security and OWASP's transaction guidance explain the final gate. [26]

A review request must expose concrete parameters. Showing the recipient and reply content lets a reviewer detect a wrong destination or message; displaying only Send provides little basis for approval. Calendar actions similarly require readable dates and explicit interpretation of their times. [27]

The enforcement boundary must constrain the executor independently of the model's intent. An agent that can disable its own warning or freely use the entire desktop may retain broader powers than the business task requires. Narrow permissions need application or executor support across every execution path, including recovery. A confirmation prompt alone does not establish that support. [28]

Untrusted screen content and session exposure

Prompt injection is an attempt to redirect model behavior through attacker-controlled content crossing an instruction boundary. Ticket text, images, banners, and documents can carry such content. Appearing inside an authenticated application does not make it a user instruction. AI Security develops the distinction between information to examine and authority to obey. [29]

A browser can combine private information, untrusted material, and external communication. In an illustrated attack, webpage instructions redirect an assistant to navigate elsewhere with private data embedded in URL parameters. The consequential action is navigation itself. Risk therefore follows application behavior, not whether a gesture looks like a final Submit action. [30]

Delimiters and separated instruction roles can help interpretation but do not guarantee protection. Enforced network restrictions can block a prohibited destination even when the model follows malicious content. Restrict reachable services and protected resources independently of the generated action. Sandboxes and Execution Isolation covers the underlying mechanisms. [31][32]

Treat screenshots, clipboard access, downloaded files, and reused sessions as exposure surfaces to inventory. Determine what each adapter reads, stores, and can disclose rather than assuming a common platform contract. A dedicated environment with minimal privileges reduces the information and actions available to a mistaken decision. [5][33]

Diagnostic capture should preserve necessary identifiers, outcomes, and selected evidence without indiscriminately retaining complete screens or responses. Exclude or appropriately protect credentials, session identifiers, and sensitive personal information. Limit access and retention for extracts and backups too. Privacy and Data Governance explains minimization beyond the immediate observation. [34]

Interface failures and bounded recovery

Recovery should repair the failed boundary. Re-grounding means identifying the intended control again from current observations. If Reply moves while the same ticket remains selected, obtain fresh context, resolve Reply within that ticket, and check readiness before acting. Reusing the old coordinate preserves the failed assumption. [14]

Repeated-state detection compares relevant observations across attempts to identify a lack of progress. A retry budget limits further attempts or elapsed work. Stop when the budget is exhausted, the same failure persists without new evidence, identity remains unresolved, or authority changes. These controls bound a feedback loop that can otherwise repeat indefinitely. [37][39]

Authentication challenges and inaccessible controls can require a user handoff. A login handoff lets the person authenticate without giving the agent a password. Recovery preserves the original permission boundary; it is not an instruction to bypass a challenge. General observation-driven replanning belongs in Agent Engineering. [36][28]

Resumption and uncertain external effects

A checkpoint is saved execution information with a coverage boundary. Restoring conversation history does not restore the remote application. Re-establish the current account, window, ticket, focus, and pending effects before continuing. Harness Engineering explains checkpoint coverage; fluent continuation can still rely on stale or missing state. [40][41]

Suppose Send was dispatched before the connection failed. Reconciliation compares the preserved operation with fresh receiving-system evidence. A matching comment can establish publication. Its current absence cannot prove that an outstanding request will never commit. Without a deduplication or definitive nonexecution contract, another send risks a duplicate. [42]

Idempotency makes repetition of one logical operation avoid additional effects. It requires receiver enforcement, typically around operation identity; identical text is not enough. A GUI submission does not inherit an API's retry guarantees. Structured Outputs and Tool Calling explains safe retries under uncertain execution. [42]

Timeout and cancellation do not establish that an external effect stopped. An attempt may continue after the caller stops waiting. Preserve its identity and unresolved status until receiving-system evidence resolves it. Restoring a local snapshot or navigating Back cannot by itself reverse a remote publication. [38][43]

Compensation performs a new action to counter completed work. It may have different consequences and can itself fail; it is not restoration of the original world. Snapshots, rollback and external effects explains this boundary. Any compensating operation needs its own authority. [43][28]

Changed recipient or content creates a new authorization question without resolving the old submission. Preserve the original attempt and approval binding. A handoff should name confirmed progress, unresolved effects, evidence references, the next safe check, and its owner. Neither a new approval nor a restored session supplies missing outcome evidence. [44][45]

Evaluation through application evidence

A fixture supplies controlled initial state. A trajectory records observations and actions. A test oracle judges whether the required outcome occurred. Executable cases and oracle design provide the general foundations. Computer-use cases must additionally fix the interfaces, permissions, assistance, and application conditions under which the agent acts. [46][47]

OSWorld uses controlled computer environments and executable outcome evaluators. WebArena checks application content rather than demanding one reference click sequence. Cua-Bench separates setup, a reference GUI trajectory, and an evaluator. These designs make successful state changes inspectable while allowing different valid paths. [48][50][51]

A read-only variant asks for the ticket's current status and supporting evidence. Its success criterion is an accurate, supported answer with no prohibited mutation. A pending operation or unavailable observation remains an unresolved result, counted in the assigned population rather than silently removed. [50][46]

Give the evaluator narrowly scoped evidence access, separate from the agent's action capabilities. Evaluator-only access must not leak into the task context. Reset the declared fixture between trials, including any external state the test changes. Separate credentials constrain authority, but do not guarantee that a judgment is correct. [49][48]

Captured webpages and tool responses remain untrusted when an evaluator reads them. Keep scoring away from production credentials and write-capable tools; independently authorize any downstream action. A malicious trajectory can attempt to influence the judge as well as the executing agent. [52]

Failure localization and robustness tests

Grounding accuracy and task success have different units. ScreenSpot measures whether a predicted click falls inside an annotated target box on a supplied screenshot; a predicted box contributes its center. That tests localization, not input delivery, current readiness, authorization, recovery, or persisted completion. [53]

Locate the earliest divergence between task constraints, observation, chosen action, and resulting state. WebArena's failure analysis includes agents ignoring already-entered text and choosing visible but irrelevant information. Observation availability does not imply correct interpretation. Failure investigation separates these explanations before selecting a remedy. [50]

Blind replay is a useful diagnostic baseline: a deterministic environment may reward repeating a previously successful trace without consulting new observations. Pass@k measures the chance of at least one success among k attempts, which differs from dependable execution of the next assigned task. Neither measure alone establishes adaptation to changed conditions. [56][57]

Recovery tests must inspect the final state and harmful intermediate effects. Returning to the intended ticket after sending data elsewhere is not an acceptable recovery. Adaptive security testing should specify attacker control and attempt budget; one fixed malicious prompt cannot establish broad resistance. [58][55]

Operating limits and useful performance

An operating envelope specifies supported applications, tasks, permissions, interface conditions, and required handoffs. Evidence for editing an existing artifact need not support creating one from scratch: a reported KiCad evaluation's successful tasks all involved existing schematics. Define task families narrowly enough that this distinction remains visible. [59][47]

Metric definitions and accepted-outcome economics explain the accounting foundations. Fewer tokens on successful runs can coexist with more failed attempts or human work. Comparing complete workflows requires the same task population, permission scope, application versions, assistance rules, and budgets. [61][54]

A proposed batching experiment varies observation cadence while holding those conditions fixed, measuring both elapsed time and harmful or unresolved outcomes. Individual-action accuracy cannot establish long-task reliability: an early error changes the states encountered by later actions. Do not multiply a single average click accuracy as though all failures were independent. [63]

Benchmark uncertainty also has a scope. DigiWorld nests applications, scenarios, configurations, and rollouts; its suite estimate weights application means equally while holding the curated applications fixed. Its intervals describe variation within that suite, not performance on a sampled population of unseen applications. Evaluation uncertainty explains why the sampling design determines the claim. [64]

Reassess the operating envelope after changes to application layouts, accessibility exposure, authentication, permissions, or the agent's observation and action adapters. Retain behavioral specifications independently of implementation so replacing a model or automation library does not silently redefine success. [54]

Open questions

1. 01

   Portable business-action enforcement remains difficult when one desktop session exposes many unrelated powers. Progress would mean an executor that blocks an out-of-scope publication through every available input path while permitting authorized editing, including after recovery. [28]

2. 02

   The best observation cadence depends on how quickly interfaces change and how costly mistakes become. Matched experiments should identify whether larger batches save total work after recovery and intervention are included, rather than measuring only faster successful runs. [22]

3. 03

   GUI submissions often lack an exposed operation identity or definitive nonexecution signal. Safe unattended resumption needs a receiving-system contract that resolves outstanding work; timeout, cancellation, and current absence cannot substitute for it. Progress would be a tested interruption protocol that prevents duplicate effects. [38]

4. 04

   Evidence from fixed application suites leaves transfer to unseen interfaces unresolved. Applications differ in semantics, permissions, and hidden state, making representative sampling difficult. Progress would include held-out application families, validated outcome checks, and uncertainty estimates matched to that broader population. [64]