The Autonomous Computer: Infrastructure for Computer Use Agents — Ang Li, Simular

“Post the photo to LinkedIn.” Ang Li, Simular’s co-founder and CEO, sends a conference selfie to an agent from his phone. The agent receives the image and starts asking questions. Behind the conversation sits a full Windows desktop, where the agent can use the mouse and keyboard to operate applications. Li leaves the interaction there; the opening demonstrates the handoff, rather than a completed publication.

That desktop is what gives computer use its reach. An API agent needs an exposed programmatic operation. A computer use agent can work through the interface people already use, including applications without suitable APIs. Its actions also leave a visible trail: windows open, pages change and fields receive input.

A recording of four named agents expands the idea from one assistant to several workers. Their jobs include product QA, research and reading customer-support information in Discord. The apparent pace needs care: hours of activity have been compressed into thirty seconds. Multiple desktops make concurrent work possible; the montage does not establish that individual agents work quickly.

Li brings a particular bias to this problem. Competitive programming, neural-network research and work on general intelligence lead him toward a neurosymbolic system: neural models combined with code. The rest of the workshop develops why an autonomous computer might need both—and why giving the agent a computer becomes an infrastructure problem of its own.

An early experiment makes the perception problem concrete. GPT-4V is asked to locate the iMessage icon on a screen. Iterative refinement eventually gets there: the first attempt is wrong, the second lands on the mailbox icon, and the third reaches iMessage. Asking for Zoom fails altogether. A successful click in one demonstration is a weak foundation for a system expected to navigate an entire computer.

The failure motivates a useful split in “autonomous computer.” Autonomy covers understanding the environment, planning, reasoning and recovering from mistakes without constant supervision. Computer covers the machine on which those decisions become actions. Simular initially concentrates on autonomy, assuming that existing computers will supply the second half.

Two development tracks follow. A coding layer aims to control the computer more reliably, while a research effort works on making vision-language models better at using it. Both need observations of the environment. Screenshots provide visual information; accessibility trees provide structured descriptions of interface elements through operating-system-specific protocols. Browser DOM trees offer another structured view within web applications.

Li compares these observations to the different sensors on a self-driving car: the steering wheel has become a mouse and keyboard. The practical question is which signal makes the next action identifiable. A model that struggles to infer a button’s location from pixels may still select it from a structured list.

Agent S is the model-agnostic harness built around these observations and actions. Its purpose is to organize computer use while allowing foundation models to change. The first generation, completed in late 2024, relied on accessibility information because visual grounding was still unreliable.

The first grounding mechanism converts interface elements into a linear list with IDs. An LLM receives the structured description and chooses the ID of the button to click. This changes the immediate problem from finding a location in an image to selecting an identified element. Li reports 20% success on OSWorld, compared with roughly 5% for the benchmark’s early-2024 starting point, and a best-paper award at the ICLR 2025 Agentic AI workshop.

The next generations change different parts of the system:

• Vision and multiple models. In early 2025, the second generation drops the accessibility tree for a vision-only approach, organized into manager and worker layers. It combines foundation models, including a seven-billion-parameter open-source model. Li presents the result as evidence that a smaller specialist can improve a larger model’s computer use within the same harness.
• Multiple attempts at inference time. The third generation, in late 2025, explores running several agents on the same task. This spends additional inference effort on alternative attempts rather than relying on one trajectory.

Does parallelism help merely because it spends more resources? The presented chart accounts for total resource budget as workers multiplied by steps. Li reports that ten-agent runs scale better in success rate under that comparison. The relationship to inspect is how the budget is distributed: several attempts can outperform putting the available steps into fewer attempts. The recording does not explain the mechanism used to choose among their results.

Li reports that Agent S exceeded the human reference performance on OSWorld in December 2025. He immediately limits the interpretation: a research-benchmark result does not establish better-than-human performance in everyday computer work. He also describes a newer OSWorld benchmark where the best result is only 20%. The harder evaluation reopens the problem rather than closing it.

A recorded CalHacks project turns Agent S into a computer that can take a FaceTime call. The user is away from a training machine and asks it to fix a CUDA out-of-memory error in VS Code. The agent opens VS Code through Command-Space, inspects the file and identifies an oversized tensor from the comments. It proposes reducing the tensor size. That is a concrete example of a conversational request becoming desktop navigation and a code-level diagnosis; the excerpt does not establish a successful training rerun.

Customers ask a stricter question: can the same workflow run on a server every day without someone watching and debugging it? Li distinguishes two success criteria:

• Pass at K: at least one of K attempts succeeds. This is useful when a developer can keep trying until a working program appears.
• Pass to the power of K: all K runs must succeed. This captures the expectation behind an unattended recurring workflow.

In the described evaluation, running the same agents three times reduces the all-runs-successful result. Varying the queries and environment makes performance worse again. The task has changed from finding a good trajectory to producing that trajectory consistently.

Clarification offers one improvement before execution begins. Ask the agent to identify ambiguity in the request and obtain the minimum clarification needed while preserving the task’s intent. This reduces the need to guess what the user meant. How much does that help repeated execution? In the presented comparison, dashed curves represent the original behavior and solid curves represent behavior with clarification. Li reports improved reliability, with the largest improvement among the shown systems coming from an Agent S configuration.

Clarification improves the input, but it does not remove the reliability question. A model-agnostic harness can benefit from new models and still produce varying behavior across runs. That gap leads back to the other development track: preserving successful work in a symbolic form.

SimuLang supplies the scripting layer. Li positions it alongside the code-based automation familiar from robotic process automation, with Playwright as a browser-world reference for a shared developer interface. The goal is to write computer workflows as source code, run them from a terminal and make them available to a coding agent such as Claude Code.

The live request is simple: visit the AI Engineer website and summarize the events. With SimuLang and its paired skill installed, Claude Code checks the setup and starts writing JavaScript using the protocol. The generated script opens the website on the local computer; a virtual machine is another deployment option. Execution produces errors, and Claude Code revises the code in response. The feedback is operational: the coding agent receives a failure from running its program and changes the program.

What information does a summarization script need to capture? Claude Code chooses an accessibility snapshot instead of a screenshot, then returns an events summary. Li opens the resulting code and describes three operations: open a window, take the snapshot and write it to a file. The useful distinction is between reading structured page information and interpreting pixels; this task can use the former.

The demonstration reduces computer use to a familiar coding-agent pattern: generate a program, execute it, inspect errors and repair it. It also produces a reusable artifact. The remaining question is how that artifact should behave when a later run encounters a different environment.

Li connects the two approaches to the stability–plasticity dilemma from continual learning. A useful system must retain the ability to repeat tasks it already knows while adapting to new circumstances. Fixed automation favors repeatability, but changes in the environment can break its assumptions. A model-driven agent can respond to unfamiliar conditions, while its decisions may vary across runs.

The proposed combination gives each approach a different job. During an initial run, a vision-language agent works through the environment and records actions as code. Once the workflow succeeds, a subsequent run tries that code first. If the environment changes and the code stops working, model-driven recovery supplies the flexibility needed to proceed. Li calls this an initial experiment, explicitly saying the reliability problem is not solved.

Product testing provides a nearby application: a computer agent can click through a dashboard and exercise its interface. Li describes a CalHacks project that used the agent to test a product, then switches to demonstrating the replay mechanism itself. In Sai’s Agent settings, the experimental option is labeled “Simular Flow code replay.”

The requested task is to find the first speaker on Tuesday on the AI Engineer website. The attempt stalls before the workflow gets going. Li restarts the agent and checks whether the VM is awake. That interruption matters: workflow reasoning depends on an available computer and a functioning agent session before any saved action can help.

While the live agent recovers, a recorded example shows Simular agents entering information into a traditional ERP on Microsoft Windows 365 for Agents VMs. The opportunity is specific: older Windows applications often lack modern APIs, yet still contain essential business processes. Li gives insurance-site data retrieval and healthcare data entry as examples of vertical applications that developers could build on the infrastructure.

Back in the live session, Li removes an existing skill and asks again for Tuesday’s first speaker. The activity log shows the agent writing automation code almost line by line, to be assembled into a file. It navigates to the schedule and retrieves the information through an accessibility tree. Saving the session as a skill preserves the code that performed the task, so the next request has an executable starting point.

What changes when the question asks for Wednesday instead? Replay mode appears, and the saved code’s day parameter changes from its Tuesday default to Wednesday. A filtering step then hits a hiccup. Li describes the run as finding the information, but also notes that it switches into neural mode—the vision-language agent—after the interruption. The result illustrates reuse with recovery, rather than an entirely uninterrupted script replay.

The important branch is what happens after each code execution. Verification checks the environment; a mismatch stops the expected path and calls for fresh exploration. The diagram separates the saved-code route from the recovery route. It answers why a parameterized script can be useful even when it cannot finish every run alone.

Li’s analogy is a school playbook: write down the steps, follow them next time, and think harder when the environment no longer permits the expected action. The economic benefit comes from skipping repeated reasoning on familiar portions. His example is that perhaps 50% of a later workflow could run as code without an LLM request; that is an illustration of the proposed saving, not a measured saving for this schedule demo. He also describes experimenting with a $500 unlimited-task plan as the product explores alternatives to charging for repeated model use.

The second half of “autonomous computer” returns through a mundane complaint: an agent operating the foreground desktop prevents its owner from using that computer. Once a workflow becomes useful, users ask to move it to a separate machine or server. Then they ask for more machines. This makes cloud desktops part of the product’s usefulness, rather than merely a hosting choice.

The product separates three responsibilities. Sai is the interface; the Sai agent performs planning and decision-making; Sai Box is the cloud computer on which it operates. Li says he has twenty machines running in the background. Developer access through a CLI and API extends the same remote-computer operations beyond the web portal, allowing another product to control the machine.

The platform vision spans different devices. Windows and macOS support are described in the session, with Linux and Android presented as forthcoming. Internal Android examples exercise capabilities that a desktop-only view would miss:

• Dragging in a game. An Angry Birds example demonstrates a drag gesture rather than just discrete clicks.
• An authentication interaction. An example moves through an authenticator and an MFA challenge, fills forms and obtains a token. Li reports success while emphasizing caution around this capability.

These are internal demonstrations, rather than a claim that the forthcoming Android offering is already generally available.

Local machines can also join the fleet. The intended combination is one agent system that can use both owned and cloud computers, with neural reasoning and symbolic automation available on top. At the workshop’s close, Li names four developer offerings: the Sai interface, open-source Agent S, open-source SimuLang and SaiBox’s virtual-desktop access. He warns that the newly released developer access should be expected to have bugs.

The questions begin with spatial memory. Li says most memory is currently reduced to text. Screen recognition also has a specialist component: a separate seven-billion-parameter open-source model, sometimes fine-tuned, rather than only a general vision-language model. His explanation for the remaining screen-understanding gap is that foundation-model training may contain too few screenshots; he presents that as a possible cause, not an established diagnosis.

Li calls the execution approach “stateful programming”: run a line of code, look at the environment and account for the changed state. In this usage, the distinction is between generating a program and executing an automation while checking its effect on a live interface. The check is essential because a correct-looking action sequence can encounter a different screen from the one it assumes.

The same reasoning supports multiple observation modes. Screenshots, accessibility information and DOM information have different corner cases. An application can omit an element from its accessibility tree while leaving it visible in a screenshot. Conversely, structured text can make an exposed element easier to identify and match. The useful combination lets one representation cover failures in another.

The consequences become clear in Li’s banking example: an agent must not add an extra zero to a payment. That raises the value of independent checks and complementary signals. He also jokes that LinkedIn’s bot detection is a “Turing test for AGI.” The joke points to a separate complication: applications may actively resist automation, so interface recognition alone does not guarantee a usable workflow.

A hospital use case sharpens the accuracy requirement. Li describes checking a completed form by presenting the result back to a model for verification. If the verifier finds something wrong, its result goes back to the agent for another attempt. This gives the system a chance to detect errors after generation instead of trusting the first entry. Li reports substantial improvement from experiments, but the session does not establish a clinical error rate or a guarantee of perfect accuracy.

Structured matching provides another kind of check. In the replay approach Li describes, accessibility text allows exact matching of exposed interface information. That is useful for deciding whether an expected target or state is present. Exact text matching establishes that particular comparison; it does not by itself prove that the entire business process is correct. The proposed reliability comes from a system of generation, execution and verification.

Access control asks a different question. An audience member wants the agent restricted to Chrome and particular websites, without access to mail. Li separates two mechanisms:

• Action monitoring. A separate guardrail watches the agent and stops actions recognized as irreversible, asking for approval.
• Operating-system restrictions. A managed VM image can block applications or actions and apply an allowlist. Li presents this lower-level restriction work as an early direction under exploration.

A monitor deciding whether to approve an action and an OS preventing access have different responsibilities.

The guardrail itself follows the code-first design. Rules handle recognized cases. When rules do not capture the situation, the decision goes to an agent and then to a human for verification. That escalation gives uncertain cases somewhere to go; the recording does not establish how completely the detector recognizes irreversible actions.

Repeated runs are intended to add more known cases to the saved program. Li says branches sometimes appear as the system encounters different scenarios. The aspiration is eventually to reduce token consumption to zero for familiar work. That is a direction of travel: the earlier demonstration still needed model-driven recovery.

For a proprietary application, the first successful run may require explicit procedural knowledge. An audience member describes a long employee-and-payroll workflow: navigate to the right area, choose the right options and check particular boxes. One route is to supply the process as an SOP, or standard operating procedure. Li describes an experiment in which a coding agent received one SOP and completed the workflow in one shot. He immediately qualifies the result as part of a demo-stage technology with many corner cases still to solve.

Another proposed route is exploration in a controlled replica of the application. With guardrails that prevent harmful actions, agents could try different paths repeatedly and become familiar with the environment. The aim is to discover an initial successful trajectory without requiring the user to specify every button. This is a proposed extension, rather than a demonstrated replacement for detailed instructions.

A later audience proposal makes the idea spatial: store a map of interface paths so the agent can add a user without inspecting every screen again. Li identifies that with the coding layer’s job of encoding a trajectory. A stable legacy application is a favorable case. A changing website is harder, because the saved path can cease to work.

The audience member proposes sending a swarm of agents to re-explore when the map breaks. Li’s condition is the cost of exploration. If trying actions has no harmful consequence, agents can explore alternatives and map trajectories. A bank is different: trying arbitrary actions can have real costs. The limiting resource is therefore more than model tokens or compute; it includes what the environment permits an exploratory action to do.

Could enough interaction data simply train away the remaining visual problems? Li is skeptical about doing so economically under the current deep-learning approach. He points to the scale of visual data and jokes about needing a couple of trillion dollars. This is his judgment about the difficulty of the training route, rather than a derived cost estimate or proof that better visual learning is impossible.

The final question returns to the code artifact. A long workflow offers many places to fail; should it be divided into scripts of two or three steps? Li expects saved programs eventually to accumulate many if/else branches, while loops and other logic. In his envisioned system, agents would write and manage code that could become difficult for humans to maintain.

Splitting the workflow can simplify each piece, but it creates another design problem: define the smaller task, its inputs and its outputs. Li leaves granularity as a tradeoff rather than prescribing a fixed script length. Reusing a successful trajectory saves repeated discovery; keeping that trajectory useful as cases multiply requires deliberate choices about checks, branches and where one piece of work ends.