Agents on the Canvas in tldraw

What becomes possible when a whiteboard is also a programmable application surface? Steve Ruiz introduces tldraw as a free online whiteboard, a London startup and an SDK for building other products. He points to Replit’s Agent canvas and another integration he calls Luba AI, then draws a narrower distinction for Google Stitch: its annotation mode uses tldraw, rather than the main canvas he shows.

Building experiments is part of improving that SDK. The canvas runtime is built with React, and its contents are React components too. That makes the whiteboard a place to build new behavior, including interfaces where AI can interpret drawings, produce artifacts and eventually work alongside the person drawing.

Make Real starts with a direct interaction: draw an interface on the canvas, send it to a vision model and ask for a working prototype. Ruiz places the experiment in 2023, when vision models were newly available, before Lovable and before the term vibe coding. Its appeal was that someone who did not code could make a technical artifact without having to inspect its implementation.

The live request is simple: make the drawing interactive. The output is a working piece of HTML, although Ruiz immediately notes that it is not quite what he requested. The next interaction matters more than the initial generation: he annotates the rendered result, asks for green, then adds red and black color samples with an instruction to use those colors. The existing website and the marks around it jointly become the revision prompt. The user can point at the thing they want changed instead of translating every visual detail into prose.

Ruiz connects this pattern to Stitch, but the live revision does not satisfy him. A rendered panel appears beside the annotated original, with a green button visible in the result; he nevertheless concludes that it has not followed the requested changes and moves on. The demonstration establishes an interface for expressing revisions, not reliable execution of every instruction.

After briefly mentioning tldraw Computer as an experiment with chains of prompts, Ruiz turns to AI as a canvas collaborator. Requests can range from drawing a cat to producing a diagram, finishing slides or completing a graph. Instead of generating a raster image, the model emits structured text outputs that create the same circles and other shapes available to the human user. The result stays within the canvas’s editable object vocabulary.

The cat becomes a test of interpretation. Ruiz asks it to blow out a candle, then sends a separate request to draw a mouse elsewhere. Multiple prompts can operate on different parts of the canvas. The application itself has no concept of a candle, yet the model interprets the object and incorporates the requested action into the drawing. Semantic understanding comes from the model; the application supplies the shapes it can manipulate.

Producing predictable geometry is harder than producing plausible semantics. Ruiz attributes some of that difficulty to less vision training data and to conflicting spatial conventions. His concrete examples show why a direction is incomplete without its reference frame:

A model can recognize an object while applying the wrong convention when placing or modifying it. Ruiz initially calls the work training, then corrects himself: the intervention here was prompt engineering to make the model behave predictably.

The next experiment replaces a single generation with an agent loop. Asked to draw the life cycle of a butterfly, the agent produces output, reviews it and iterates until it judges the task complete. Its sidebar follows the coding-agent conventions familiar from Cursor: users can inspect its reported thinking and work, and reject results. The completed canvas diagram connects Egg, Larva (caterpillar), Pupa (chrysalis) and Adult (butterfly).

The diagram works, but the interaction still feels to Ruiz like handing his keyboard to an AI. That does not make the pattern useless. He cites Lovart and MagicPath as design applications using tldraw, and tutoring as another useful setting: an agent can help with homework or fill in an existing worksheet. To demonstrate contextual completion, he begins a D&D character sheet with his name and fields such as class and age, then asks the agent to complete it. As he turns toward the next experiment, the sheet assigns him the fighter class.

Fairies gives the agents a visible presence inside the workspace. These small animated characters can be picked up and thrown around; hold one too long and it reacts. They still accept familiar requests such as drawing a cat, but their positions and animations expose agent state in the same space where their work happens. Instead of monitoring several terminal windows or a composer interface, the user can see where each agent is acting relative to the others.

The agents can also observe one another’s work. Ruiz gives separate Fairies requests to add a hat and a neck to the same cat. Both can work with the shared drawing concurrently. This makes collaboration more concrete: the user sees agents modifying parts of a common artifact, rather than receiving a finished replacement from a sidebar.

A group request adds orchestration. Ruiz selects three Fairies and asks for more animals. The workflow proceeds through distinct responsibilities:

1. Select a leader for the group.
2. Have the leader scout the existing canvas.
3. Create a to-do list and delegate its tasks to the other agents.
4. Observe the resulting work and judge whether it is correct and complete.

Shared state is the difficult part. Ruiz describes agents as effectively blind while they are working, creating a risk that their efforts overlap. The leader’s role is supervision rather than drawing, but the demonstration does not specify a locking or conflict-resolution protocol.

Fairies serves as an accessible introduction to multiple agents, but the next task moves beyond animal drawings. Ruiz places a long description of an e-book app on the canvas, summons the Fairies and asks them to make its wireframes. They continue working while he introduces a more permissive way for agents to control the application.

Giving agents more access raises a different problem: what is safe to expose in a hackable application? tldraw already has a runtime API, and coding models can write against it. A generic code sandbox alone does not meet this experiment’s needs, however. The agent also needs the DOM, browser rendering and screenshots to see the effects of its changes.

Ruiz’s solution is an Electron wrapper around tldraw. He asks Claude to create a small HTTP server with an endpoint that treats posted content as JavaScript and executes it. That bridges the coding agent to the running editor, but it also grants arbitrary code execution. Ruiz explicitly warns against using this approach in an app, framing the offline, file-based desktop experiment as a risk the individual user accepts. Local ownership of that risk is not technical isolation or a guarantee of safety. While he explains the setup, the Fairies’ background task produces e-book reader wireframes.

One possible use is a round trip between code and a diagram: visualize the code, edit the diagram, then update the code to match. Ruiz describes that workflow but skips its demonstration. Instead, he draws a small interface with controls for leg length and T-shirt color and asks the agent to make it interactive. His point is that the prototype does not need to stop at the interaction primitives already exposed by the drawing tools: although he says tldraw lacks the relevant built-in hover and click primitives here, the agent can write code against the editor to supply behavior.

While the interactivity request runs, Ruiz describes how readily agents propose script injection and modifications to local software. He recounts a teammate, Max, asking Claude to remove podcasts from Spotify; in his account, the agent was willing to work through the application’s minified code or bundle. This is an anecdote about the agent’s willingness to modify software, rather than a demonstrated Spotify implementation.

The live canvas task returns something unexpected: a separate HTML site. It shows a character with a colored shirt, color choices and a long pointer that Ruiz objects is not even a slider. He wanted behavior inside tldraw, so he redirects the agent back to the canvas. The character also blinks, but that extra behavior does not establish that the requested controls work.

Ruiz sees broad potential in agents modifying desktop applications and speculates about similar access to websites. Yet the ongoing attempt illustrates a narrower result: an agent can enthusiastically change software while misunderstanding where the change should live and what interaction it should implement. More access expands the space of possible actions; it does not by itself align those actions with the request.

The requested interactivity remains unsuccessful. Ruiz first suggests releasing the experiment, then corrects himself that it is already released. He returns to his preference for local-first applications and files that matter independently of the app, before abandoning the live attempt with an invitation to catch him later for a working demonstration.

The unfinished example still gives those preferences a practical motivation. A local, file-based application can expose itself to Claude and other agents so they can script and modify it. In Ruiz’s framing, maximizing that agency requires letting users accept the corresponding risks themselves. He points to OpenClaw as an example of this attitude toward powerful, potentially dangerous tools.

Work continues on the canvas experiments. Ruiz invites people to explore Fairies and find behaviors that surprise them, as they have surprised him. He adds that the Fairies have IRC too, without demonstrating its protocol or architecture—another avenue for experimenting with how agents communicate while sharing a workspace.