The Spatial Harness: Bringing Agents to the Canvas — Max Drake, tldraw

Max Drake, a product engineer working on agents at tldraw, opens by giving an agent a job before the talk has properly begun: find a specification linked from an email and a Notion document, then build it in the local tldraw desktop app. The request quietly contains the whole challenge. The agent must cross applications, recover the right context, translate a written specification into spatial behavior, and keep working while the presentation continues. The result will return near the end—and it will not be a clean demo victory. 0:13

tldraw is three things here: a free infinite-canvas whiteboard, the London company that makes it, and, most importantly for this story, the SDK underneath it. Selection, resizing, arrows, viewport behavior, and matrix math are difficult enough that teams building a slide designer, whiteboard, or other spatial product can spend their time rebuilding the canvas instead of their actual application. The SDK packages those mechanics so a product can begin above that layer. 1:20

LLM applications reproduced the same trap one level higher. A team could imagine an agent rearranging objects in space, yet still lack established ways to represent the scene, expose useful actions, or show what the agent was doing. tldraw already supplied multiplayer primitives—live synchronization, cursors, selections, and viewports—which suggested that a canvas built for human collaborators could also become a shared place for human and agent work. 3:19

Coding agents provide the contrast. Their prompts, source files, searches, and generated code are largely text, close to the medium on which language models were trained. Two-dimensional editing adds geometry and visual state: which objects exist, where they sit, what is currently visible, and how an action will alter the scene. Even a request to align interface elements can fail unless the surrounding harness translates between model-friendly representations and spatial operations. Drake calls the text-in/text-out account an oversimplification, but it explains why the canvas needs substantial extra engineering. 4:19

The first project, Teach, tackles perception and action before autonomy. The prompt is simple: “make the mouse blow out the candle.” The canvas does not contain a privileged semantic object called a mouse; it contains ordinary shapes arranged so that a person recognizes one. The system supplies both a screenshot and underlying canvas data, allowing the model to combine visual interpretation with structured information about the objects. 5:20

The observable change is small but instructive. The model interprets the drawn mouse and candle from the rendered scene and canvas data, then produces actions that add wind and smoke in the appropriate positions. The edit reads as the mouse blowing out the candle. This requires more than recognizing the drawing: the model must connect its canvas actions to their effect on the composition. In this run, wind appears first and smoke follows, with positions Drake considers correct. 5:31

What information has to cross the model–canvas boundary for that edit to work? The diagram makes the paired representation and action flow visible. The screenshot supplies appearance and composition; canvas data supplies explicit object state; the harness turns the model’s intended change back into spatial actions. Teach performs this in one prompt, so it demonstrates grounded editing rather than a persistent agent that can search, plan, and recover over multiple steps.

The tldraw Agent Starter Kit wraps that perception-and-action capability in a harness that supports multi-step work. Drake describes its code as MIT licensed. Instead of responding once, the agent can create goals, maintain todos, change its own viewport, inspect another part of the canvas, and continue toward a result. 6:31

The example begins with a cat and a request: somewhere else on the canvas are potential friends; bring one to the cat, whose favorite color is red. The agent needs to look beyond the initial view. It creates intermediate goals and changes its viewport to inspect the area containing potential friends, working toward the request. The demonstrated progress establishes planning and navigation, without confirming that a friend was selected or brought back. Viewport movement becomes the spatial equivalent of a coding agent searching a repository to find where something is defined. 7:01

This changes the role of the harness. Teach translates one instruction into one edit; the Starter Kit supports repeated observation and action through three mechanisms:

• Planning: The agent creates todos that break the request into intermediate goals.
• Active perception: It changes its viewport to reveal information elsewhere on the canvas.
• Execution: The harness gives the agent canvas actions it can use while pursuing those goals.

The model still needs spatial grounding, but it can now seek information beyond the initial frame.

The next experiment, Fairies, starts from the canvas’s multiplayer character. People should be able to work alongside agents, and agents should be able to work with one another. Each agent therefore appears as a visible character on the board. A fairy can be grabbed and thrown, assigned a hat or color, and adjusted with an intentionally ridiculous leg slider. The whimsy does real interface work: when several agents share a canvas, appearance provides a quick identity cue. 8:18

A single fairy can receive a task such as drawing a cat, but the more important interaction appears when several are selected together. The selection opens a group chat, turning spatial grouping into a way to choose a team. Drake asks the group to prepare a memo containing financial data for fiscal year 2025. One fairy begins planning, while the others become available for assigned work. 9:18

The animations are “cute and funny,” but they also act as status indicators. With ten agents running, a person should not need to open ten chat logs merely to discover that one agent is working, another is waiting, and another is bored. The board makes identity, location, and current state visible at once. This is where the canvas becomes more than a place to display agent output: it becomes an operations interface for supervising concurrent work. 10:18

How does a selected group become coordinated work? One fairy acts as orchestrator: it defines and assigns a task, waits while another fairy performs it, receives a notification when the work finishes, and is prompted to review it. The live demonstration briefly appears stalled by connectivity, then resumes with one fairy having created the task and another working on it. The diagram exposes the handoff that the character states make visible.

Fairies reveal their main architectural limitation next: they are “trapped in the canvas.” Building with them requires opting into a canvas-native harness, which makes it harder to reach files, repositories, pull requests, and other systems outside the board. The next experiments preserve the spatial coordination interface while letting existing coding agents perform work in their normal environments. 11:20

The motivation comes from tldraw’s launch process. As deadlines tighten, the team abandons ordinary task tracking and makes a large dependency graph on a regular tldraw board. Arrows show what blocks what; color shows completed work; the whole board exposes what is in progress and what can happen next. Its value comes from combining movable, informal shapes with a high-level view of project topology. 12:20

The Tech Tree app makes each node executable. A task is also a coding agent that can be started and allowed to work autonomously. In the demonstration, one node has finished gesture controls for a multimodal canvas application. Drake opens the resulting pull request, inspects the result briefly, merges it, and the graph marks that task complete. The visible state change—from an active task node to a completed one—comes from repository work performed behind the interface. 13:20

New work can also begin spatially. Drake draws directly on the canvas, wraps the drawing as a task called “facial animation canvas control,” assigns it to Claude, and presses Run. This treats a sketch or selected region as the task’s input rather than requiring the user to translate everything into a separate chat. The graph remains a management layer above the agents: nodes carry work and status, while edges retain dependency information. 14:20

How does a task move from the board into repository work and back? The diagram follows the demonstrated route: a person starts a task node, a coding agent produces a pull request, and merging the result leads to a completed node. Multiplayer makes this a shared project space. A colleague can join, add or edit tasks, and inspect work already in progress, coordinating people with agents through the same graph.

The opening demo returns after thirteen minutes without the requested fluid simulation. That miss matters: access to email, a specification, files, and an editor does not guarantee timely completion. Rather than hiding it, Drake uses the unfinished run to explain the mechanism underneath. The tldraw desktop app runs locally and exposes its live editor instance through a server, allowing an agent such as Claude Code to write ordinary JavaScript against the editor. The canvas becomes a programmable environment rather than a sealed agent application. 15:20

This arrangement preserves the agent’s access to the wider computer. In one colleague’s experiment, rectangles drawn in tldraw become controls for arranging real desktop windows. The coding agent creates the canvas geometry and uses an external automation mechanism to move the applications; Drake suggests AppleScript as a possibility but does not establish the implementation. The important connection is supported: the canvas supplies an ephemeral spatial interface, while the coding agent reaches beyond it to operate the desktop. 16:28

The most gleefully excessive example is Pong played with desktop windows. tldraw runs in a corner while actual application windows act as game objects. It is silly in exactly the useful way: it demonstrates that canvas primitives can define an interface without confining the action to canvas shapes. A temporary spatial UI can control files, windows, or other resources available to the external agent. 16:59

What bridge lets a drawing become an action elsewhere on the computer? The editor server exposes programmable canvas state to the coding agent; the agent combines JavaScript against tldraw with whatever computer access its environment provides. The diagram separates those two connections. The editor connection changes canvas geometry, while the computer connection applies external actions. Instead of putting the entire agent harness inside the canvas, the existing agent gains a spatial interface it can script.

The opening agent did make partial progress: it accessed Gmail and found the Notion document and its specification. Implementation remained the intended next step in Drake’s account, and the simulation was still unfinished as the talk ended. The closing claim is an interface direction: the canvas can become a shared place where people and agents inspect one another’s work, coordinate spatially, and connect that visible work to real tools and data. 17:29