# Agents on the Canvas in tldraw

Steve Ruiz

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/sPUjIBH5Cwg-agents-on-canvas-in-tldraw) · [Video](https://www.youtube.com/watch?v=sPUjIBH5Cwg)

## Agents on the Canvas in tldraw

From sketches that become HTML to agents that edit a shared canvas, tldraw’s experiments expose the practical gap between generating an artifact and collaborating on it.

### What can a programmable whiteboard become?

[0:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=34s)

What becomes possible when a whiteboard is also a programmable application surface? Steve Ruiz introduces tldraw as a free online whiteboard, a London startup and an [SDK](https://github.com/tldraw/tldraw) for building other products. He points to Replit’s Agent canvas and another integration he calls Luba AI, then draws a narrower distinction for Google Stitch: its annotation mode uses tldraw, rather than the main canvas he shows.

Building experiments is part of improving that SDK. The canvas runtime is built with React, and its contents are React components too. That makes the whiteboard a place to build new behavior, including interfaces where AI can interpret drawings, produce artifacts and eventually work alongside the person drawing.

### A sketch becomes a working prototype

[1:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=118s)

[Make Real](https://makereal.tldraw.com/) starts with a direct interaction: draw an interface on the canvas, send it to a vision model and ask for a working prototype. Ruiz places the experiment in 2023, when vision models were newly available, before Lovable and before the term vibe coding. Its appeal was that someone who did not code could make a technical artifact without having to inspect its implementation.

![A canvas panel titled Make Real shows a browser sketch, arrows, blue exclamation symbols, and a red Continue annotation.](https://ai.engineer/api/talk-essay-media/aa6bd1f83eaeee228d3d874a4eec2ec0cc5400f5ab4611948066989e68f33d04)

Make Real illustrated with a browser sketch and a looping annotation arrow.

### The result becomes part of the next prompt

[3:46](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=226s)

The live request is simple: make the drawing interactive. The output is a working piece of HTML, although Ruiz immediately notes that it is not quite what he requested. The next interaction matters more than the initial generation: he annotates the rendered result, asks for green, then adds red and black color samples with an instruction to use those colors. **The existing website and the marks around it jointly become the revision prompt.** The user can point at the thing they want changed instead of translating every visual detail into prose.

Ruiz connects this pattern to Stitch, but the live revision does not satisfy him. A rendered panel appears beside the annotated original, with a green button visible in the result; he nevertheless concludes that it has not followed the requested changes and moves on. The demonstration establishes an interface for expressing revisions, not reliable execution of every instruction.

![Two Ready to begin? panels appear side by side: the left has a dark button, drawn marks and color swatches; the right has a green button.](https://ai.engineer/api/talk-essay-media/616872ae3f6c70de5492fe7cd794119b5f4aa31bc038d5f9e39d85b6ae9acffd)

An annotated page sits beside a rendered version with a green button.

### Generate editable shapes, then resolve spatial ambiguity

[4:43](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=283s)

After briefly mentioning [tldraw Computer](https://computer.tldraw.com/) as an experiment with chains of prompts, Ruiz turns to AI as a canvas collaborator. Requests can range from drawing a cat to producing a diagram, finishing slides or completing a graph. Instead of generating a raster image, the model emits **structured text outputs** that create the same circles and other shapes available to the human user. The result stays within the canvas’s editable object vocabulary.

The cat becomes a test of interpretation. Ruiz asks it to blow out a candle, then sends a separate request to draw a mouse elsewhere. Multiple prompts can operate on different parts of the canvas. The application itself has no concept of a candle, yet the model interprets the object and incorporates the requested action into the drawing. Semantic understanding comes from the model; the application supplies the shapes it can manipulate.

Producing predictable geometry is harder than producing plausible semantics. Ruiz attributes some of that difficulty to less vision training data and to conflicting spatial conventions. His concrete examples show why a direction is incomplete without its reference frame:

| Convention | Meaning |
|---|---|
| Cartesian coordinates | Y increases upward |
| Browser coordinates | Y increases downward from the top-left origin |
| Viewer-relative left | Left from the viewer’s perspective |
| Stage left | Left from the performer’s perspective |

A model can recognize an object while applying the wrong convention when placing or modifying it. Ruiz initially calls the work training, then corrects himself: the intervention here was **prompt engineering** to make the model behave predictably.

### Iteration improves the artifact, but still hands over control

[7:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=454s)

The next experiment replaces a single generation with an **agent loop**. Asked to draw the life cycle of a butterfly, the agent produces output, reviews it and iterates until it judges the task complete. Its sidebar follows the coding-agent conventions familiar from Cursor: users can inspect its reported thinking and work, and reject results. The completed canvas diagram connects Egg, Larva (caterpillar), Pupa (chrysalis) and Adult (butterfly).

![A Butterfly Life Cycle diagram connects Egg, Larva (caterpillar), Pupa (chrysalis), and Adult (butterfly) with circular arrows, alongside a dark chat sidebar.](https://ai.engineer/api/talk-essay-media/fe426b4330c9dd01045c154fc75488e42be12ccb7f0d8af724e22f614c93f854)

The completed butterfly life-cycle diagram appears beside the agent’s response.

The diagram works, but the interaction still feels to Ruiz like handing his keyboard to an AI. That does not make the pattern useless. He cites Lovart and MagicPath as design applications using tldraw, and tutoring as another useful setting: an agent can help with homework or fill in an existing worksheet. To demonstrate contextual completion, he begins a D&D character sheet with his name and fields such as class and age, then asks the agent to complete it. As he turns toward the next experiment, the sheet assigns him the fighter class.

### Bring the agents onto the canvas

[9:51](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=591s)

[Fairies](https://fairies.tldraw.com/) gives the agents a visible presence inside the workspace. These small animated characters can be picked up and thrown around; hold one too long and it reacts. They still accept familiar requests such as drawing a cat, but their positions and animations expose agent state in the same space where their work happens. Instead of monitoring several terminal windows or a composer interface, the user can see where each agent is acting relative to the others.

The agents can also observe one another’s work. Ruiz gives separate Fairies requests to add a hat and a neck to the same cat. Both can work with the shared drawing concurrently. This makes collaboration more concrete: the user sees agents modifying parts of a common artifact, rather than receiving a finished replacement from a sidebar.

A group request adds orchestration. Ruiz selects three Fairies and asks for more animals. The workflow proceeds through distinct responsibilities:

1. Select a leader for the group.
2. Have the leader scout the existing canvas.
3. Create a to-do list and delegate its tasks to the other agents.
4. Observe the resulting work and judge whether it is correct and complete.

Shared state is the difficult part. Ruiz describes agents as effectively blind while they are working, creating a risk that their efforts overlap. The leader’s role is supervision rather than drawing, but the demonstration does not specify a locking or conflict-resolution protocol.

Fairies serves as an accessible introduction to multiple agents, but the next task moves beyond animal drawings. Ruiz places a long description of an e-book app on the canvas, summons the Fairies and asks them to make its wireframes. They continue working while he introduces a more permissive way for agents to control the application.

### Runtime access widens the capability—and the risk

[13:35](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=815s)

Giving agents more access raises a different problem: what is safe to expose in a hackable application? tldraw already has a runtime API, and coding models can write against it. A generic code sandbox alone does not meet this experiment’s needs, however. The agent also needs the DOM, browser rendering and screenshots to see the effects of its changes.

Ruiz’s solution is an Electron wrapper around tldraw. He asks **Claude** to create a small HTTP server with an endpoint that treats posted content as JavaScript and executes it. That bridges the coding agent to the running editor, but it also grants arbitrary code execution. Ruiz explicitly warns against using this approach in an app, framing the offline, file-based desktop experiment as a risk the individual user accepts. Local ownership of that risk is not technical isolation or a guarantee of safety. While he explains the setup, the Fairies’ background task produces e-book reader wireframes.

One possible use is a round trip between code and a diagram: visualize the code, edit the diagram, then update the code to match. Ruiz describes that workflow but skips its demonstration. Instead, he draws a small interface with controls for leg length and T-shirt color and asks the agent to make it interactive. His point is that the prototype does not need to stop at the interaction primitives already exposed by the drawing tools: although he says tldraw lacks the relevant built-in hover and click primitives here, the agent can write code against the editor to supply behavior.

### Permission to modify software does not ensure the intended result

[15:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=958s)

While the interactivity request runs, Ruiz describes how readily agents propose script injection and modifications to local software. He recounts a teammate, Max, asking Claude to remove podcasts from Spotify; in his account, the agent was willing to work through the application’s minified code or bundle. This is an anecdote about the agent’s willingness to modify software, rather than a demonstrated Spotify implementation.

The live canvas task returns something unexpected: a separate HTML site. It shows a character with a colored shirt, color choices and a long pointer that Ruiz objects is not even a slider. He wanted behavior inside tldraw, so he redirects the agent back to the canvas. The character also blinks, but that extra behavior does not establish that the requested controls work.

![A white browser page displays a smiling character in a blue shirt, a long black pointer extending right, and colored circles below with blue selected.](https://ai.engineer/api/talk-essay-media/d39d2d49f448c0c4bd542955007f694f0d8f1f7fc7b3faa9063923ea1a1212c3)

The browser result shows a blue-shirted character with a long pointer and color choices.

Ruiz sees broad potential in agents modifying desktop applications and speculates about similar access to websites. Yet the ongoing attempt illustrates a narrower result: an agent can enthusiastically change software while misunderstanding where the change should live and what interaction it should implement. More access expands the space of possible actions; it does not by itself align those actions with the request.

### Local files give users room to accept powerful tools

[17:46](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1066s)

The requested interactivity remains unsuccessful. Ruiz first suggests releasing the experiment, then corrects himself that it is already released. He returns to his preference for local-first applications and files that matter independently of the app, before abandoning the live attempt with an invitation to catch him later for a working demonstration.

The unfinished example still gives those preferences a practical motivation. A local, file-based application can expose itself to Claude and other agents so they can script and modify it. In Ruiz’s framing, maximizing that agency requires letting users accept the corresponding risks themselves. He points to [OpenClaw](https://docs.openclaw.ai/gateway/security) as an example of this attitude toward powerful, potentially dangerous tools.

Work continues on the canvas experiments. Ruiz invites people to explore Fairies and find behaviors that surprise them, as they have surprised him. He adds that the Fairies have IRC too, without demonstrating its protocol or architecture—another avenue for experimenting with how agents communicate while sharing a workspace.

## Resources

- [Make real, the story so far](https://tldraw.dev/blog/make-real-the-story-so-far) — Steve Ruiz explains the original sketch-to-website experiment and how annotations became revision prompts.
- [Make Real](https://makereal.tldraw.com/) — Turn selected canvas sketches into website prototypes using a model provider key.
- [tldraw Computer](https://computer.tldraw.com/) — Build visual workflows from connected components that generate and transform data.
- [tldraw Fairies](https://fairies.tldraw.com/) — The public canvas-agent experiment demonstrated in the talk.
- [tldraw SDK repository](https://github.com/tldraw/tldraw) — React infinite-canvas SDK with custom shapes, runtime editor control and starter projects.
- [Agent starter kit](https://tldraw.dev/starter-kits/agent) — Build an agent that observes canvas content and modifies editable shapes through an iterative loop.
- [Introducing tldraw offline](https://tldraw.dev/blog/tldraw-offline) — The July 2026 desktop release adds local files and persistent agent-authored scripts.
- [Electron security](https://www.electronjs.org/docs/latest/tutorial/security) — Security guidance for desktop applications that combine browser rendering with native capabilities.
- [OpenClaw security model](https://docs.openclaw.ai/gateway/security) — Explains operator trust, tool permissions and isolation for a powerful personal agent.

## Slides

### AI Engineer Europe

The AI Engineer Europe logo opens the recording.

[0:00](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=0s) [AI Engineer Europe logo in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/24e440bf5dfd4ecccd4fffdc9ef9243be07fff10b16e87c6b80fbc594828e3ea.webp)

AI Engineer
EUROPE

### Presenting sponsor

Google DeepMind is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=5s) [Google DeepMind presenting sponsor card.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a43b90b83b6eca77e82f00387f025461fb2cd84d99ed7f55e0f7cc2bbdddc7d.webp)

PRESENTING SPONSOR
Google DeepMind

### Platinum sponsors

Braintrust, WorkOS and OpenAI are listed as platinum sponsors.

[0:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=10s) [Braintrust, WorkOS and OpenAI platinum sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/38a3f22d6d4f28f63d280c964d0dbe27bd2e869276fd95106e3cecf3ad0a70dc.webp)

PLATINUM SPONSORS

Braintrust
WorkOS
OpenAI

### Steve Ruiz and tldraw

Steve Ruiz introduces tldraw as both the online whiteboard he uses for his slides and an SDK for building other products.

[0:42](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=42s) [Steve Ruiz introduction with a circular name graphic and tldraw logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc56ceb51ed43d972515195647a7d0ca542dc512be2e911523941b18707aab6c.webp)

Steve Ruiz
@steveruizok

tldraw

Circular graphic: Steve Ruiz

Interface text:
AIE / Slides
Share
S M L XL
Shape
124%
MADE WITH TLDRAW

Event branding: AIE; AI Engineer EUROPE

### Replit agent canvas

Steve Ruiz points to Replit’s Agent canvas as a product built with tldraw’s canvas SDK. The project shows an agent’s debugging log beside an empty canvas.

[0:56](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=56s) [A Replit project shows an agent conversation beside an empty canvas.](https://assets.ai.engineer/sites/aiecode2025/sha256/17d1d70efc71da9d4a8b93a47ea4c14a2af220b9cee7ea969ac7802b34a025ea.webp)

There's a cn not defined error in Dashboard.tsx. Let me quickly check and fix it:

2 actions

The cn function is exported from utils.ts but Dashboard.tsx is not importing it. Let me check and fix the import:

2 actions

Now let me check for similar issues across all pages:

Ran grep

Good — no other files have the issue. Let me check logs quickly to ensure the fix resolved it:

Checked console and system logs

Both servers are running cleanly and data is being fetched successfully. Let me now present the app to you:

2 actions
Presented output
Pilates Training App
Website
Open

Checkpoint made 10 days ago
Worked for 5 minutes

Ready to share? Publish this App in one click.
Publish to domain
Available
pilates-coach--steve135.replit.app

Interface text:
replit.com/@steve135/Pilates-Coach
Pilates Coach
Upgrade
Preview
Database
Canvas
Creations
Add core API endpoints and frontend components for a ...
Invite
Publish
Make, test, iterate...
Plan

AI Engineer EUROPE

### Stitch coffee app designs

Stitch displays four generated coffee app screens on its canvas. The speaker says Stitch’s main canvas is not built with tldraw, but its annotation mode uses tldraw.

[1:07](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=67s) [Stitch displays coffee calculator, library, recipes, and settings screens.](https://assets.ai.engineer/sites/aiecode2025/sha256/3fb74a6f304f19e21b919e85f4b83d105439202547f2aa7f9313d857625c6f41.webp)

Coffee Calculator
[Beginning of heading obscured]
Strength
Light   Medium   Strong
Serving Size   250ml
Coffee Type
Drip
Your Recipe
Water   250ml
Ratio   1:16.7

Coffee Library
Search for coffee...
Origin
Roast Level
[illegible]
Colombian Supremo
Origin: Colombia | Roast: Medium
Notes: Caramel, Nutty
Best for Pour-Over
View Details

My Recipes
Search recipes
Morning V60
1:16 ratio - 20g coffee, 320g water
Afternoon AeroPress
1:15 ratio - 15g coffee, 225g water
Cold Brew Concentrate
1:4 ratio - 100g coffee, 400g water
Espresso Shot
1:2 ratio - 18g coffee, 36g yield

Settings
Alex Latte
[redacted]
Edit Profile
Unit Preferences
Weight & Volume   Metric
Temperature   Celsius
App Settings
Dark Mode
Push Notifications
More
About
Support

Interface text:
Coffee Calculator
The design has been generated for you. Would you like any changes or further details on this design?
Coffee List
Saved Recipes
Settings
Export
Share
What would you like to change or create?
Agent log
3.0 Flash
102%
AI Engineer EUROPE

### Stitch annotation option

Stitch’s main canvas is not built with tldraw, the speaker says, but its annotation mode is. The demo shows Annotate in a coffee design’s context menu.

[1:13](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=73s) [A context menu is open over the coffee app designs, showing an Annotate option.](https://assets.ai.engineer/sites/aiecode2025/sha256/a28c8d2afc21d783979fcad02bafe7a5a17da6084ba793368abc8f12a4c1d584.webp)

Yes, create all of them

The designs have been generated for your coffee calculator app:
• Coffee Calculator: A screen for users to input coffee parameters like strength, serving size, and coffee type to get a calculated result for their brew.
• Coffee List: A screen displaying a list of different coffee types with their characteristics and brewing suggestions.
• Saved Recipes: A screen where users can view, edit, and manage their saved coffee calculation recipes.
• Settings: A screen for app settings, unit preferences, and user profile management.

What would you like to do next?

Coffee Calculator
Strength
Light  Medium  Strong
Serving Size
Coffee Type
Drip
Your Recipe
15g  250ml  1:16.7
Calculate Brew

Coffee List
[illegible]

Saved Recipes
My Recipes
Morning V60
Afternoon AeroPress
Cold Brew Concentrate
Espresso Shot

Settings
Alex Latte
[redacted]
Edit Profile
Unit Preferences
Weight & Volume  Metric
Temperature  Celsius
App Settings
Dark Mode
Push Notifications
More
About
Support
Privacy Policy
Log Out

Interface text:
Coffee Calculator
Generate
Modify
Preview
More
Export
Share

Copy
Copy as
Cut
Duplicate
Focus
Favourite
Delete

GENERATE
Instant Prototype
Regenerate
Desktop Web Version
Predictive Heatmap
Variants

EDIT
Edit
Annotate
Design System

PREVIEW
New Tab
Show QR Code
Mobile 390×884
Tablet 768×1024
Desktop 1280×1024
View Code
Export
Download

Coffee List
What would you like to change or create?
3.0 Flash
Agent log
41%

Chrome
stitch.withgoogle.com/projects/[illegible]
AIE
AI Engineer EUROPE

### Annotating a Stitch design

Stitch’s annotation mode focuses the coffee library and opens a change prompt. Ruiz says this mode uses tldraw’s canvas, while Stitch’s main canvas does not.

[1:17](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=77s) [An annotation interface focuses on the coffee library screen with a change prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fe671d5dbb817f167eb50e4f15265f43386d0378eb478284b83e01292f41042.webp)

Coffee Library
Search for coffee...
Origin
Roast Level

Colombian Supremo
Origin: Colombia | Roast: Medium
Notes: [illegible]
Best for Pour Over
View Details

Ethiopian Yirgacheffe
Origin: Ethiopia | Roast: Light
Notes: [illegible]
Perfect for Drip Coffee
View Details

Sumatra Mandheling
Origin: Indonesia | Roast: Dark
Notes: [illegible]
Perfect for French Press
View Details

Interface text:
Describe your change...
Cancel
Apply
stitch.withgoogle.com
Memory usage: 155 MB
AIE
AI Engineer EUROPE

### Steve Ruiz and tldraw

Steve Ruiz introduces tldraw’s whiteboard SDK and its React-based canvas. He builds projects with the SDK to improve it, and many recent projects involve AI.

[1:28](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=88s) [A tldraw canvas introduces Steve Ruiz and displays the tldraw logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/da9d8706da42af7e3613a8513fd3983ea7caf3121a5843065fcf53edfc96ec15.webp)

Steve Ruiz

Steve Ruiz
@steveruizok

tldraw

AIE
Engineering the future of AI

Interface text:
AIE / Slides
Share
S M L XL
Aa Aa Aa Aa
Shape
124%
MADE WITH TLDRAW

### Make Real

Ruiz introduces Make Real with a sketch that links an ellipsis menu to an interface, then loops the Continue button back to the menu.

[1:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=118s) [Diagram showing a drawn interface and a Continue button in a Make Real workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f64c7422fdba77894127af5b454a7f73f1540fca1c6fc2af2f5001cfad41fcb.webp)

Make Real
Continue
makereal.tldraw.com

Interface text:
AIE / Slides
make real - tldraw
makereal.tldraw.com
Memory usage: 154 MB
Share
S M L XL
50%
MADE WITH TLDRAW

AIE
Engineering the future of AI

### Make Real photo-booth demo

Make Real sends a hand-drawn photo booth, including color and pixelation controls, to a model to make a working prototype. A blank area sits beside the drawing as the presenter waits.

[2:38](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=158s) [A hand-drawn portrait interface with color and pixelation controls on the Make Real canvas.](https://assets.ai.engineer/sites/aiecode2025/sha256/b597015e0bce48b40f65b896adf0b3aa78c86b928030741fdfa38d748718c197.webp)

Make Real
2023!
https://x.com/tldraw/status/1724892287304646868

Color
Pixelated

Interface text:
makereal.tldraw.com
Page 1
Make Real
S M L XL
Aa Aa Aa Aa
74%
MADE WITH TLDRAW
AIE
AI Engineer EUROPE

[3:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=190s) [A hand-drawn photo-booth interface beside an empty output area in Make Real.](https://assets.ai.engineer/sites/aiecode2025/sha256/56ff7cd5e2def5d4e37b2cac74a6b29c1b179b8ed5f2d793e69830e708a2cf24.webp)

Make Real
2023!
https://x.com/tldraw/status/1724892287304646868

Color
Pixelated

Interface text:
makereal.tldraw.com
Page 1
Make Real
56%
MADE WITH TLDRAW
AIE
AI Engineer EUROPE

### Make Real continue-screen iteration

The presenter marks up a generated HTML screen, crossing out its heading and pointing “green” at the continue button to build a new prompt from the existing interface.

[3:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=234s) [A hand-drawn continue button beside a generated screen headed “Ready to begin?”](https://assets.ai.engineer/sites/aiecode2025/sha256/2930187ab033690dbecd0ab026083988be5f050543fa54289269b0c0d3d13cd3.webp)

continue

Ready to begin?
You're all set. Let's get started on your journey.
continue
Double click to interact

AIE
Engineering the future of AI

Interface text:
makereal.tldraw.com
Page 1
Make Real
S M L XL
52%
Reading UI
MADE WITH TLDRAW

[3:59](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=239s) [A generated interface is marked with a crossed-out heading and an arrow labeled green pointing toward its continue button.](https://assets.ai.engineer/sites/aiecode2025/sha256/02eabcfda18a87f1137e84ffd1f0c3dd972916d5943d9c4cacb5bf733b08b619.webp)

continue

Ready to begin?
You're all set. Let's get started on your journey.
continue
green
Double click to interact

AIE
Engineering the future of AI

Interface text:
makereal.tldraw.com
Page 1
Make Real
S M L XL
Aa Aa Aa Aa
52%
Reading UI
MADE WITH TLDRAW

### Adding color instructions

The presenter adds a green button instruction and red and black swatches to the existing interface, building a prompt that includes the original website.

[4:16](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=256s) [A close view of the green annotation, red and black swatches, and the words use these colors.](https://assets.ai.engineer/sites/aiecode2025/sha256/0a7d42b8908cf7519c5677819000b5bfb7c5c2792ef277fb50a66888ecc881be.webp)

continue

green

use these colors

AIE
Engineering the future of AI

Interface text:
makereal.tldraw.com
Page 1
Make Real
S M L XL
Aa Aa Aa Aa
Double click to interact
125%
MADE WITH TLDRAW

### Annotated interface and generated result

The canvas pairs a green button instruction with red and black swatches. The generated version has a green continue button, but the speaker says it missed his instructions.

[4:27](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=267s) [The annotated original interface appears beside a generated version with a green continue button; red and black swatches sit below the original.](https://assets.ai.engineer/sites/aiecode2025/sha256/30d2a9cd1b7ad161fd9f097eb5435d4b5592a12765f85d5e25ccd07303316816.webp)

continue

Ready to begin?
You're all set. Let's get started on your journey.
continue

green
use these colors

Ready to begin?
You're all set. Let's get started on your journey.
continue

Interface text:
makereal.tldraw.com
Page 1
Make Real
S M L XL
Aa Aa Aa Aa
Shape
Double click to interact
Double click to interact
52%
MADE WITH TLDRAW
AIE
AI Engineer
EUROPE

### Computer and Teach

Computer depicts prompt chains converging; Teach introduces AI as a canvas collaborator, with a “draw a cat” prompt.

[4:51](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=291s) [Overview showing a Computer diagram and a Teach canvas with a cat-drawing prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/78d1711af2069c29532b1e749d5421bde99f5425c21e0507fb1c778a5b03b92f.webp)

Computer
computer.tldraw.com

Teach
draw a cat
teach.tldraw.com

Interface text:
AIE / Slides
teach - tldraw | Help AI learn to draw
teach.tldraw.com
Memory usage: 161 MB
Share
S M L XL
50%
MADE WITH TLDRAW
AIE
AI Engineer EUROPE

### Prompting Teach

Teach is prompted to draw a cat in a selected canvas. The speaker says it uses structured text outputs to create shapes rather than an image.

[5:03](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=303s) [An empty selected canvas above a prompt asking for a cat.](https://assets.ai.engineer/sites/aiecode2025/sha256/a7f5ed9ee84728b3a8ea2a4accb7e8d80badd7314a260942d41fb7c4f4629a56.webp)

draw a cat

Interface text:
teach.tldraw.com/editor
AI Summit
AI Training Enabled
Steve Ruiz
36%
MADE WITH TLDRAW
AIE
AI Engineer
EUROPE

### Generated cat

The “draw a cat” prompt produces a gray cat assembled from shapes. The speaker explains that the system uses text structured outputs to make shapes available in the editor.

[5:26](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=326s) [A gray cat made from simple shapes appears on the canvas.](https://assets.ai.engineer/sites/aiecode2025/sha256/93b41c01f4da97fd959926594366660130dc07345842567be72f11674e784b2a.webp)

AIE
Engineering the future of AI

draw a cat

Interface text:
teach.tldraw.com/editor
AI Summit
AI Training Enabled
Steve Ruiz
S  M  L  XL
36%
MADE WITH TLDRAW

### Cat with facial details

A cat assembled from shapes has whiskers, a pink nose and a downturned mouth. The speaker says the model uses structured text outputs to make canvas shapes.

[5:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=334s) [A closer view of the gray cat with whiskers, a pink nose and a mouth.](https://assets.ai.engineer/sites/aiecode2025/sha256/adf52ed3567a4f12f2e279e2fb748b8fabd50f0423f77e5b026d327390ef13cf.webp)

AIE

Engineering the future of AI

Interface text:
teach.tldraw.com/editor
AI Summit
AI Training Enabled
Steve Ruiz
S M L XL
85%
MADE WITH TLDRAW

## Full transcript

[0:00](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=0s) [upbeat music] Hello.

[0:15](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=15s) Hey, I'm gonna, I'm gonna kick off. Uh, sorry, we're starting a little bit late here.

[0:22](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=22s) I am Steve Ruiz from tldraw. Does anyone know tldraw? Yes. Hey. All right, fans, fans in the room. Uh, if you don't know tldraw, tldraw is kinda, kind of a couple of things.

[0:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=34s) tldraw is a online whiteboard. You can go to it. It's free. It's really nice. Um,

[0:41](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=41s) I'm gonna be using it for my slides. Uh, tldraw is also a startup. We're based here in London. And tldraw is also an SDK, something that you can use to build other products.

[0:52](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=52s) So if you've used, uh, Replit's new, um,

[0:57](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=57s) uh, Agent canvas, then that's built with our canvas. If you've used, uh, um, Luba AI's new canvas, that's built with our canvas. If you've used Stitch's new canvas, that's not built with our canvas. [laughs]

[1:12](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=72s) But if you go into this, like, kinda, uh, um, annotate mode,

[1:19](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=79s) this actually is our canvas. [laughs]  So we're, we're in there somewhere. It's a Angular app, you know. Uh, so anyway, um, tldraw is... Again, company makes a whiteboard, makes a whiteboard SDK.

[1:32](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=92s) Part of the idea with the SDK is that you can build cool things with, uh, with the SDK, which means it's part of my job to build cool things with the SDK to improve it.

[1:41](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=101s) Um, and a lot of those things recently have, have involved AI, right? Hackable canvas runtime s- built with React. In fact, the canvas is also just React components, so component, component, component.

[1:55](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=115s) Um, which means you can do some pretty cool stuff.

[1:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=118s) Uh, first one, uh, I'm gonna be at the mercy of the demo gods here and the, the internet and, and other things. So, uh, bear with me here. Um, does anyone remember this app Make Real?

[2:09](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=129s) Maybe, maybe this tweet, uh, about Make Real. Did anyone, did anyone remember seeing this tweet back in 2023 when the vision models came out? All right, cool. Uh, this was one of the first projects to kinda, like, break containment, uh, in, um, in AI.

[2:24](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=144s) I... Don't... We're gonna go to my phone. Fuck this. [laughs]

[2:30](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=150s) Um, which, which may, may itself be a disaster. Uh, so the, the basic idea with Make Real was that you could, um, use the canvas, draw this, the, and then send that to a, a model and, and have it make it into a working prototype, which sounds very, very quaint s- in 2026.

[2:47](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=167s) But in 2023 it was all the rage because there was no Lovable, there was no kind of vibe coding, hadn't been termed or coined as a, as a term.

[2:55](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=175s) Uh, and so this was one of the first projects where non-technical people could make technical stuff without having to code or to look at code.

[3:04](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=184s) And so again, we are at the mercy of the, the, uh, several fucking various internet, uh, [laughs]

[3:12](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=192s) internet gods. So, so we'll see. I'm gonna, I'm gonna let this cook for a minute. Um, but at the risk of leaking my API keys, I will try and switch to a faster model.

[3:22](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=202s) Oh, no, it's, it's coming. All right. Maybe I just gave it something hard to, uh, hard to work on. We'll see. Hang on a second.

[3:30](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=210s) Basic idea, something like that. Uh, yeah, there, there, here's my API keys. I always gotta rotate them every time I, uh, do this demo sadly. Uh, we'll see how much, how much gets spent before I get done with the talk. [laughs]

[3:46](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=226s) Um, but, you know, it's a very simple prompt. It's just like, go, go make this interactive. This is, uh, this is not what I asked for, but we'll see.

[3:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=234s) All right. Good job. Cool. And this is a, this is a working, working thing, which is great. It's like a real little bit of HTML. But you could also annotate on top of it, and you could say like, "Hey, uh, actually make this green, and why don't we use these colors?

[4:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=250s) I'm gonna make this red and, uh, and, and black. Um, use these colors." Right? And so you're kinda constructing a prompt here, even the prompt that includes the old, uh, website.

[4:25](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=265s) And, uh, there's not many apps that have actually, like, kinda used this. On- only now, only like the, the Google Stitch that I mentioned before, uh, this idea of...

[4:36](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=276s) Well, it didn't do anything that I asked it for. Terrible demo. We're gonna, we're gonna move on. Things have changed in the last couple years. Anyway,

[4:43](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=283s) uh, there was another one in tldraw Computer, which I'm actually not gonna go into, but this was, uh, a couple of, like, chains of prompts. But eventually we, we, we're like, "Hey, uh, AI on the canvas actually might be pretty cool.

[4:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=294s) Maybe we should, um, have the AI kinda work as a collaborator. Like, work with you on the canvas." So the first one of those was pretty, pretty straightforward idea where you, you could say like, you know, "Draw a cat."

[5:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=310s) You could do anything. You could say, um,

[5:13](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=313s) uh, "Draw me a diagram," you know, "Finish my slides," "Complete this graph that I'm working on," things like that. Uh, and unlike maybe image models, uh, like diffusion models and things like that, it's not building an image.

[5:25](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=325s) It is using text structured outputs to, to make the same things that I could make, right? So I have, I have tools, I have, like, circles and shapes and like this.

[5:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=334s) Uh, and it's funny to see how these things have changed over to... Oh, it's sad. [laughs]

[5:43](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=343s) Um, uh, but, uh, the, uh, you know, the fun of, of this is as a way of exploring the model and, and what the model knows and how it can comport all that stuff.

[5:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=354s) Uh, make the cat blow out the candle. Um, that's pretty cool.

[6:01](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=361s) You could also do something like, uh, draw a mouse. So multiple, um, multiple prompts at the same time on different parts. And so even though I didn't tell it what a candle was, and I certainly-- like, the application doesn't know what a candle is.

[6:12](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=372s) Uh, and I'm not even sure that cats can, [blows] like, blow. But it's correctly interpreted that and kind of incorporated it into the, into the design. So, uh,

[6:25](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=385s) with incredible detail as well. [laughing] Still sad. Uh, so the, [chuckles] um, this was really fun because this was, like, solving a lot of problems that might not be obvious. Like, like, vision models when it comes to, to structured data, um, number one, there's much less vision training data than there is for text.

[6:46](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=406s) Uh, number two, a lot of that training data, like, conflicts in ways that text does not and other, other types of things don't. So, for example, the, the Y-axis on a, on a Cartesian graph, as you go up, that number goes up, right?

[6:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=418s) So zero, one, two, three, four, five. Uh, on the web, the Y-axis goes up in this direction, right? The top left corner is zero, your top left corner here.

[7:07](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=427s) But as you go down, the Y goes up. There's left, right. Like, there's your left, there's stage left. There's all sorts of, uh, uh, things that, that conflict within language, uh, and within images.

[7:21](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=441s) So, um, training the model to kind of behave predictably and, and produce things like this is, uh, wa- was really-- Well, I use training, uh, prompt engineering the model to, [laughs] to do it was, was really tricky.

[7:34](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=454s) But this was fun, um, but we felt like it, it didn't go really far enough because it was just one-shotting, right? We wanted to do an agent. So this is what Cursor looked like back in, uh,

[7:45](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=465s) twenty twenty-five or something like that when I did this. Um, draw a diagram, uh, of h- the life

[7:53](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=473s) cycle of a butterfly. So this put it into a kinda, like, agentic loop like you might have seen, seen elsewhere. Um, and I'm sure you, you interact with dozens of times a day by, by now for this crowd, um, where you have it produce an output and then review the output and kind of iterate until it thinks

[8:15](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=495s) it's done. Um, and we really tried to hew to the conventions at the time of, um, you know, coding agents that were-- where these agents, um, this agent loop was seen most, most often, where, yeah, there's kinda like a lot of subfeatures like rejection, you know, seeing it's thinking, think- seeing how it works.

[8:33](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=513s) Um, great. And now we have the, uh, the butterfly life cycle on the canvas. Pretty cool.

[8:41](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=521s) However, this was still not really enough because as, as cool as this was, it still felt like-- I don't know. It felt like I was handing my keyboard to some, some other AI rather than someone collaborating with me.

[8:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=534s) Um, although this model has been used really well in, uh, a lot of design apps that use tldraw, uh, things like Lovart or MagicPath, um, and in, in education, especially where you have this kinda tutor of like, "Help me with my homework and help me fill out my, um, you know," uh...

[9:12](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=552s) Oh, gosh, let's see if I can do this on the fly. Steve Ruiz, uh, class, you know, age, whatever. Um, and you can kinda ask it to, like, uh,

[9:27](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=567s) complete my D&D character sheet, right? And it'll, it'll kinda pick up what you're doing and fill out forms and do, do, do fun stuff like this. Maybe I'll come back to that as it, as it kinda chooches along.

[9:41](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=581s) The-- What I really wanted is, like, to bring the agent out of the sidebar and into the, the canvas itself. Um, oh, I'm a fighter.

[9:51](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=591s) Nice. Nice. All right, I'll take that. Uh, and so we did, and we did it with, uh, Fairies, which maybe, maybe you saw, maybe not. Um, these are like little, little guys on the, on the canvas.

[10:04](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=604s) You can kinda throw them around. Uh, they don't like to be held for very long. They'll start freaking out. Uh, yeah, okay. [laughing] So, uh, [laughs]

[10:13](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=613s) and, uh, but you can do the same thing like, you know, draw a, a, draw a cat or something like that. Now, putting the agents on the canvas have a whole bunch of interesting things.

[10:21](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=621s) You can see the state of the agent, right? These are multiple agents that I'm kinda running. I- in, in coding terms, these would be multiple terminal windows or something like that, or this would be in compos- Composer.

[10:32](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=632s) But you can kinda see what they're doing in a way that, uh, uh-- Hang on. I'm zoomed way out.

[10:38](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=638s) I did all the sprites myself. [laughs] Um, and, you know, not only can you kinda see its thinking, but you can see its action. You can see where it-- where in the project it's sort of like acting relative to the other, um, other agents.

[10:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=654s) So, and, you know, th- these other agents can, can see each other, what they're doing. So if I ask this one to draw, um, a hat on the cat, uh, and I draw this one, uh, draw the cat's neck. [chuckles]

[11:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=670s) We missed the, we missed the neck. Um, [laughs]

[11:14](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=674s) uh, they'll, they'll, they'll get to work, right? And they're, they're able to kinda work with each other's stuff at the same time. Um, but we could also ask them to work together.

[11:23](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=683s) So if I grab all three of these, uh, these fairies, Fairies Helen and Joan, um, draw some more animals. Uh, one of them will be elected leader. So this one is, is the leader, and it's gonna go scout kinda what's going on on the canvas, and then it's gonna create a to-do list, and it's gonna delegate that

[11:42](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=702s) to-do list to the other agents, right? Um, this is all, like--

[11:48](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=708s) We, we were doing this in like December, October of, of last year, uh, and

[11:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=714s) we're figuring this stuff out at the same time that, that a lot of people were figuring out agent orchestration. This idea of like, okay, how do we give them shared state?

[12:01](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=721s) How do we, uh, you know, have a leader, follower? Like, how do we manage the fact that these things are essentially blind while they're working and prevent them from kind of overlapping, uh, in terms of, like, what they're doing?

[12:11](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=731s) And so you can kind of see the, the, the leader here isn't doing any of the work. Uh, but it is gonna kind of, like, observe... Oh, no, that's, that's the leader [laughs].

[12:20](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=740s) That's the leader. Um, it's observing, uh, and [laughs], um, uh, and, and judging and, and establishing whether this is, like, done or not and, and whether it's done correctly.

[12:33](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=753s) Um, it's still not enough, right? Fairies are fun. If you wanna play with this, by the way, this is at fairies.tldraw.com. Um, in the same way that Make Real was a really good introduction to, um, to just AI at, at all, right?

[12:53](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=773s) S- draw something, click a button. Um, Fairies is a great way to talk about, like, multiple agents kind of working together. Uh, and they can do real work. Let me try and grab a...

[13:05](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=785s) Like, this is a big description of, like, a, an e-book or something like that. And if I summon my fairies, uh,

[13:13](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=793s) uh, make this, make the, make, make the wireframes for this app. Cool. And I'll, I'll just kind of let them, let them get to work while we keep talking.

[13:24](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=804s) Um, the... I started 10 minutes late. I'm gonna, I'm gonna take another five minutes before I jump. Uh, the, the next step for this one

[13:35](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=815s) is to, to kind of give more access to the canvas to the agents. Um,

[13:43](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=823s) and there's really... We started to kind of run into the barriers of s- of safety. Like, what is actually safe to, to do with our hackable thing for, for users?

[13:54](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=834s) Um, because we have a runtime API. You can just code against it, right? And AI's are really good at coding, so maybe we could do some sandboxed, you know, stuff.

[14:01](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=841s) But no, because w- we need the DOM, we need the, kind of the browser as a way to see what's going on. We need to be able to generate screenshots, all this stuff.

[14:08](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=848s) Um, so we decided to use our, uh, our desktop app instead. So I, over the holidays, I threw together a, um, an app that does, Electron wrapper that wrapped tldraw.

[14:20](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=860s) Uh, and I opened a port, essentially. I said, you know, "Okay, Claude, like, make a little HTTP server and, and open a, put up an endpoint. And anything that gets posted to that endpoint, uh, treat it as JavaScript and run it."

[14:37](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=877s) Which [laughs]  Which is a terrible idea. Not a good i- like, don't, don't, don't do that on, on your app. However, for an, an offline desktop app that is file-based, like, what's the worst that could happen?

[14:50](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=890s) You could hurt yourself, I guess, but you're not gonna hurt the rest of us. Uh, and look at them, look at them going. Building my little e-book reader. That's fantastic.

[14:57](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=897s) Thank you, Fairies. Doing the, the [laughs]. Uh, so, so what does that, what does that give us, though? Um, I'm gonna, I'm gonna skip the demo where, uh, as you can imagine, I could, I could say, "Hey, visualize this code.

[15:11](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=911s) Make a diagram." Cool. All right, I'm gonna change the, the diagram. Update the code to match the diagram. Easy, right? You can have these kind of, uh, like, let's pull up the, the level of abstraction that we want.

[15:22](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=922s) Um, but the, the more surprising stuff was actually where I was like, you know, okay, like, check this out. Uh, I'm gonna draw a little user interface or whatever, right?

[15:31](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=931s) And I want this to be a leg length,

[15:35](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=935s) and I want this to be a T-shirt color. Uh,

[15:41](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=941s) and even though tldraw doesn't really have the ability to... We don't have, like, primitives for on hover or on click or it's not, like, it's not a fully thing.

[15:50](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=950s) This thing can write code against the editor. So, like, uh, make it interactive.

[15:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=958s) And, and we'll see, we'll see where it gets to. Um, so far, the results on this have been, like, really, really cool in ways that are super strange and disturbing.

[16:10](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=970s) Uh- [laughs] Because, like, asking, like, the AIs are like, "Sure, let's do some script, script injection," right? Like, that's, the way that it documents itself is, like, this is how you s- you should do this.

[16:21](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=981s) Um, it has no qualms at all, by the way, changing stuff that's on your desktop, on your computer. If you've ever wanted to, like, for example,

[16:30](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=990s) like, we, uh, uh, one of our team, Max, was like, "You know what? I don't like podcasts in my Spotify. I wanna get rid of podcasts in my Spotify app.

[16:37](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=997s) Claude, can you just do that?" And it's like, "Sure, let me go through the minified code or the bundle of the thing, and let me just rip and tear." [laughs]

[16:44](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1004s) Uh, and it's happy to do it. It makes them happy. Uh [laughs], they like it. Um, oh, what the fuck was that? [laughs]

[16:55](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1015s) Um, [laughs] I don't even know what you did. You made a, uh, an HTML s- what the? Oh, that's so weird. What? Like, it, it created a new HTML site out of this?

[17:06](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1026s) And this is the pointer? It's not even a slider. No, I want it, I want it in, in the tldraw. All right. It's just- Yeah. We, we, we love it.

[17:14](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1034s) It's blinking as well. I don't know if you caught that. [laughs]

[17:18](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1038s) Come on, do it. Yeah, there we go [laughs]

[17:21](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1041s) . [laughs] Let's see if it can... Come on, let's go.

[17:25](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1045s) Um, so yeah, like, it, there's really, like, uh, no limit to what, what, what it can do with a desktop app, and it's happy to do it in a way that, um, I can almost tell that it, it, it, uh, would love to do this to websites.

[17:39](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1059s) Like, it would lo- like, just let me just get my, get my claws in there. Um, all right. Come on, come on.

[17:46](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1066s) Still not working. We're gonna, we're gonna, we're, we're gonna, we're gonna go. Set up the interactivity. Come on. Um, this is gonna be really fun. I think we're gonna just release this.

[17:56](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1076s) Uh, we're, the, um, I mean, the, it is released, but the, uh, the notion that you can take...

[18:05](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1085s) Like, I love local-first apps. I love file over app. I love, like, there's all these ideas that up to now have kind of been curiosities and, uh, almost, like...

[18:15](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1095s) Hang on. Oh, come on. Oh, that's such a, such a disappointment.

[18:25](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1105s) We're gonna have to... Catch me later. [laughs] I'll make it work. Uh, but now actually the, like, the idea of a local file-based thing that is, is able to expose itself to, to, um, to, to Claude and agents, like, locally in order to, to essentially script inject kind of motivates a lot of that, that stuff which previously was

[18:43](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1123s) idealistic into, like, well, that's the only way that you could do this. If you really wanna maximize the agency in order to maximize what it can do, and take the risk, uh, and take on that risk, then y- you kind of just need to hand that to the user and say, "Good luck."

[18:58](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1138s) Um, I think OpenClaw does this pretty well of, like, this, these are sharp tools.

[19:04](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1144s) Have fun, you know? Anyway, uh, that is my agents on the canvas talk. Uh, work continues. If you wanna play with the Fairies, I highly recommend it 'cause it's super fun, and you will find things that surprise you [laughs], that have surprised me.

[19:19](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1159s) Uh, they have IRC as well. Let me see. Yeah, anyway. Uh, and if you wanna follow along with tldraw,

[19:28](https://www.youtube.com/watch?v=sPUjIBH5Cwg&t=1168s) we are on Twitter, X [REDACTED:username], and then I'm [REDACTED:username] and post a lot about this stuff. So, uh, thank you for coming. Cheers. [audience applauding] [outro music]
