Agents on the Canvas in tldraw

Talk transcript

1. 0:00

   [upbeat music] Hello.

2. 0:15

   Hey, I'm gonna, I'm gonna kick off. Uh, sorry, we're starting a little bit late here.

3. 0:22

   I am Steve Ruiz from tldraw. Does anyone know tldraw? Yes. Hey. All right, fans, fans in the room. Uh, if you don't know tldraw, tldraw is kinda, kind of a couple of things.

4. 0:34

   tldraw is a online whiteboard. You can go to it. It's free. It's really nice. Um,

5. 0:41

   I'm gonna be using it for my slides. Uh, tldraw is also a startup. We're based here in London. And tldraw is also an SDK, something that you can use to build other products.

6. 0:52

   So if you've used, uh, Replit's new, um,

7. 0:57

   uh, Agent canvas, then that's built with our canvas. If you've used, uh, um, Luba AI's new canvas, that's built with our canvas. If you've used Stitch's new canvas, that's not built with our canvas. [laughs]

8. 1:12

   But if you go into this, like, kinda, uh, um, annotate mode,

9. 1:19

   this actually is our canvas. [laughs] So we're, we're in there somewhere. It's a angular app, you know. Uh, so anyway, um, tldraw is... Again, company makes a whiteboard, makes a whiteboard SDK.

10. 1:32

    Part of the idea with the SDK is that you can build cool things with, uh, with the SDK, which means it's part of my job to build cool things with the SDK to improve it.

11. 1:41

    Um, and a lot of those things recently have, have involved AI, right? Hackable canvas runtime s- built with React. In fact, the canvas is also just React components, so component, component, component.

12. 1:55

    Um, which means you can do some pretty cool stuff.

13. 1:58

    Uh, first one, uh, I'm gonna be at the mercy of the demo gods here and the, the internet and, and other things. So, uh, bear with me here. Um, does anyone remember this app Make Real?

14. 2:09

    Maybe, maybe this tweet, uh, about Make Real. Did anyone, did anyone remember seeing this tweet back in 2023 when the vision models came out? All right, cool. Uh, this was one of the first projects to kinda, like, break containments, uh, in, um, in AI.

15. 2:24

    I... Don't... We're gonna go to my phone. Fuck this. [laughs]

16. 2:30

    Um, which, which may, may itself be a disaster. Uh, so the, the basic idea with Make Real was that you could, um, use the canvas, draw this, the, and then send that to a, a model and, and have it make it into a working prototype, which sounds very, very quaint s- in 2026.

17. 2:47

    But in 2023 it was all the rage because there was no Lovable, there was no kind of vibe coding, hadn't been termed or coined as a, as a term.

18. 2:55

    Uh, and so this was one of the first projects where non-technical people could make technical stuff without having to code or to look at code.

19. 3:04

    And so again, we are at the mercy of the, the, uh, several fucking various internet, uh, [laughs]

20. 3:12

    internet gods. So, so we'll see. I'm gonna, I'm gonna let this cook for a minute. Um, but at the risk of leaking my API keys, I will try and switch to a faster model.

21. 3:22

    Oh, no, it's, it's coming. All right. Maybe I just gave it something hard to, uh, hard to work on. We'll see. Hang on a second.

22. 3:30

    Basic idea, something like that. Uh, yeah, there, there, here's my API keys. I always gotta rotate them every time I, uh, do this demo sadly. Uh, we'll see how much, how much gets spent before I get done with the talk. [laughs]

23. 3:46

    Um, but, you know, it's a very simple prompt. It's just like, go, go make this interactive. This is, uh, this is not what I asked for, but we'll see.

24. 3:54

    All right. Good job. Cool. And this is a, this is a working, working thing, which is great. It's like a real little bit of HTML. But you could also annotate on top of it, and you could say like, "Hey, uh, actually make this green, and why don't we use these colors?

25. 4:10

    I'm gonna make this red and, uh, and, and black. Um, use these colors." Right? And so you're kinda constructing a prompt here, even the prompt that includes the old, uh, website.

26. 4:25

    And, uh, there's not many apps that have actually, like, kinda used this. On- only now, only like the, the Google Stitch that I mentioned before, uh, this idea of...

27. 4:36

    Well, it didn't do anything that I asked it for. Terrible demo. We're gonna, we're gonna move on. Things have changed in the last couple years. Anyway,

28. 4:43

    uh, there was another one in tldraw Computer, which I'm actually not gonna go into, but this was, uh, a couple of, like, chains of prompts. But eventually we, we, we're like, "Hey, uh, AI on the canvas actually might be pretty cool.

29. 4:54

    Maybe we should, um, have the AI kinda work as a collaborator. Like, work with you on the canvas." So the first one of those was pretty, pretty straightforward idea where you, you could say like, you know, "Draw a cat."

30. 5:10

    You could do anything. You could say, um,

31. 5:13

    uh, "Draw me a diagram," you know, "Finish my slides," "Complete this graph that I'm working on," things like that. Uh, and unlike maybe image models, uh, like diffusion models and things like that, it's not building an image.

32. 5:25

    It is using text structured outputs to, to make the same things that I could make, right? So I have, I have tools, I have, like, circles and shapes and like this.

33. 5:34

    Uh, and it's funny to see how these things have changed over to... Oh, it's sad. [laughs]

34. 5:43

    Um, uh, but, uh, the, uh, you know, the fun of, of this is as a way of exploring the model and, and what the model knows and how it can comport all that stuff.

35. 5:54

    Uh, make the cat blow out the candle. Um, that's pretty cool.

36. 6:01

    You could also do something like, uh, draw a mouse. So multiple, um, multiple prompts at the same time on different parts. And so even though I didn't tell it what a candle was, and I certainly-- like, the application doesn't know what a candle is.

37. 6:12

    Uh, and I'm not even sure that cats can, [blows] like, blow. But it's correctly interpreted that and kind of incorporated it into the, into the design. So, uh,

38. 6:25

    with incredible detail as well. [laughing] Still sad. Uh, so the, [chuckles] um, this was really fun because this was, like, solving a lot of problems that might not be obvious. Like, like, vision models when it comes to, to structured data, um, number one, there's much less vision training data than there is for text.

39. 6:46

    Uh, number two, a lot of that training data, like, conflicts in ways that text does not and other, other types of things don't. So, for example, the, the Y-axis on a, on a Cartesian graph, as you go up, that number goes up, right?

40. 6:58

    So zero, one, two, three, four, five. Uh, on the web, the Y-axis goes up in this direction, right? The top left corner is zero, your top left corner here.

41. 7:07

    But as you go down, the Y goes up. There's left, right. Like, there's your left, there's stage left. There's all sorts of, uh, uh, things that, that conflict within language, uh, and within images.

42. 7:21

    So, um, training the model to kind of behave predictably and, and produce things like this is, uh, wa- was really-- Well, I use training, uh, prompt engineering the model to, [laughs] to do it was, was really tricky.

43. 7:34

    But this was fun, um, but we felt like it, it didn't go really far enough because it was just one-shotting, right? We wanted to do an agent. So this is what Cursor looked like back in, uh,

44. 7:45

    twenty twenty-five or something like that when I did this. Um, draw a diagram, uh, of h- the life

45. 7:53

    cycle of a butterfly. So this put it into a kinda, like, agentic loop like you might have seen, seen elsewhere. Um, and I'm sure you, you interact with dozens of times a day by, by now for this crowd, um, where you have it produce an output and then review the output and kind of iterate until it thinks

46. 8:15

    it's done. Um, and we really tried to hew to the conventions at the time of, um, you know, coding agents that were-- where these agents, um, this agent loop was seen most, most often, where, yeah, there's kinda like a lot of subfeatures like rejection, you know, seeing it's thinking, think- seeing how it works.

47. 8:33

    Um, great. And now we have the, uh, the butterfly life cycle on the canvas. Pretty cool.

48. 8:41

    However, this was still not really enough because as, as cool as this was, it still felt like-- I don't know. It felt like I was handing my keyboard to some, some other AI rather than someone collaborating with me.

49. 8:54

    Um, although this model has been used really well in, uh, a lot of design apps that use tldraw, uh, things like Loveart or Magic Path, um, and in, in education, especially where you have this kinda tutor of like, "Help me with my homework and help me fill out my, um, you know," uh...

50. 9:12

    Oh, gosh, let's see if I can do this on the fly. Steve Ruiz, uh, class, you know, age, whatever. Um, and you can kinda ask it to, like, uh,

51. 9:27

    complete my D&D character sheet, right? And it'll, it'll kinda pick up what you're doing and fill out forms and do, do, do fun stuff like this. Maybe I'll come back to that as it, as it kinda chooches along.

52. 9:41

    The-- What I really wanted is, like, to bring the agent out of the sidebar and into the, the canvas itself. Um, oh, I'm a fighter.

53. 9:51

    Nice. Nice. All right, I'll take that. Uh, and so we did, and we did it with, uh, Fairies, which maybe, maybe you saw, maybe not. Um, these are like little, little guys on the, on the canvas.

54. 10:04

    You can kinda throw them around. Uh, they don't like to be held for very long. They'll start freaking out. Uh, yeah, okay. [laughing] So, uh, [laughs]

55. 10:13

    and, uh, but you can do the same thing like, you know, draw a, a, draw a cat or something like that. Now, putting the agents on the canvas have a whole bunch of interesting things.

56. 10:21

    You can see the state of the agent, right? These are multiple agents that I'm kinda running. I- in, in coding terms, these would be multiple terminal windows or something like that, or this would be in compos- composer.

57. 10:32

    But you can kinda see what they're doing in a way that, uh, uh-- Hang on. I'm zoomed way out.

58. 10:38

    I did all the sprites myself. [laughs] Um, and, you know, not only can you kinda see its thinking, but you can see its action. You can see where it-- where in the project it's sort of like acting relative to the other, um, other agents.

59. 10:54

    So, and, you know, th- these other agents can, can see each other, what they're doing. So if I ask this one to draw, um, a hat on the cat, uh, and I draw this one, uh, draw the cat's neck. [chuckles]

60. 11:10

    We missed the, we missed the neck. Um, [laughs]

61. 11:14

    uh, they'll, they'll, they'll get to work, right? And they're, they're able to kinda work with each other's stuff at the same time. Um, but we could also ask them to work together.

62. 11:23

    So if I grab all three of these, uh, these fairies, Fairies Helen and Joan, um, draw some more animals. Uh, one of them will be elected leader. So this one is, is the leader, and it's gonna go scout kinda what's going on on the canvas, and then it's gonna create a to-do list, and it's gonna delegate that

63. 11:42

    to-do list to the other agents, right? Um, this is all, like--

64. 11:48

    We, we were doing this in like December, October of, of last year, uh, and

65. 11:54

    we're figuring this stuff out at the same time that, that a lot of people were figuring out agent orchestration. This idea of like, okay, how do we give them shared state?

66. 12:01

    How do we, uh, you know, have a leader, follower? Like, how do we manage the fact that these things are essentially blind while they're working and prevent them from kind of overlapping, uh, in terms of, like, what they're doing?

67. 12:11

    And so you can kind of see the, the, the leader here isn't doing any of the work. Uh, but it is gonna kind of, like, observe... Oh, no, that's, that's the leader [laughs].

68. 12:20

    That's the leader. Um, it's observing, uh, and [laughs], um, uh, and, and judging and, and establishing whether this is, like, done or not and, and whether it's done correctly.

69. 12:33

    Um, it's still not enough, right? Fairies are fun. If you wanna play with this, by the way, this is at fairies.tldraw.com. Um, in the same way that Make Real was a really good introduction to, um, to just AI at, at all, right?

70. 12:53

    S- draw something, click a button. Um, Fairies is a great way to talk about, like, multiple agents kind of working together. Uh, and they can do real work. Let me try and grab a...

71. 13:05

    Like, this is a big description of, like, a, an e-book or something like that. And if I summon my fairies, uh,

72. 13:13

    uh, make this, make the, make, make the wireframes for this app. Cool. And I'll, I'll just kind of let them, let them get to work while we keep talking.

73. 13:24

    Um, the... I started 10 minutes late. I'm gonna, I'm gonna take another five minutes before I jump. Uh, the, the next step for this one

74. 13:35

    is to, to kind of give more access to the canvas to the agents. Um,

75. 13:43

    and there's really... We started to kind of run into the barriers of s- of safety. Like, what is actually safe to, to do with our hackable thing for, for users?

76. 13:54

    Um, because we have a runtime API. You can just code against it, right? And AI's are really good at coding, so maybe we could do some sandboxed, you know, stuff.

77. 14:01

    But no, because w- we need the DOM, we need the, kind of the browser as a way to see what's going on. We need to be able to generate screenshots, all this stuff.

78. 14:08

    Um, so we decided to use our, uh, our desktop app instead. So I, over the holidays, I threw together a, um, an app that does, Electron wrapper that wrapped tldraw.

79. 14:20

    Uh, and I opened a port, essentially. I said, you know, "Okay, Claude, like, make a little HTTP server and, and open a, put up an endpoint. And anything that gets posted to that endpoint, uh, treat it as JavaScript and run it."

80. 14:37

    Which [laughs] Which is a terrible idea. Not a good i- like, don't, don't, don't do that on, on your app. However, for an, an offline desktop app that is file-based, like, what's the worst that could happen?

81. 14:50

    You could hurt yourself, I guess, but you're not gonna hurt the rest of me. Uh, and look at them, look at them going. Building my little e-book reader. That's fantastic.

82. 14:57

    Thank you, Fairies. Doing the, the [laughs]. Uh, so, so what does that, what does that give us, though? Um, I'm gonna, I'm gonna skip the demo where, uh, as you can imagine, I could, I could say, "Hey, visualize this code.

83. 15:11

    Make a diagram." Cool. All right, I'm gonna change the, the diagram. Update the code to match the diagram. Easy, right? You can have these kind of, uh, like, let's pull up the, the level of abstraction that we want.

84. 15:22

    Um, but the, the more surprising stuff was actually where I was like, you know, okay, like, check this out. Uh, I'm gonna draw a little user interface or whatever, right?

85. 15:31

    And I want this to be a leg length,

86. 15:35

    and I want this to be a T-shirt color. Uh,

87. 15:41

    and even though tldraw doesn't really have the ability to... We don't have, like, primitives for on hover or on click or it's not, like, it's not a fully thing.

88. 15:50

    This thing can write code against the editor. So, like, uh, make it interactive.

89. 15:58

    And, and we'll see, we'll see where it gets to. Um, so far, the results on this have been, like, really, really cool in ways that are super strange and disturbing.

90. 16:10

    Uh- [laughs] Because, like, asking, like, the AIs are like, "Sure, let's do some script, script injection," right? Like, that's, the way that it documents itself is, like, this is how you s- you should do this.

91. 16:21

    Um, it has no qualms at all, by the way, changing stuff that's on your desktop, on your computer. If you've ever wanted to, like, for example,

92. 16:30

    like, we, uh, uh, one of our team, Max, was like, "You know what? I don't like podcasts in my Spotify. I wanna get rid of podcasts in my Spotify app.

93. 16:37

    Claude, can you just do that?" And it's like, "Sure, let me go through the minified code or the bundle of the thing, and let me just rip and tear." [laughs]

94. 16:44

    Uh, and it's happy to do it. It makes them happy. Uh [laughs], they like it. Um, oh, what the fuck was that? [laughs]

95. 16:55

    Um, [laughs] I don't even know what you did. You made a, uh, an HTML s- what the? Oh, that's so weird. What? Like, it, it created a new HTML site out of this?

96. 17:06

    And this is the pointer? It's not even a slider. No, I want it, I want it in, in the tldraw. All right. It's just- Yeah. We, we, we love it.

97. 17:14

    It's blinking as well. I don't know if you caught that. [laughs]

98. 17:18

    Come on, do it. Yeah, there we go [laughs]

99. 17:21

    . [laughs] Let's see if it can... Come on, let's go.

100. 17:25

     Um, so yeah, like, it, there's really, like, uh, no limit to what, what, what it can do with a desktop app, and it's happy to do it in a way that, um, I can almost tell that it, it, it, uh, would love to do this to websites.

101. 17:39

     Like, it would lo- like, just let me just get my, get my claws in there. Um, all right. Come on, come on.

102. 17:46

     Still not working. We're gonna, we're gonna, we're, we're gonna, we're gonna go. Set up the interactivity. Come on. Um, this is gonna be really fun. I think we're gonna just release this.

103. 17:56

     Uh, we're, the, um, I mean, the, it is released, but the, uh, the notion that you can take...

104. 18:05

     Like, I love local-first apps. I love file over app. I love, like, there's all these ideas that up to now have kind of been curiosities and, uh, almost, like...

105. 18:15

     Hang on. Oh, come on. Oh, that's such a, such a disappointment.

106. 18:25

     We're gonna have to... Catch me later. [laughs] I'll make it work. Uh, but now actually the, like, the idea of a local file-based thing that is, is able to expose itself to, to, um, to, to Claude and agents, like, locally in order to, to essentially script inject kind of motivates a lot of that, that stuff which previously was

107. 18:43

     idealistic into, like, well, that's the only way that you could do this. If you really wanna maximize the agency in order to maximize what it can do, and take the risk, uh, and take on that risk, then y- you kind of just need to hand that to the user and say, "Good luck."

108. 18:58

     Um, I think OpenClaude does this pretty well of, like, this, these are sharp tools.

109. 19:04

     Have fun, you know? Anyway, uh, that is my agents on the canvas talk. Uh, work continues. If you wanna play with the Fairies, I highly recommend it 'cause it's super fun, and you will find things that surprise you [laughs], that have surprised me.

110. 19:19

     Uh, they have IRC as well. Let me see. Yeah, anyway. Uh, and if you wanna follow along with tldraw,

111. 19:28

     we are on Twitter, X [REDACTED:username], and then I'm [REDACTED:username] and post a lot about this stuff. So, uh, thank you for coming. Cheers. [audience applauding] [outro music]