tldraw computer

Talk transcript

1. 0:00

   [on-hold music] My name's Steve, uh, Steve Ruiz.

2. 0:17

   I am from a company that I started called tldraw. tldraw started as a, um... Well, a couple things. Started as, like, a, a, a digital ink library that then, uh, Christopher had me im-implement in Excalidraw.

3. 0:30

   When I was working on that, I was like, "You know, there should probably be, like, a kind of a, a really good SDK for building these types of things."

4. 0:37

   And I'd already worked on a couple of projects that, uh,

5. 0:41

   were kinda going in that direction, so I did. Turned out, if you build a canvas that other people can use, people will, will build cool stuff with it. So

6. 0:52

   today I'm gonna be talking about some of the stuff that we've done with, uh, AI using our, kind of our own toys, playing with our own canvas here. So I'm here in tldraw.com.

7. 1:03

   It's a free whiteboard. Um, you can come on, use it, make your diagrams, make your slides. Uh, very, very similar in, in use case to, uh, to Excalidraw actually.

8. 1:14

   Um, but there are a few things that are, are kinda special here, and I'll show you real quick. So again, this is tldraw.com, free, uh, end user whiteboard application.

9. 1:25

   Uh, and we also have tldraw.dev, which is the, um, SDK website. If you wanted to build stuff with tldraw, um, then you could go to tldraw.dev and learn all about the, the code and the documentation and how to do that.

10. 1:41

    The cool thing about the, the canvas is that it is, um... Well, I'll, I'll skip this one for a second. It is just normal web stuff. It's, like, React all the way down.

11. 1:49

    So for example, I can do, like, things like, you know, play YouTube videos and, uh, you know, still interact with them, still draw on top of them. But the, uh...

12. 1:57

    Yeah, every one of these little shapes, including, like, doing some pretty cool stuff like, um, you know, have a whole code editor here. This is just CodeSandbox that's embedded in tldraw.com.

13. 2:08

    This one is, uh, Figma, um, that, like, just is embedded in tldraw.com.

14. 2:13

    Uh, even y- if you really like Excalidraw, you can even use Excalidraw, uh, inside of, uh, tldraw.com. So, um, and I'm pretty sure, I hope, hope this doesn't break my slides, but if I paste the own- the tldraw inside of itself, um, then we can, we can kind of...

15. 2:31

    Let me see if I can draw inside of tldraw. Oop. Hang on a second.

16. 2:37

    Uh, wee. Yeah, right? Was that... We're, we're kind of modifying the, the inner from the outer, whatever. I'll let you think about how that works. Um, but yeah, and, and it has a lot of, like, kinda little details, I'll do this really quick, of, like, you know, nice arrows that just, you know, perfectly kinda follow the different

17. 2:54

    shapes of things and, you know, boxes where the, um, uh, you know, the, the, the, the corners of the boxes always stay in the corner of this, right? So that's part of our value propositions, that we, we take care of all these little, like, little details, make sure the corners are right, make sure the arrows are right,

18. 3:11

    stuff like that. Um, we did a couple of different

19. 3:17

    AI stuff on top of this, and some of these are gonna work, some of these are not, are, are not gonna work. Um, did we find out, is, is, uh, is Fal in the room here?

20. 3:26

    Uh, the... Okay. Well, in 2023 we had, um, a lot of success with, with Make Real. I'll skip this one for now. Uh, Make Real was the idea that, um, people were using tldraw for whiteboarding,

21. 3:43

    uh, as well as wi- like, drawing wireframes. And the idea would be like, well, what if we could take the diagrams that we were drawing, the wireframes that we were drawing, and we could just kinda, kinda make them real, right?

22. 3:54

    What would, what would be involved in that? Um, and so we, when the, uh, um, when the vision models came out, like GPT-4 with Vision... Uh, that's annoying.

23. 4:08

    I'll have to do it myself. We, uh, we realized you could just send a screenshot to, um, continue, boom, uh, to the model and say, "Hey, model, you're a web developer.

24. 4:19

    Your designers just gave you this lo-fi thing. Can you, can you create a, a higher... Like, can you actually prototype this? Can you build it?" And the models could do that really well.

25. 4:27

    Um, as usual, I'm gonna kinda, like, give this a second to, to load while we, uh... [chuckles] All right. They're really running with this input. All right. Well [laughs]

26. 4:38

    the models have since become very, very ambitious. Uh, here, here's another good one. Uh, let's say I wanna have a stop motion application where, like, I have a, a, a feed from my camera, and I wanna be able to take pictures, um, and I wanna be able to, like, see all those pictures there, but I also wanna

27. 4:53

    be able to play them, like, in series. Uh, using only the, the input here, right? I won't even do the title, just, just to be, uh, to be fun.

28. 5:04

    The model will, will spin off on that, and it will eventually... We can kinda watch it generate. But it will eventually come up with this. I just did this, um, during the last talk where, uh, that, that's my app.

29. 5:16

    You know, I can kinda do this. It's doing the onion skinning, uh, and there, there's my GIF, right? And not surprising. I mean, you can add images to cursor and stuff like that, and it, it just works really well.

30. 5:29

    Um, but the fun part is because, I'm gonna stop this, um, because this is back on the canvas, you can actually, um, annotate on top of the website and use that as the next prompt to kinda click this and kinda, kinda just generate the next one.

31. 5:45

    And I've done this already, but you can see that, yeah, sure enough, it made the button solid like I asked it to. And so using these drawing tools as a way of not only generating stuff, but annotating and, like, kinda iterating through these, uh, you can get some pretty, pretty wild results.

32. 6:03

    This came out at the end of 2023. It was, uh, one of the first kind of-

33. 6:08

    ... tools that let people that, um, couldn't program and couldn't create software to, to, to kind of do it, and it was, uh, it was pretty remarkable. So, like, this, this being the input, uh, leads to, um, you know, leads to an app.

34. 6:23

    But you might have seen there the, uh, that it, it just did a little flash of green. You know, there was a bug involved. So I just took a screenshot of the bug and, and sent it together with the, uh, the original source and said, "Hey, uh, can you, can you fix that particular bug?"

35. 6:38

    And, and yeah, it did. So it's, uh, it's pretty cool. The, um, if I don't crash my browser. Hey. All right, so that's Make Real. Uh, we also did this one called Draw Fast, which may or may not work.

36. 6:50

    I'm just gonna see if it does. This used a, uh, a thing called, like, latent consistency models. I think that's the name. Basically, like, uh, uh, create an image for me as fast as possible, and we will see if I can wake up the, uh, the server here.

37. 7:07

    Oh, hey, look. Hey, this normally doesn't work. Special. Uh, where you have a drawing, you have an image being created from the drawing, and I, as I change the drawing...

38. 7:16

    Oh, come on, do it. Uh, then the image is gonna, gonna change as well. Um, you can even take these things and flatten them like this, and now I can interact with the, um, the model, the images like this.

39. 7:30

    And, you know, let's say I'm gonna rotate it or, or maybe stretch it out really big. Uh, and

40. 7:37

    in, in, in good circumstances, this stuff works almost in real time. But you'll have to, you'll have to accept the, uh, well, uh, whatever. [laughs]

41. 7:46

    Uh, the, the one, one moment of, uh, of, of working as the best that we're gonna get. I'm gonna need to use two hands to do this. But no.

42. 7:56

    If I just make a whole bunch of people, will they...

43. 7:59

    Oh, 'cause they, they're running 'cause they're all sideways, right? I got it. Anyway, this is Draw Fast. Uh, but the one that I'm gonna talk about, uh, mainly is tldraw computer.

44. 8:12

    So this is how... Well, I'll just, I'll just do it. This is a kind of a graph full of these little, other little components. Uh, I am gonna say,

45. 8:22

    uh, AI engineer, um, MCP observability. I don't know, whatever. Uh, [laughs]

46. 8:33

    uh, conference. Uh, and I'm gonna draw a, a picture too of, like, maybe a, uh... I'll just do, like, a big, um, uh, top hat or something like that.

47. 8:45

    I don't know, whatever, with some playing cards in the, in the brim. Got it.

48. 8:51

    Write a short commercial is the instruction here. Uh, I'll even do please, uh,

49. 8:58

    and run it. Okay, so a couple of things are gonna happen all at once here. This graph is gonna execute. Right now, the instruction is creating a script for itself, and then it just executed the script.

50. 9:07

    Sorry, this goes fast. Wrote the text. Now it's generating speech. It's al- also generating an image based on this. Um, each one of these blocks accepts inputs and produces outputs.

51. 9:19

    So this image [laughs] is based on, uh, [laughs] our, our, our text, which was based on this, this instruction, which was based on these inputs. Uh, and then it's, it's, you know, creating speech right now.

52. 9:32

    Um, and that's gonna be whatever. The AI Engineer Conference is where innovation- You got it. And then I can, I can keep piping it on, and it'll, you know, but this time it'll make it sad and, and serious, and we create an image based on that, right?

53. 9:48

    So we-- It, this is cool. Um, the--

54. 9:53

    Each one of these things, like I said, has this script of, like, how should I use my inputs? What should I produce based on my inputs? So for this write a short commercial, it's something like, it's tiny.

55. 10:04

    I'll, I'll read it. Analyze inputs, looking for guidance on the product services style or other requirements for the commercial. Based on the inputs, write the text for a short commercial script.

56. 10:12

    Output the result, right? And it'll repeat those same instructions based on whatever I give it, and it'll, it'll pipe it out in the same sort of data that, uh, um, is acceptable as inputs by the next, next thing down the line.

57. 10:28

    Um, we did this in, uh, collaboration with Google. They, uh, came to me and said, "Hey, we have, uh, Gemini 2 coming out, and we wanna launch with a bunch of cool demos and a bunch of cool partners.

58. 10:38

    Um, do you wanna have, uh, you know, be a part of that?" I'm like, "Awesome. Does that mean we get early access to the, the new models?" You know, these, um, they had shown the, you know, using your phone and kinda like, you know, where did I leave my keys, and all that type, type of real-time stuff.

59. 10:51

    And they're like, "No." I'm like, "All right." I'm like, "Do I-- Anything?" "No, no, you gotta work with what you got." So, uh, cool. All right, we'll do that.

60. 11:00

    The, uh, and so we did. Um, you know, Gemini 1.5 was out. That's pretty cool. But also, Gemini Flash was out, and Flash was fast and pretty good and multimodal.

61. 11:12

    So that was the kind of the inspiration for this. As we worked on it more... That's, that's good. That's good. Sad and serious AI engineer conference. [laughs]

62. 11:22

    Um, yeah, it's good stuff. As we worked on this more, we, we realized that, like, you could, you could kind of do computer stuff with it. You could kind of like take a, uh, an instruction, say something like, like increment or like, I'll, I'll do this, like add up, uh, all your inputs,

63. 11:41

    and then you give it some inputs, uh, like, um, you know, whatever, uh, two,

64. 11:48

    and, uh, it's hard to do this, you know, 11, and it will come up with, uh, like you, like you kind of expect. It'll, it'll come up with, whatever, 13.

65. 11:57

    But the execution here is not being done in code. The execution is being done by a language model. Languages models are capable of this kind of like nonlinear thinking.

66. 12:07

    Um, so if I gave it two and, uh, octopus, um, as the inputs and asked it to add that up-

67. 12:15

    Um, well, the octopus is not a number, but

68. 12:19

    if you forced me to, which we do in the prompt, uh, infer a number from whatever, you know, uh, maybe, maybe, maybe it's eight and, and eight and two should make 10, and there you go, right?

69. 12:31

    And, you know, if it, if it was a, uh, you know, a camera feed and it is me-- I'm gonna try and do this. Uh, hold on a second.

70. 12:40

    Um, four. You know. Like, is it gonna be 14?

71. 12:49

    Maybe. Yeah, there we go, right? So, like, it's, it's able to use... Thank you. Yeah. [laughs] [clapping]

72. 12:59

    And it's not-- it shouldn't be that, like, surprising, you know? It's, it's just, uh, you know, multimodal model, take a bunch of inputs, uh, produce outputs.

73. 13:09

    Um, we, we kind of went further with this. I'm gonna have to jump to... And by the way, the, the, the killer use case for this, if it's not immediately obvious, is turning your daughter's, uh, drawings and stuff into, uh, pictures and stories and piping them all around, right?

74. 13:23

    Uh, but the, um... Where is this one? This is a good one.

75. 13:30

    You can also do-- I was playing a lot of Factorio at the moment as, [chuckles] as well. Um, oh, no, this is the wrong one. Hang on. All the way down at the bottom.

76. 13:40

    Grab this one. And so, uh, the idea of having these, these machines that even include cycles and loops, and that'll just operate forever. Um, so in this one, it comes up with a random pop song,

77. 13:54

    adds it to a list, feeds it back in so it doesn't repeat.

78. 13:58

    Asks, "Is this song about love?" Uh, and then sorts it according to, you know, well, again, we're working with language models, so we have a Boolean value of yes, no, or maybe.

79. 14:08

    Um- [laughs] So we have... [laughs] And, and it, it feeds back around and it, it, it kind of pipes and I can just leave this forever just spending my, my, uh, the credits that Google gave me to, uh, to burn.

80. 14:20

    Um, and yeah, it-- this is, this is really fun. tldraw computer. It, it got pretty popular. Not, not necessarily as popular as the, uh, Make Real, but it was, uh, it's, it's pretty amazing what you can do, and it really rewards creativity, to put it lightly.

81. 14:33

    Um, I have seen people using this to, to do actual multi-stage prompting, um, you know, decision-making analysis. And you can imagine this being asynchronous and somewhere up in the cloud, and maybe that's, that's what we do next, where we say,

82. 14:49

    "Take this CSV of, uh, email addresses of people who've engaged with our product. Email them all. Get a response. Do sentiments analysis. If they like it, you know, do something next, uh, you know, so forth.

83. 15:01

    Wait for me, you know, text me and say, like, 'Should I really email this person again?' Maybe I say yes." So having a big, long, long-lived asynchronous process, um, that could be run in parallel, this would be a great, great interface for, for designing that.

84. 15:16

    And everyone seems to get this. Um, when we, when we [chuckles] originally, uh, did this, the creative prompt, the, the kind of the, the philosophy of this project before we-- uh, I went home and prototyped, it was like, I want a computer that works the way that I thought a computer worked before I knew how a computer works,

85. 15:34

    right? Where you would just have, like, I want this stuff, and I wanna do this to it, and then I wanna take the results and go over here. So that's, uh, that's tldraw computer.

86. 15:43

    Um, I wasn't gonna show teach, but I will show teach, uh, which is, uh, create a flowchart that begins with AI and ends with engineer. Incorporate existing shapes. Um, when you have a really cool hackable canvas, like an SDK for canvas with, like, a runtime API, um, it, it plays really, really well with other AI tools.

87. 16:06

    Um, and you can really quickly-- even though these models aren't, like, great at this, um, you can really get it to work with the canvas in a way that is, um,

88. 16:19

    kinda like a virtual collaborator. Like, you can kinda get it to do stuff. I mean, the demo that I always show and that I'll do really quick is the whole, like, you know, draw a cat.

89. 16:30

    Um, somewhere on this page is a uni- uh, a pelican riding a unicycle. But I, uh, [chuckles]

90. 16:36

    um... There's a lot of stupid drawings here. Uh, but yeah, draw a cat, and it'll draw a cat. But, you know, it's, it's doing this stuff not as a, as an image.

91. 16:45

    It's not painting pixels in the way that, like, Midjourney would. It's, it's doing it as text. It's, like, kind of returning a structure that I can map into, to, to shapes on the canvas.

92. 16:55

    And so, you know, I can, I can work with them myself. I can correct it. Um, and it can, it can work with my stuff as well. So if I do, like, uh, this, it's, like, orange, uh, and I say, uh, "Make the cat,

93. 17:09

    the cat blow out the candle." I didn't tell it was, it was a candle, but, uh, let's see if it can do it. Um, I don't think cats can actually blow.

94. 17:21

    But I don't know that for sure. Uh, [laughs]

95. 17:25

    uh, this is using Claude as, as the back end. If you wanna know how this works, definitely catch me up afterwards. Um,

96. 17:31

    yeah, right on. [laughs] [clapping] Hey, and we get smoke as well. That's wonderful, right? Uh, the, uh, tldraw, a lot of this stuff, and, and, in fact, I would say our, uh, our advantage over the bigger companies in this space is that, uh, shitty but amazing is definitely on brand for tldraw.

97. 17:50

    Uh- [laughs] And yeah, if, if this seems like a good problem that you might wanna work on, definitely talk to me because we have some tools that make it easier.

98. 17:59

    Um, people build all sorts of crazy stuff with tldraw. Um, this is, uh, Grant Kat's liquid, you know, uh, simulation that's using tldraw as, like, the, the geometric physical in- you know, like, control layer, I don't know, authoring layer on top of it.

99. 18:15

    Um, companies build really cool stuff with tldraw, like Observables is built with tldraw now. It's, it's incredible. Um, I think we're only... It, it-- not even scratching the surface, uh, of what can be done with this paradigm and these tools.

100. 18:31

     Please build something amazing. Uh, I got the canvas. We have the technology. So that's my talk. Uh, thank you very much. [clapping] [outro music]