AI and Human Whiteboarding Partnership

[upbeat music] [audience applauding] Thank you so much for the intro.

I'm so excited to be here, uh, talking about, like, figuring out, like, how do we, like, AI and human, like, work in the world of whiteboarding. And, uh, I built Excalidraw, and if you v- don't know about it, like, you'll see, like, many, uh, thing about it.

And one of th- the expectation you probably have, uh, about a speaker at the AI engineer conference is that I talk about AI on every single sentence for the entire talk.

So I'm just going to give you a warning, I'm only going to do it for the second half of the talk. But the first half is going to be a good setup, and I...

You should, like, pay close attention because it's going to be, uh, really interesting content. So let's get started. So we are, uh, five years ago. Uh, I'm working at Facebook January 1st, and as you...

If you've been at any, like, big company, there's, like, the performance review process. And so we, uh, you have to, uh, give feedback around, like, all the people you've been working with, like, how good or bad you've been doing, and same for your manager.

And one thing that's, uh, like, I'm probably not unique in this room, but there's this little devil in me that is like, "Nope, you're not going to do it until the very, very last minute."

And this is called procrastination, and I'm really bad at this, and I've tried, like, many techniques over the years. And the one, uh, at that time that I, uh, worked on is c- what I call active procrastination.

So I know I'm not going to be able to write the performance reviews until, like, the last minute, so may as well do something actually useful, uh, in the meantime.

Otherwise, I would be, like, doom scrolling on my phone- [laughing]

... and, uh, not be productive. And so what I ended up doing is, uh, I wanted to write a blog post for a long time, and so I was like, "Okay, I'm going to write a blog post."

And so I used a tool called Zwibbler in order to, uh, draw, draw, like, this kind of like handswritten, like, illustrations. And unfortunately at the time, the website Zwibbler was down. [chuckles]

And I was like, "No, come on." I've wanted to procr- to not procrastinate and everything, and then it's still down. And so what did I do? I'm like, "Okay, we're going to go, like, one level deeper.

What would it take to actually re-implement Zwibbler? How hard would it be?" And so I opened the code sandbox, and I, like, drew, like, uh, wrote some code to use rough.js.

And now, hey, we- I got a demo. And the thing I didn't expect is I posted about it on Twitter, and, like, it exploded. Like, 200 likes on this first tweet, and, like, so many people, like, replied and commented and were excited about it.

And I was like, "Oh, this is so cool." And so this was the beginning of Excalidraw. Now, one of the thing is if I were to do this again, uh, in the future, uh, like right now, I would not, like, start, uh, writing code myself.

I would probably do, uh, use any of the coding agent that we've seen. And the good thing is I'm not the only one. So if you look at two weeks ago, uh, during the Anthropic keynote, uh, when they were- wanted to demo Claude's, uh, AI agents, they actually used, uh, Excalidraw for this.

And they actually had the agent re-implement, uh, the most popular request, which is a table layout. Unfortunately, they haven't sent a PR yet. So if you... anybody's working at Claude here- [laughing] ...

we welcome pull requests. Anyway, so one thing you probably have noticed is I've said AI multiple times, so I knew you were not going to last the entire half of the talk without it.

So now let's get back to the non-AI part of this. So, uh, this was, uh, like, uh, in the story, we are, like, in, uh, January 1st, 2020. And three months later, uh, you probably all know what happened:

COVID. And so everybody had to go home and now had to figure out, like, how are you working from home? Like, what's life like? And a lot of people, one of the big thing that they try to figure out is whiteboarding.

And so when you're in the office, you have a physical whiteboard, and you can draw stuff, and you can collaborate with people. But when you're, uh, at home, you don't have any of this.

And I was very confused because, uh, a lot of people, like, started coming to me around like, "Oh, actually Excalidraw is a really good, uh, whiteboarding, uh, solution." And I was, like, very confused because I built Excalidraw for drawing, uh, illustration for my blog post.

And why-- what does it have to do with whiteboarding? And then I started, uh, being interested around, like, okay, what are the other people doing whiteboarding, like, other tools doing?

And one thing I quickly realized is that all of the tools at the time basically took the physical limitation of whiteboarding and mapped them one-to-one, uh, with the online versions.

So for example, uh, a f- a whiteboard, like a physical whiteboard, has a fixed size. And so what everybody did, they created a fixed size A, create a new whiteboard.

And then if you run out of space, then, like, you're out of luck. Now with Excalidraw, you can just, like, scroll around and, like, write-- start writing somewhere else, and it just works.

Then the next one is, uh, drawing text. And so on a physical whiteboard, you have to trace every single letter. And on the mouse, it's actually very, very hard.

And so on Excalidraw, I was like, "Eh, like, we have a text box. Like, you can select, uh, and this is fine." And when I did the dry run for this talk, like, the people were telling me, like, there's no way.

Like, even MS Paint in, like, 30 years ago had, like, uh, keyboard support. But this was the state of, of the arts, uh, at the time. And one thing to, uh, to remember is nobody, like, thought about online whiteboarding, uh, at the time.

So, like, all the project were pretty small and, like-

N-not very used. And, uh, now all of a sudden there's been a huge influx. So this is why it was so bad. And then the, it c- like the list keeps going on.

So if you want to start erasing, uh, they would actually have you, like, erase every single pixel. But on Excalidraw, you can select and delete and just, just works.

And then, uh, like, I don't know if you've seen, like the, like the Reels, TikTok video, where, like, people, like, draw, like, perfect circles like this, uh, on a blackboard.

And this is, like, what you had to do, uh, for, like, doing this online. But with Excalidraw, you can just, like, select a shape, like a circle, and, like, you just draw it, and it just works fine.

And finally, there are things you cannot do, uh, on a physical whiteboard. So for example, like copy-pasting is, like, just doesn't work. But if you're working online, like this is the baseline expectation that you're able to, like, select and copy-paste.

And so this is, like, all of this, like, really was a big breakthrough in my mind around the idea of, uh, the physical versus virtual whiteboard. And this is, I feel, like, the same place and time we're in with AI integration right now.

So in practice, what, uh, a lot of the AI integrations today, they're basically taking, like, whatever we're doing right now and then, like, sprinkling AI or re-implementing with, with AI.

And the end result is, like, pretty bad because, like, A, you don't know how it works. But the thing is, unlike Excalidraw and the online, like, where we had the experience, right now with AI, like, nobody has experience.

And so this is why I'm super excited about this conference, where, like, all of the people, like, trying things out are actually sharing the lessons to the other with, uh, with each other.

And so hopefully, like, as we keep doing these conferences, now we're going to bring the state of the world, uh, with AI to be much more, uh, much better.

So now I talked a lot about, like, Excalidraw, the early days and everything. And so a lot of people have been asking us, like, "H-how do you integrate AI with Excalidraw?"

And we've tried a bunch of things. So now for the second part, I'm going to walk through what are the things we've tried, what are the things that have worked, what are the things that haven't worked.

So the first one is actually not us, but Görkem, who is, I think, is in the room somewhere, uh, working at fal.ai. And so he discovered this, uh, really cool machine learning model that takes, uh, an Excalidraw diagram, uh, with like a palm tree or like a sun, and then he's able to gener- uh, to generate real-time,

uh, live picture, like, bit of the image, uh, uh, in, uh, in reaction to this. And so this is, like, what's mind-blowing to me, where like, oh, now like machines can actually generate, uh, images and the model is working really well.

And, like, the, I think the demo is still working, so if you, uh, if you want, like, you should try it. But one of the issue is when we started, like, integrating with Excalidraw, we realized that people don't actually draw realistic images on Excalidraw.

And so this is one, like, the first, like, key lesson that we found, is that just because there's a model that exists doesn't mean that it should be within your product.

And I know right now, like, there's a huge pressure for all of us to, like, just add whatever kind of AI into our product that, uh, like, in order to be, like, AI first and whatever.

But I really, like, want you to think about, like, what are the actual AI integrations that actually make sense rather than, like, "Hey, I just put in AI and that's it."

And in practice, if you do that, uh, then actually, like, the whole product experience is going to be, like, way worse than if you didn't have any AI at all.

So this was, like, the first, uh, learning. Now, people do not draw realistic images on Excalidraw, but they do draw, uh, diagrams. And so now the question is, okay, so LLMs are magical and, like, they're going to generate diagrams, so let's try it.

And so we tried, and that didn't work out. And so this is an illustration, but in practice it was even worse. Like, there was not, there was no resemblance of a house or anything.

It was just, like, bad. And this is where, like, we started, uh, realizing that, like, just doing this wasn't going to cut it. So we need to, like, uh, be, uh, like, a bit better.

But one thing is yesterday I was chatting, uh, with, uh, somebody at, working at Gemini, and he showed me a demo of, like, Gemini actually outputting Excalidraw files. So my talk is already outdated, and, uh, we probably need to get back and, like, start coding on this.

But anyway, let's continue. So the thing though that, uh, we found out is, uh, LLMs were actually able to generate something called MermaidJS. And so the, the name of the product is super weird, but in practice, it's a way to describe a diagram.

And so in this case, you're just, um, writing flowchart LR A AI to B Engineer, and then, uh, it draws, uh, this specific diagram. And so LLMs were able to, like, realistically generate, uh, this kind of, uh, like, MermaidJS syntax.

And this was really exciting because now what we are able to do is to, like, encode the knowledge of, uh, like, uh, the diagram and Mermaid supports like many different types of diagram.

And so now we're able to, like, encode all of this and, like, all of the prompts can actually generate the diagrams. Now the challenge is, like, this output SVG or Canvas and not Excalidraw files.

So what do we do? So in practice, I can check at work. And so she implemented an Excalidraw backend for MermaidJS, and now we finally have our very, uh, first very, very successful integration of AI within Excalidraw.

So now what you can do is to, like, uh, set a prompt and, like, put in anything that you want. So in this case, a flowchart starts with AI and an engineer.

But, uh, we've seen people, like, uh, draw, like, org charts, diagram or TCP or the network request, uh, kind of diagram or architecture diagram, uh, where you can see like classes and inheritance, those kind of things.

And then you get, uh, a result, which is an actual Excalidraw, uh, component that you can play with. And one of the thing that's is the, like, the title of my talk is the human and AI, uh, partnership.

One of the thing that you've probably been very frustrated is, like, you start and you generate something that, like, looks ninety percent like what you want. But then you're like, "Eh, this is an image, I can do anything with it."

And this is, like, the magic of this integration with Excalidraw, is now this is just Excalidraw, and you can use Excalidraw to, like, make it however you want. And so, for example, you can get to something like this, where you start modifying, uh, the layout, just moving things around.

You start adding colors, and you really make it to where and to what you want to be. And so, I think this is the, the mindset that we're going to see more and more with successful AI integrations.

It's no longer going to be like one shot, "Hey, I'm asking the AI something, and the AI give me, like, something perfect." But how do you get something that's close enough, and then you as a human can start modifying?

And so right now, this is, like, a just one way, uh, like, one round trip operation. But I foresee a future where, like, now you're going to be able to go back to the AI and say, "Hey, modify this," and then, like, go to the merge folder.

We don't have this right now, but, uh, this is what I think the future is going to look like. So now this is, uh, one integration, so I'm going to go through a few other AI integration that we've been, uh, working on within Excalidraw.

So the next one is, uh, one of the things you can do with the SaaS offering of Excalidraw is managing file, creating a bunch of different files and sa- syncing in the cloud.

And as if you've been using this, most of your, like, documents are going to look like Untitled one, Untitled two, Untitled three, Untitled four. And one of the thing that, uh, LLM can actually really good at is generating names for these documents.

And so now, whenever you're creating a document, we wait until you stop working on this, so we actually have content, and then we're going to prompt you, "Hey, these are, like, all of the, uh, name that we think would match."

And then you just click, and then you're done. And so this way, like, it actually removes the burden on you to do it, and AIs are really good at this.

Next one is, while, uh, people are not using Excalidraw to generate, like, good-looking, uh, like, wall, uh, painting kind of things, they... People tend to, like, want to have, like, a lot of infographics, a lot of illustrations within their, like, within their slides or, like, uh, their diagrams.

And one thing that we are doing, which is in, uh, like, alpha, like, uh, closed right now, but we're going to open source and to, uh, ship in a few weeks, is the ability to generate, uh, illustrations.

And so we are really hyped about this because I think it's going to help, uh, a lot of people. And finally, the thing that, like, kills me the most, I'm using Excalidraw every day, and I usually want to integrate logos within my, uh, diagram or my slides, those kind of things, and I cannot find a reliable way

to remove the background from all of the Spacekey logos. So I just took a screenshot of the AI Engineer Warfare, uh, from the website, and I'm like, "Hey, please, like, how do I remove the background?"

And the thing that I wanted to do is to challenge all of you, like, you're all AI engineers, and probably some- someone f- from all of you can create a model that, uh, removes the background for logos.

And if you're able to make it work within the browser, send it to me, and we'll make it, uh, happen, and we'll, like, ship it within Excalidraw. So I'm really excited that, uh, some of you are going to be able to help us, uh, with this and help all of the people using Excalidraw.

So this is, uh, the end of my talk, and I have a few conclusions for you. So the first one is, right now, as I mentioned, we're in the, like, physical whiteboard to virtual, uh, whiteboard, like, kind of moments, uh, happening within, like, all of the product, but specifically, like, for the design products.

And now we need to figure out, like, what is the AI-native way, uh, to implement those interactions. And so really excited about this conference to, like, start showing, like, all of the potential.

The second one is please, please, please resi- resist the urge to actually ship all of the random model that exists within your app, and really only ship the one that, like, you feel like are actually adding to the product and helping, uh, your, uh, customers, uh, getting...

Like, doing what they actually want to do. And the last one is LLMs love structure. So if you're able to take your problem domain, problem space, and find a way to represent it in some kind of a domain-specific language, now it's going to be most like, uh, most likely, uh, going to work way better to, like, have

the LLM target this domain-specific language. And then you take this language, and without AI, you actually output it to your own system. So this is the end of my talk.

I'm really excited, uh, that we have this, uh, event and conference, and I'm also excited that, like, Steve, uh, is going to be talking next about tldraw. Like, we've been working together for a long time, and, like, he's probably going to blow your mind with all his, uh, examples.

So thank you. [audience applauding] [upbeat music]