Books reimagined: AI to create new experiences for things you know — Łukasz Gandecki, TheBrain.pro

Talk transcript

1. 0:00

   [upbeat music] So my name is Lukasz Gandecki, and I've been programming since I was a little kid, and I want to tell you about my newest project, um, Books Reimagined.

2. 0:24

   So how to use AI to create new experiences for things you already know. So how it all started, uh, I was reading a book about, uh, Donald Trump re-election, and since, as you can hear, I'm not from the United States, um, uh, uh, there was a bit...

3. 0:42

   a few too many characters to me. I didn't follow everyone, uh, so I decided to vibe code my way through the understanding. I, uh, built a little bit of an AI companion application.

4. 0:54

   Looked terrible, but, uh, it gave me context for the people that were on the page with a little bit... Uh, it found the images for them and gave me a little bit of a summary in the context of the, of the page that I was at.

5. 1:10

   And a month later, it turned into, um,

6. 1:19

   it turned into something different. So this is gonna be the Snow Q- this is one of the first experiences we've built. This is The Snow Queen book, and this is the part where the sorcerer's, uh, apprentices are flying away with the mirror that distorts the reality.

7. 1:35

   Um, so all right. [book narrating] So it tells a story about they're flying and flying, and the heaven is so far away.

8. 1:50

   Uh, there's music, and it reads, but you can't hear it. I'm sorry. But then the crash happens, and, uh, the mirror shatters, and it distorts everything all around. So this is one of the first experiences we've built and, uh...

9. 2:08

   But it's all in Polish, so I want to sh- actually demonstrate one that we built just for this conference that's in English. This is 1984.

10. 2:18

    And, um, what's interesting here, which I don't think I'll be able to show you, is that you can send a quick voice note to the book and ask, "What's going on in this scene right now?"

11. 2:34

    No, I don't really have audio. But the, the point is that there's many different, uh,

12. 2:42

    AI voice assistants, but they are all almost always just terrible, if not all of them, to be honest. Siri is terrible. We had a demo from, uh, Google yesterday.

13. 2:53

    They were saying up front that it works fifty percent. Uh, it's usually there's a delay. The... They start talking in the wrong s- uh, position, uh, I mean, at the wrong time.

14. 3:04

    Uh, then they interrupt you. So we've built here a system where it... you hold it as to just, uh, specify when you are speaking, and you... then you let it go, and it immediately, hundred milliseconds, responds to you, and then you could scroll further and then ask a question like, "What happened between last time I asked a

15. 3:25

    question and now?" And it can summarize what's going on. Um, so you have to believe me that. You can check later on bookgenius.net. Um, another thing that we were talk- thinking about is the search.

16. 3:37

    That's very, uh, common thing, searching. Uh, so the most normal search would be just exact search. But if you want to... The way our brains work, they don't memorize the pages.

17. 3:49

    So if you want to find a scene where Winston met O'Brien, then exact search is not gonna work, but embeddings work. So you can quickly find the scene you were thinking about this way, and then you can, uh, go to that, go to that, uh, spot, read a bit more, then you can go back to the place

18. 4:10

    where you were reading. But there is also, uh, one step forward... I mean, one more thing you can do, you can basically say, uh, "Talk about all the way the party, um, propaganda works," and you can do deep research, and it's gonna actually read the whole book till the, to the point that you finished at to give

19. 4:33

    you the answer. Is that very useful? It's gonna take a couple minutes. I'm gonna go back to presentation. So I started with vibe coding vanilla JS, very confusing code, but it gave me the freedom to iterate very quickly.

20. 4:49

    Um, you basically don't know what you don't know, and if you start to... Especially right now, the time it takes to plan everything up front is often wasted because you can much quicker just tell your thinking to the AI and generate something that works, and then you see, "Oh, that's actually not that great.

21. 5:05

    Let's try this and that." And I realized that throwing away code that you poured your heart into often feels terrible, like you're vested. You've spent so much time. But throwing away code written by a, by AI actually feels great.

22. 5:18

    Um, so I would ex- describe this as waves of changes. So basically, um, once I start feeling that the... I don't rewrite the whole code base day after day.

23. 5:31

    Like, the, the amplitude of the waves is getting lower and lower, and there comes a time where I can start old school engineering. Uh, I can start adding tests and refactor, but there are traps to refactoring.

24. 5:43

    Do I refactor the worst piece of code? I would suggest that it's better to focus on low-hanging fruits. So for example, I had a piece of a code for, uh, from OpenAI audio processing, and it's like JavaScript very quickly written, no types, very confusing, but I never have to touch it, so I'm, I'm not refactoring it, although

25. 6:02

    it was very tempting. So we often think about refactoring by adding this, how bad, how painful, how easy. But if something is very bad and very easy to change, but it's not painful at all, then it's probably not a good idea to change it.

26. 6:14

    So I would suggest that it's better to look at how bad the code is, multiplied by how painful and multiplied by how easy. And when all those factors are taken into consideration, then it starts making sense to, uh, make a decision.

27. 6:27

    So a lot of the AI, uh, experiences that we see and talk about are basically either ChatGPT wrappers or image generators, uh, or half-working useless voice assistants, including Siri.

28. 6:39

    Um, so our approach was to hide the AI from the user. Uh, so when we produce the books, the AI does the initial draft, and we do the rest, and I would argue that the human touch is invaluable in, in situations like this.

29. 6:54

    Uh, like not... AI cannot tell if the music that it generated is good. It cannot say if the graphics are good-looking or if the avatar is actually matching the vibe of the person, uh, that, uh, the book is talking about.

30. 7:09

    So we want to make the AI disappear, and, uh, multiple things connected together, simple things, simple building blocks, make for the magical experience for the reader. There's nothing really new here.

31. 7:19

    You could already ask a friend about, a question about the book, but is your friend available twenty-four/seven and all-knowing? Probably not. You can already search, but is the search the spoiler-free search?

32. 7:31

    Is it natural language search or exact match? Um, so I think that beautiful graphics help you get into the mood of the book and help you with the character recall.

33. 7:42

    Uh, and music that matches the scene makes it the experience like watching a movie, and we know that music influences the emotions hugely, and it's very nice when, um, you're reading the book, and the, the music just flows with the book and, and gives you this great experience.

34. 7:58

    So nothing new, but at the same time, completely new, which is what AI allows us to do nowadays. And I would encourage everyone to think about those tiny little niches where, uh, we can create some completely new experiences on top of something that we have known for so, such a long time.

35. 8:14

    So in thousands of years, it was never possible to read books like this, nor even really produce books like this because if I had to do all those graphics and music for every single book, it would cost me, I don't know, hundreds of thousands of dollars per book.

36. 8:26

    So it never meant... ma-made sense to do this. So how do we do this? The process is we use a combination of LLMs to the scene analysis, book characters detection.

37. 8:36

    We give the AI an overall music theme. So we say for, uh, Sherlock Holmes books, for example, that it's like Victorian London and all that, and it should be noir music and kind of on a sad note.

38. 8:48

    Um, so with scene, scene analysis, plus mood detection, we do music generation, and we also do structured XML with metadata. So for example, we have a text like this, and we re...

39. 9:00

    AI is very good at doing this kind of a mapping, which then is very easy for us to use in the, uh, book when we say, okay, like we can display the avatars that are in the scene.

40. 9:09

    It would be very time-consuming for a person to go through the whole book and map every single thing like this. So today, we are open sourcing the player, so anyone can create the Netflix-style experiences for books.

41. 9:21

    And if you want AI that feels like magic, not like chatbots, come talk to me. We build AI experiences that ship and delight and not slides, although I hope the slides were nice.

42. 9:32

    So thank you, and you can find me at those places. [clapping] [outro music]