LLM Knowledge Bases: a practical guide

Talk transcript

1. 0:00

   [upbeat music] All right. Are we ready to go? Okay, we're ready to go.

2. 0:17

   Are y'all ready to go? Yeah. We ready? Yeah. All right, let's do this thing. Uh, all right. Well, hello everyone. I'm Ben Holmes, uh, Developer Relations Lead at Warp.

3. 0:28

   Uh, you might have heard of Warp as a terminal that's really nice to use for you and your coding agents. You may have also heard of it as a cloud platform that helps you build out software factories.

4. 0:38

   If you haven't, we have a booth in the expo. You can go talk to us about it. Uh, but today I'm actually gonna be talking about something called LLM knowledge bases.

5. 0:47

   Uh, maybe a show of hands. How many people have an Apple Notes folder that's a complete disorganized mess that you don't know what to do with? Okay, a lot of people.

6. 0:56

   Uh, and how many of y'all have used Obsidian before, maybe? And how many of you use it now regularly? It's, it's a little less people. That's all right. Uh, I'm actually not gonna be showing y'all too much on there, but if you're an Obsidian head, this talk is definitely gonna be for you.

7. 1:13

   Uh, so we're gonna be talking about the ways that you can organize all of your sort of raw thoughts and notes that you're taking both between you and your LLMs into something that you can actually navigate and make sense of, and even remember and discover ideas you might have forgotten about.

8. 1:28

   Um, so our goal is to go from, pan over to this guy, uh, jumbled mess of scattered thoughts like this, all the way to something that's nice, organized and viewable like this.

9. 1:42

   But I'm getting ahead of myself. Uh, our goal is to initially start with just a raw folder of all of your thoughts. We're gonna be tracking our notes as Markdown for this example, and I'll be showing you the app that I actually built myself called Hubble, that'll let you navigate and write notes easily inside of there.

10. 2:00

    Uh, but you can use any Markdown viewer, Warp, Obsidian, whatever you like. Uh, then we'll talk about how you might turn that raw set of thoughts into something that's a bit more organized and interconnected, so enriching notes to connect related backlinks and tagging things based on categories so that you can find them or so agents can find

11. 2:18

    them. Then we'll go a little bit further into wikis, actually generating, uh, something that you can click around, around areas of research that you have, personal interest, et cetera, uh, that qualify the people, concepts and sources that all of your notes contain.

12. 2:34

    And then finally, we'll talk a bit about visualizations like that fancy graph view I was showing you. It's very easy to do these days now that agents can access your machine and use HTML.

13. 2:43

    Um, so the first thing I wanted to talk about is how you get raw thoughts down actually, uh, because it is pretty easy now to whip out Apple Notes and write something.

14. 2:53

    Uh, but I would argue it should be even easier, uh, to get everything down. So something that I'm gonna show you here. Uh, maybe another show of hands, how many people use Wispr Flow or something like that?

15. 3:04

    Okay, that's not enough hands. It's honestly not enough hands. Uh, because you may not be aware, but voice dictation, even though it is pretty awkward to talk into your computer with a bunch of coworkers around, it is the fastest way to get your thoughts onto paper.

16. 3:19

    It's going to be, like, two hundred words per minute, I believe is the average. Unless you're an absolute Olympic typist, it will be faster than any other method that you have.

17. 3:27

    So I suggest getting some sort of voice input tool, uh, just to get down context. There are a number of these, and by the way, for the people who aren't paying for Wispr Flow subscriptions, you're in luck.

18. 3:38

    You don't have to pay any more for this kind of stuff. There are local models that can do this, so a bit of a side tangent on tool chains that you should probably check out.

19. 3:46

    Uh, I love highlighting this one. It's called Handy. Uh, it's an open source tool for voice dictation that just uses a local model, stays on device, very easy to use.

20. 3:55

    Uh, there's another one called Voice Inc, which is very similar. Uh, this one I believe is, like, a twenty dollar lifetime fee just to get updates in the app.

21. 4:03

    Might as well do it just to support the developer. That's the one that I use. And it gives you a hotkey, uh, as well as a mobile app. So if you wanted to get something down, you could just hold either the Option key or the Function key on your computer and it'll start writing down everything that you're

22. 4:18

    saying, and it will turn it into a formatted, um, set of sentences, paragraphs, et cetera, on the other side. Cut off a little at the end there, but you can see the punctuations are cur-- coming in great.

23. 4:29

    And this whole, uh, set here with all these paragraph breaks, this was just one long transcription I wrote down after listening to a podcast. This was, uh, just me listening to the Acquired podcast about the founding of Walt Disney.

24. 4:42

    Uh, a little bit off-beat for me, uh, but I thought it was an interesting read just about, uh, how these sorts of businesses are founded. Uh, so I decided to take out a voice dictation app and write it down.

25. 4:53

    And that's what I've been doing for all of my notes at this point. Uh, either coming to my computer or on my phone, I just kind of type into it to get all the raw data.

26. 5:00

    And the reason I'm suggesting you do this is if you want to get to a point where you can actually have LLMs generate wikis, visualizations, et cetera, you need a lot of raw data.

27. 5:11

    You need a lot of raw materials. So don't worry if you're being a little bit scrappy, a little bit rambly. You're not formatting things with perfect bullet points. That's fine.

28. 5:20

    The goal should just be get down as many thoughts in the moment as possible. Maybe it's a meeting transcript, maybe it's research notes after you've read an important passage.

29. 5:29

    Whatever it is, just being able to get your idea down on the paper, uh, makes it very effective for LLMs to go access it later. [smacks lips] Um, so now let's talk a little bit about turning those raw materials into something more useful.

30. 5:41

    Uh, this is another note that I took about the founding of Ferrari. I went down a little rabbit hole when I started listening to this guy. Uh, and this is what you can get to if you just ask an LLM to do a little bit of recon for you.

31. 5:53

    So you'll notice that in the center is the same note, same sort of style of, like, a voice dictation transcript. But we've got some more details now. We can see the tags that are for the topic, so we can see this was a podcast.

32. 6:06

    It's related to these focus areas. You can also see the source. It was able to pull down the URL and find it. It also wrote down when it was enriched.

33. 6:14

    We'll talk about why I do that in a second. And at the bottom, it also finds some backlinks. Uh, those-- This is related to that other podcast you were listening to.

34. 6:21

    Let's go ahead and link them together. And that web gets tighter and tighter the more things that you get down. Uh, and the way that you can get-

35. 6:28

    From this to something like this is through an agent skill, as you might expect. So I have a skill for that right over here, and it's just called enrich note.

36. 6:39

    Enrich note. So right here, it's just showing how you might instruct an agent to take one of those note files and turn it into something that's more useful. So here it's just saying, "Let's go ahead and enrich that note."

37. 6:50

    Put a little timestamp on there, so if we ask the agent to do another pass, it remembers that some other agent did it in the past. Uh, have it generate some tags.

38. 6:59

    I actually put all of my tags into this little, uh, reference folder over here. That way, the agent isn't inventing new tags every time it goes through and tries to add more detail.

39. 7:09

    This gives it a concrete list of things to look through. I actually instruct the agent to be reluctant to add new tags because Claude loves to get creative. Uh, so just telling it, "Please don't do that."

40. 7:21

    Uh, use the instructions, but if you really find a pattern, go ahead and add something to this list. You can go ahead and do that. Uh, also telling it to go research the source with web tools.

41. 7:30

    At this point, all the agents have web search. If not, Exa is a great little tool that you can add into your harness to go find that for you.

42. 7:37

    The last one's those related notes. So find backlinks, use some file calls, find related notes using key term search, and put them into the bottom, so then we have that interconnected web.

43. 7:48

    Uh, and to run this skill, you can of course do it manually with whatever harness. Uh, I'm gonna be using Warp just to show you here because, totally unbiased, I'm a really big fan of Warp.

44. 7:58

    Can you believe it? Uh, but it also does give you access to any model, including open weight. So if you've ever been wanting to experiment with like GLM 5.2, which is very good at this sort of task, you certainly can.

45. 8:09

    In fact, I'll-- Let's go ahead and flip over to that just to try it. And I'll go ahead and call this skill by hopping into our little note here.

46. 8:19

    I might go ahead and call enrich note, and then I'll pull in this note from earlier, grab that file path, and just tell it to go ahead and add more detail to this guy.

47. 8:29

    And I can show you a completed run of this too. Uh, right over here, we can see, uh... This was-- [smacks lips] Well, we'll talk about this in a moment, but asking it to do it on a sort of automation across all of your notes.

48. 8:41

    Uh, it go ahead and f-- It will go ahead and find any notes that weren't enriched in the past. So remember that at the top here, we gave it a little timestamp every time it gets enriched.

49. 8:51

    So the next time we ask an agent to do it, it'll just look for anything that's not tagged yet. It'll go through, run that whole flow across all of those notes.

50. 8:58

    It'll make some modifications to the files. And then on the other side, you get a more organized notebook than you had before. It's very easy to do. Uh, we'll let this guy cook in the background while I talk a little bit about wikis.

51. 9:10

    Uh, so the next step after we've gone ahead and combined everything into related ideas... You might be fine with this, actually. I'll show you how useful this is just on its own.

52. 9:20

    So this is like a total, uh, mind dump of everything in here. Like, this was me reading The Left Hand of Darkness. It's a very good book, by the way.

53. 9:28

    And you can see I was just taking spare notes across different chapters whenever I found something interesting, and it was able to combine all those related notes, so I can just click through and find them.

54. 9:37

    Uh, so I'm not hunting around. Where did I save in my Apple Notes the last time I wrote about this, like two weeks ago? It's able to find it by related, um, topics.

55. 9:45

    So if you're someone that likes following Wikipedia rabbit holes, it kind of invents a Wikipedia rabbit hole of your own thoughts. It's kind of fun. Uh, but if we want to make this a little bit more browsable, we might tell it to generate a wiki on top of this.

56. 9:59

    So take that set of raw ingredients and cook it together into something a little bit more, um, hearty. I don't know. It's a weird metaphor. Uh, let's go into these wikis.

57. 10:09

    So these are a couple that I generated around topics that I'm interested in. Uh, one of them is probably predictable, you know, latest in AI news. Anytime I listen to a podcast or, uh, read a research document about the, like, system cards of the latest models, for example, uh, I wanna have a wiki that actually pulls all

58. 10:29

    of that in and groups them by related topics. So this wiki that you're seeing right here, uh, if we click into it, it'll give you a set of people, places, and things, basically.

59. 10:41

    So we get all of our sources. So these are some of the topics that we're most interested in. Uh, we can see a few in here, some of them expected, some of them not.

60. 10:50

    We see like a Ralph Loop rabbit hole, for example. We sim-- see a number of concepts that are pulled out of all of these ideas. Uh, and then down here we see some people, all the people you might expect, some maybe not.

61. 11:01

    Like, this is something that it pulled together from a intersection with AI and music. Adam Neely is actually a jazz musician that's been talking about how AI has affected the music industry.

62. 11:11

    So I wrote that down, found it was a little bit interesting, pulled it into our latest AI wiki so that we can find it easily. Also, organizations. You would expect all of these, I assume.

63. 11:21

    This one more music related. Uh, this one, the thing you should go try after this talk, et cetera. Uh, and all of that is generated programmatically. I didn't write any of this because all I have time to do is generate the raw ingredients, not connecting it all together myself.

64. 11:37

    Uh, so to have, to have an agent generate this, this is actually a gist from Andrej Karpathy. This is where the LLM knowledge base idea kinda came together. You can find this really easily now if you just search his name and then wiki.

65. 11:49

    It's kind of, uh, uh, gotten a lot of usage. So this is the simple principle of it. We want to take a raw directory, which is where we're taking all of our spare Apple notes from earlier, and we want to combine it into whatever focus area that we care about.

66. 12:06

    So we grab the raw sources, we create a wiki, and, uh, really you can tweak this to taste as much as you want. He even calls this an idea.

67. 12:14

    So what I did to generate that latest AI wiki was just describing, "These are the things that I care about. They're in this raw folder. Here's the rest of his gist.

68. 12:23

    Go ahead and generate a wiki for me." And on the other side, you end up with something very organized like this. Uh, I was able to do this for another topic too that's a little off-beat.

69. 12:32

    Uh, this one was, uh, because I was listening to the Bible in a Year podcast. This is something that I wanted to study recently, totally outside of my wheelhouse, with a number of characters that I forget the name of.

70. 12:43

    If anyone's read religious texts, you know that all the names sound very similar and can be hard to pronounce. Uh, so having something that can pull together all the people with references to what each of them did is very useful.

71. 12:55

    So I can click through here, and it generated sort of an entry for who that character is. You can imagine this is also useful in the workspace. If you take a lot of meeting notes and you want to have a people section of all the people that you've met with, interesting clients if you're in customer success, it

72. 13:09

    can generate all of those for you. And then it can create backlinks over to any related meetings you've had with them, maybe the source links. So going back to our raw notes, if you want to go to the raw materials, you could click through and find those.

73. 13:23

    Uh, really all we're trying to do is make sense of all these notes that we take every day and make it something a little bit more browsable, a little bit more readable.

74. 13:31

    Um, so the next thing I want to talk about beyond that is, uh, it's really nice that you can ask an agent to generate these on demand, but as we can see with our little agent running in the background, it can take time, uh, for an agent to go generate these things.

75. 13:45

    So ideally, we have a flow that will sort of do this in the background for us, maybe while we sleep, maybe on an automated schedule or something like that.

76. 13:55

    Uh, so that's kind of the next thing to talk about, which is taking these skills that enrich the notes, generate wikis, update them for you over time, and turn them into something that runs on a cadence, maybe daily, maybe weekly.

77. 14:09

    Um, and there's a little platform over here. There's a number of ones that do it, by the way. If you've played with like Codex app, they have an automations tool that spins up tasks to run on your machine every day.

78. 14:19

    But it means your laptop has to be cracked open when it runs because it's a local automation. If you want something that runs in the cloud, uh, I will shout out that we do have something called oz.dev.

79. 14:29

    If you go search for that in your web browser, it'll help you set up with automated schedules, uh, as well as, like, other triggers, like maybe a Slack message, iMessage, whatever you want to set up.

80. 14:40

    Uh, these I set up on schedules. So this was one that I set up with the, uh, wiki skill that I showed you a little bit earlier, and I encoded it into this set of instructions right here.

81. 14:52

    Uh, so the way this guy works, and I'll just go ahead and whiteboard this for you live and show you how it works, is, uh, we create an automation that takes our folder of Markdown, syncs it into a box, maybe like a cloud sandbox powered by Docker if you need it.

82. 15:06

    Uh, otherwise you don't. And then sync it back up when the agent's done. Uh, so the way that I've set it up is using, uh, the Obsidian headless CLI.

83. 15:15

    So Obsidian has a really nice tool that lets you take a bunch of Markdown on your computer and sync it somewhere else and then pull it back down. You could also just do like a Git clone too if you want to be a little bit less creative, put all of your notes in a GitHub folder, so that

84. 15:28

    way a cloud agent could pull it down and do it for you. I prefer using Obsidian CLI for this just because it avoids having to, like, push and pull your notes.

85. 15:38

    No one has time for that. I just want it to sync in the background. That's what this does. Uh, so it's a tool that you can install into a cloud sandbox.

86. 15:46

    It'll sync down that folder of Markdown. Uh, you can run a process inside of it. Maybe you have, like, your enrich note skill that I was showing you earlier.

87. 15:54

    Just instruct the agent, "Run enrich note, uh, across end notes that are, uh, not enriched yet," and then let it do all of its tool calls, encode diffs, and then by the time it's done, it'll sync it back up.

88. 16:07

    That's really the flow. It's super simple. Uh, and that's what we've instructed this cloud runner to do. So if you're unfamiliar with Oz, no problem. Uh, all this is, is a cloud sandbox that'll stand up every day, and it'll instruct the agent to do this.

89. 16:21

    So this is the prompt. And the prompt is, "I gave you the Obsidian CLI in your environment. Use that, pull down that wiki, go ahead and update it," and I put in some special instructions for my own setup, "and then sync it back up," so that way when I come back to my computer in the morning, I

90. 16:37

    wake up to a perfectly fresh wiki that I can review. It's like the daily paper, but it's your own, which is so exciting. Uh, we see another schedule that I set up over here for the enrich note phase, generating those backlinks, making it easier for you to navigate around everything.

91. 16:51

    I made that a skill. Again, we tell it to sync everything down. We tell it to run enrich note, and then we push it back up. So by the time I go on my computer, there everything is.

92. 17:01

    Uh, and I can show you one of these runs. You can actually view it in your browser. Uh, so over here, this is where it was running enrich note across all of our notes from earlier.

93. 17:10

    And we can see that same thing I was showing you, but this was from the cloud. So while I was sleeping, it looked at all the notes that I took throughout the day, and it tied them all together in something that was easy to browse, uh, which is super clean, super easy to use.

94. 17:24

    Uh, and we can also take a look back if we want to at that run from earlier just to see if it did its job. Yeah, look at that.

95. 17:31

    Um, so this was that note earlier that was just a jumbled mess. Now we're seeing, "Here's the podcast." I did some web search. I found it. So if you want to go listen to it again, there it is.

96. 17:39

    We have a heading now, and we have some related notes, the Ferrarium, um, podcast. We have some notes from some other founder podcasts. There we go. Don't have to do any work anymore.

97. 17:47

    All you have to do is voice dictate in your phone, and all this stuff comes out on the other side.

98. 17:52

    The last thing I want to talk about, uh, just to expand your mind a bit beyond the Markdown, doing something a little bit more exciting, is turning all of your thoughts into visualizations, graph views, charts, whatever you want to do.

99. 18:07

    Uh, and that's something that I've set up inside of this, uh, notes app I've been showing you this whole time. You can also look at HTML pages inside of here.

100. 18:14

     So I told it because I wanted to, like, "Hey, I want to take all these Markdown files, and I want you to just build with HTML and Tailwind some sort of graph view."

101. 18:23

     So instead of me clicking around a wiki and following a bunch of links, I want to see a bird's-eye view of everything that we're writing down here, figure out common patterns and common areas of interest.

102. 18:33

     And we can look at them right here. And this is not a tool that you have to install, by the way. I told an agent, "Build this for me," so you can do that now.

103. 18:41

     Um, so we see some stuff in the center that's very stray thoughts. I'm a scattered person. It makes sense. Uh, we have some books. Clearly, I need to read more.

104. 18:49

     We have some interest in, like, startup founding here. We have AI and engineering. Of course, we have faith and scripture because I was reading the Bible in a year, as I mentioned earlier.

105. 18:57

     We see all of that grouped up inside of here. And I also told it, "Let's make these nodes something clickable." So now I can see whatever note that was.

106. 19:06

     I can see how it's connected to something else, so we can see this is connected to a number of ideas. You just kinda walk around the graph. It's useful just to get an idea of what you're actually interested in and where you have gaps in your thinking, but it's also useful if you wanna drill down.

107. 19:20

     And also just to show you this is something that you control, it's like, can, can you put it in space? I don't know. Let's put it in space. How about that?

108. 19:27

     Now it's in space. Uh, so it looks a little bit more like a star constellation. So we can click in there, we can sorta look around, see how everything is connected.

109. 19:36

     It's really nice. You could also connect into like a burndown chart if you're trying to build up a habit tracker of how often you're writing notes on certain things.

110. 19:44

     You can invent one of those. This is kind of like a GitHub chart. Clearly, I don't have a super consistent habit, but it's building, and we can track it inside of here.

111. 19:52

     Uh, whatever you wanna put together, you can just ask an agent to do it. Um, and that's really, that's really the end of the talk. That's everything I wanted to share with y'all.

112. 20:01

     If I have any sort of parting gifts or things to check out, I'll first call out, like, if you want that automation platform, oz.dev, uh, it's built by Warp, warp.dev/oz, however you wanna look it up.

113. 20:12

     Uh, this is the tool that you can use to set up schedules to update notes in the background. And if you wanna try that little notes app I was showing you, hubble.md.

114. 20:21

     It's free and open source. We're open to contributions. There's a lot of things missing, as you can imagine. Uh, so if you're looking for an Apple Notes that's agent-accessible, here we go.

115. 20:30

     This is what you can go try out. I hope to see you in the issue logs. Uh, with that, uh, it's been great chatting with y'all. I will, uh...

116. 20:38

     Oh, I'm supposed to plug myself, aren't I? Let me plug myself. There we go. If you need to follow me, there I am. [laughs] I'm, uh, developer relations at Warp.

117. 20:46

     You can find the whole team, uh, both at the booth and also on Twitter and everywhere else. Uh, and also our side event, which we have tonight. Um, so yeah, with that, I'll let y'all go.

118. 20:55

     Hope you enjoy the rest of the conference. Thank you so much. [audience applauding] [outro music]