Building a Smarter AI Agent with Neural RAG

Talk transcript

1. 0:00

   [upbeat music] All right, so I was gonna give a live demo coding. [laughs]

2. 0:21

   No. [clapping] But, uh, well, I will, but I know you all are actually here to hear a cool story. So I'll tell you a story about web search built for AI, and then we do some coding at the end.

3. 0:32

   And the story will end with this slide, uh, one API to get any information from the web.

4. 0:39

   And you'll know what this means by the end. But the story starts in 1998,

5. 0:44

   and what you're looking at is the, the state-of-the-art in information retrieval in 1998. You type in a word, Australia, to this new search engine called Google, and it magically finds you all the documents that contain the word Australia from the web.

6. 0:58

   It's crazy. Um, and the, the big insight of Google was they had this PageRank algorithm, so, uh, the results are, are ranked by authority based on the graph structure of the web.

7. 1:08

   And this was a clever algorithm, and it was really cool. I was [REDACTED:age] at the time. So if I was conscious, I would have thought this was cool.

8. 1:15

   Um, okay, and now our story, our, now our story, uh, skips twenty-three years to 2021. Um, by this point, I was conscious, barely. And, uh, [laughs] uh, and I, I noticed that, you know, GPT-3 had recently come out, and it was this magical thing that you could input a whole paragraph explaining exactly what you want, uh, and it

9. 1:36

   would really understand the subtleties of your language and give you an output that exactly matched. Um, and it's hard to remember how magical this was, but it was really magical in 2021.

10. 1:45

    And at the same time, I noticed there was Google, which, you know, you type in a simple query like shirts without stripes, and it would give you shirts with stripes, which is crazy.

11. 1:54

    Uh, [laughs] it, like, it doesn't understand the word without, um, because it's doing a keyword comparison algorithm. And so I decided that for the next at least ten years, I'm gonna devote myself to building a search engine that combines the technology of GPT-3, uh, to, with a search engine to make a search engine that actually understands what you're

12. 2:13

    saying, uh, at a deep level and understands all the documents on the web at a deep level and gives you exactly what you asked for. This is a very big idea, and, and we're working on, we've worked on it for four years and, uh, a lot of progress.

13. 2:26

    But it would change the world if, if you actually solve this problem. And so in 2021, uh, we, we, we joined YC. Summer 2021, uh, we raised a couple million dollars, and we did what every YC startup should do.

14. 2:37

    We spent half of it on a GPU cluster. I'm joking. You shouldn't do that. [laughs]

15. 2:43

    Um, and, and then we also followed YC's advice, uh, where we didn't talk to any users or, or customers for a year and a half, and we just did research.

16. 2:52

    Um, again, you shouldn't do that. You should talk to users. But in our case, it made sense because we were trying to solve a really hard problem, which is like redesign search from scratch, um, using the, the same technology as GPT-3, this, like, next-token prediction idea with transformers.

17. 3:04

    What if you could apply the same thing, uh, to search? And this is actually one of our, uh, WanDB training runs. Um, the purple one, I believe, is, was a breakthrough where it, like, really, it really, like, learned.

18. 3:13

    Uh, there was, like, a, a few breakthroughs along the way, uh, involving, like, random datasets and different, uh, transformer architectures that we were trying. And this purple one, like, really it starts to, like, work well.

19. 3:23

    Um, and the general idea we had was like, okay, so yeah, what is, what is a search engine? You have, like, a trillion documents on the web. Um, and traditional search engines, uh, on a very high level will create, like, a keyword index of those documents.

20. 3:35

    So for each document, you, you are, you say, you ask what are the words in those document, and you create this big inverted index where you map from, like, words like brown to all the documents that contain that word.

21. 3:45

    Um, and then at search time, you know, when a shirts without stripes comes in, you do some crazy keyword, uh, comparison algorithm and get the top results. That's obviously a simplification of what Google does, but at a fundamental level, it's doing, it's like a keyword comparison.

22. 4:00

    But the idea was, like, what if you could actually... So with transformers, like, the, the big thing is, like, what if you could turn each document not into a set of keywords but into embeddings?

23. 4:08

    Uh, and these embeddings can be arbitrarily powerful, right? Like, it's a list of... An embedding is just a list of, of, of numbers, and, uh, it could represent lots of information, so.

24. 4:16

    An embedding, it doesn't just capture the words in the document, but also the meaning, the ideas in the, in the document and the way people refer to that document on the web.

25. 4:23

    And, you know, an embedding can be arbitrarily big, and so it, like, of course, uh, in the limit, it would just destroy keywords. And so you have this, like, arbitrarily powerful representation.

26. 4:32

    Um, and now the fundamental idea was just, like, the bitter lesson. What if we could, like, you know, train transformers to output embeddings for documents? And if we keep getting more and more data that's high quality, we could, uh, make a search engine that actually understands you.

27. 4:43

    And, um, the way it would work at inference, at, at search time is, like, a search comes in, a query comes in like shirts without stripes. Traditional search engines would use the above thing where they would do a, a very fancy keyword comparison and a bunch of other things.

28. 4:55

    Um, and then instead, we would just embed the shirts without stripes and compare it to the embeddings of all the trillion documents.

29. 5:02

    And, you know, after a year and a half, we actually had a new search engine that worked in a very different way. Uh, and you search shirts without stripes on Google-- I'm sorry, on Exa, [laughs] and you, um, you get a list of results that actually, uh, are not, do not have stripes.

30. 5:13

    Uh, it's a simple, uh, example, but, like, you could, uh, it could handle, like, more, way more complex queries, like paragraph-long queries.

31. 5:21

    And when we launched this in November 2022, we got a lot of excitement on Twitter. Um, this is a very new paradigm for search. You could do all sorts of interesting queries that you couldn't do before.

32. 5:30

    And then two weeks later, this happened. It was a small tweet. [laughs]

33. 5:36

    And, uh, this is a visual depiction of San Francisco at the time. [laughs] Um, you guys probably all re-remember this. [laughs]

34. 5:44

    And then this is a visual depiction of the Exa team at the time. [laughs] Because ChatGPT completely changed the way we interact with the world's information. You know, like, everyone can now use an LLM to just, like, talk, talk to their computer and, and get information.

35. 5:58

    And we were thinking, "Wait, is there even a role for search in this world? Like, these LLMs are so powerful." And then very quickly we realized, yes, there is a role because LLMs don't know everything on the web.

36. 6:08

    So for example, if you ask an LLM like GPT-4, "Find me cool personal sites of engineers in San Francisco," um, it'll, it, it can't. Like, it just doesn't have that in the weight, so it'll apologize, whatever.

37. 6:18

    Um-

38. 6:19

    And, you know, there's a very simple information theory argument here where it's like there literally isn't enough information in the weights of GPT-4 to store the whole web. GPT-4, call-- like, we don't know exactly how many, uh, parameters.

39. 6:29

    I think someone leaked it on YouTube once. But it's like, oh, you know, a couple trillion parameters. You could call it, like, less than ten terabytes, uh, in the weights of GPT-4.

40. 6:36

    And then the internet is, like, over a million terabytes, and that's just the documents on the web. Uh, there's also images and video, and it's way more. Um, actually the, the, the web-- if you look at-- I, I did a tweet recently about the, the size of the web, and it's r- it's in the exabyte range, um,

41. 6:51

    and our name is Exa. It's not a coincidence. Um, anyway, so, like, uh, uh, LLMs, uh, need to search the web just from this simple argument, and they're gonna need to do that for a long time, which, um, if you talk to ML researchers, they'll say the same thing.

42. 7:02

    It's just like, it, it's too hard. Also, the web is constantly updating. That's another problem. It's not just the size of the web, it's the constant updating-ness of the web that makes it very tricky.

43. 7:08

    So LLMs always will need search. That's great. Um, and so when you combine an LLM with a search engine like Exa, you can handle these, uh, queries. So, like, find me cool personal sites and engineers in SF.

44. 7:19

    Uh, the LLM will search Exa, get a list of personal sites, uh, and then, like, use that information to output the perfect thing for the user. You're all very familiar with this.

45. 7:28

    Like, LLMs plus search, it's obvious now, right? Like, everyone knows about it. But now let me tell you a secret about search that most people don't know. Um,

46. 7:38

    and the secret is that traditional search engines were not built for this world of AI. Traditional search engines were built for humans, uh, and humans are not-- are very different from AI.

47. 7:49

    Uh, so every search engine like Google, Bing, you name it, uh, was built in a different era for this kind of creature. Uh, this, this slow flesh human that's typing keywords and wants to read a few links and really cares about UI of the page and all these things.

48. 8:04

    Like, it's a lazy human. They type simple keywords. Google is great for this creature. Um, [chuckles] Google was optimized for this creature. It gives you exactly the kinds of things you would click on.

49. 8:14

    But AIs are very different. Um, this-- like, an AI can gobble up information like crazy. This is a much slowed down version of what our AIs, AIs probably feel like inside.

50. 8:24

    Uh, and so AIs are very different. They wanna use complex queries, not simple ones, to find not a couple links, but just tons of knowledge, as much knowledge as they could get, 'cause they actually have the patience to just analyze it all extremely fast.

51. 8:35

    And so the o- the, the search algorithm that's optimal for this type of creature is not the same [chuckles] algorithm that's optimal for the human. Like, that would be crazy if the same algorithm that was optimal for humans was optimal for, uh, AIs.

52. 8:48

    And so, like, all the t- a lot of the tools, the search tools that we're talking about these days on Twitter and stuff like that, they're still using, like, the old traditional search combined with AIs.

53. 8:57

    It's just not the right puzzle fit. Um, so Exa, we're really trying to think of, like, what is the right search engine for this AI world.

54. 9:04

    And so just a few examples, uh, we could dive deep into, um, to, to-- of how AIs are different. Well, AIs want precise, controllable information. So, oh, by the way, when I say AI, I'm usually-- I'm talking about, like, an AI product.

55. 9:16

    So imagine, like, in this case, like, a VC that's using, uh, an AI system to find a list of companies, uh, because they wanna invest. So, you know, they're looking for some-- what's the next big thing?

56. 9:24

    What's the next big thing that feels like Bell Labs? Well, when they tell their AI what they want, the AI will then go search a search engine, right? And if it searches a search engine like Google, it'll get a list of results that humans like to click on, but it's not very information dense, and it doesn't even

57. 9:37

    match what the person asked for. The, uh, or, sorry, what the AI asked for. The AI asked for startups working on something huge that feels like Bell Labs. It should get a list of startups.

58. 9:44

    It's kind of a crazy idea, but what if search engines actually returned exactly what you asked of them and not what you want to c- what Google knows you'll click on?

59. 9:52

    And so with AIs especially, they just want a search engine that returns exactly what they asked for. Because what's, what really the world's gonna look like is you're gonna interact with your AI agent, and you're gonna a-ask for something, and then it's gonna make tons of searches.

60. 10:03

    Like, okay, maybe they want startups working on something, like, similar to Bell La- Bell Labs. Maybe they want startups working only in New York City that have this quality and that quality.

61. 10:09

    And, and, uh, and it'll do all sorts of searches, and it just wants a search API that just does what it asks. And, and so you need a search engine like that.

62. 10:16

    So Exa's like that. Um, another difference between AIs and humans is AIs wanna search with lots of context. Again, if you're-- if you have an AI assistant and you talk to it all day, and then you ask for restaurants or apartments or, or what have you, uh, the AI has lots of context on you.

63. 10:30

    So it should be able to search with this, uh, large multi-paragraph thing saying like, you know, "My human is a software engineer, and it likes these types of things, and I like these types of things," and like, "Can you give me, uh, you know, restaurants that match those preferences?"

64. 10:42

    Uh, and so you need a search engine that could literally handle multiple paragraphs of text. But traditional search, like, search engines like Google were not meant to do that because humans would never type in multiple paragraphs 'cause they're too lazy.

65. 10:52

    And so Google was optimized for, like, simple keyword queries. So Google, I think, has, like, a, a few dozen keyword limit. Uh, whereas, uh, Exa can handle, like, multiple paragraphs o-of text.

66. 11:02

    Another big one where AIs are different than humans is AIs want comprehensive knowledge. Uh, like, if you give a human 10,000 links or 10,000 pages, it doesn't know what to do with that.

67. 11:11

    Like, it would take ten days of extreme patience to process all that. But AIs can do it in three seconds if it's parallelized, right? So if I'm an, a, a VC and I want a report on, like, all the companies in a space, I want literally all the companies, and there's a huge amount of value to getting

68. 11:26

    truly all of them and not just, like, the 10 or 20 that Google is able to find. And so you need a search engine that exposes the ability to return 1,000, 10,000, whatever it is, uh, and also has this semantic ability to, like, you know, when you say, like, every startup funded by YC working on AI, you

69. 11:40

    actually can get all of them. So, like, Google literally just can't do this at all.

70. 11:45

    Okay, I hope that through these examples, we see that the space of possible queries is actually, like, way larger than people realize. Uh, and u-until, like, 2022, we were kind of in this, like, top left blue world.

71. 11:57

    Uh, so the, the circle is, like, the space of possible queries, and the blues are, like, uh, you know, specific subsets of that space. And so, like, we were all in that top left corner of blue for a long time, where you could, you know, we could ha- uh, search engines could handle, like, uh, like, basic keyword

72. 12:12

    queries like Stripe pricing or, uh, someone's GitHub page or Taylor Swift's boyfriend or whatever it is. Uh, after 2022, everyone started to want the top right blue, uh, circle, where it was like, "Hey, uh, actually I wanna make queries like explain this concept to me like I'm a five-year-old," or, "Here's my code.

73. 12:28

    Can you, like, debug it?" The-this is a form of query. It doesn't require search, but it's a, it's a, another type of query that was introduced to the world in 2022.

74. 12:36

    And then, like- Uh, there's other types of queries like these semantic queries, like people in San Francisco who know Assembly. Uh, as far as I'm aware, E-Exa is like the-- I mean, Exa kinda like introduced this kind of query and, and, uh, and does re-really well on them, on, on those queries.

75. 12:49

    And then there's these like really complex queries like, find me every article that argues X and not Y from an author like Z. And we're starting to now have systems like, uh, Exa's like websites product that could handle these things.

76. 12:59

    And I think this is actually a huge space because this like turns the web into like a database you could filter however you want, and that's really what AIs want.

77. 13:06

    They want this like full control database like query system that they could just get whatever they w- need for their user. And then there are the queries that no one has thought of yet.

78. 13:14

    Um, like every week, we get tons of queries and like, "Oh, wait, that's a really interesting type of query that, uh, [chuckles] that no search engine could do right now."

79. 13:21

    And, and eventually we'll try to, you know, handle all the, the queries that are possible. But there, there are so many new types of queries now because we have these AI systems and the stakes, like the, the expectations have just gotten way higher.

80. 13:33

    Okay, so now you c- we end our story, uh, with the same slide. One API to get any information from the web. So again, like Exa's trying to, if you go back, like handle not just like the keyword queries, but also the semantic queries, and also the super complex queries, and eventually all queries.

81. 13:50

    Um, we, we want one API that could like give these AI systems whatever knowledge they want. You have the AI and you have Exa providing, uh, the knowledge. Oh, I only have four minutes.

82. 13:59

    Okay. [chuckles] Um, okay. So that's-- Oh, let's see. Oop.

83. 14:06

    How do I go to a different part of my computer? Hmm.

84. 14:13

    Uh, if I change to the code editor, how do I do that? Let's see. What? Oh, it's there. Oh, but I can't see it. That's so weird. Use the mirror down here.

85. 14:24

    Oh, cool. Okay. [laughs] Okay, um- Are you using the mirror? There we go. Okay. Cool. Well, first of all, just, just very quick explanation of this is our, our, our search dashboard.

86. 14:35

    We could try different queries. I would just point out like in the search, uh, API endpoint, uh, you know, we exposed lots of different toggles. So first of all, you just try out a query and get, uh, it shows you the code, and it gets you, uh, a list of results.

87. 14:50

    Uh, and it exposes tons of different types of filters that you might wanna do. For example, like number of results, ten, a hundred, a thousand, whatever it is. Uh, you could have like date ranges or, you know, I only wanna search over these domains.

88. 15:00

    And it's a lot of toggles, but I think the, the point is actually you want the toggles 'cause your AI is actually gonna be calling this. You want a search engine that gives you full control.

89. 15:07

    Um, and we have like neural and keyword search, so you could try different ones. Um, okay, let me sw- quickly jump to the, the code. Okay, so I prepared this like code, uh, agent.py.

90. 15:19

    So we made this agent, uh, agent Mark, and Mark loves to make markdown out of things. Anything you give it, it'll make markdown-- Mark will make [chuckles] markdown. Uh, and so in this case, uh, we're going to here.

91. 15:32

    Well, I guess in this case, let's try, uh, this query, uh, personal site of engineer in San Francisco who likes information retrieval.

92. 15:42

    Uh, well, this is, this is the kind of a query that neural would be a lot better at. Whoa.

93. 15:48

    Okay. Have to save it. Oh, we're running the wrong agent.

94. 16:00

    Okay, so it's just, it's making a query to get like a list of personal sites of engineers in San Francisco who like information retrieval, and, and Mark, the agent, is just making a markdown output of that.

95. 16:08

    That's a very neural type query. You also might want, uh, do, uh, a different type of query, which is like a more keyword heavy one. Let's see, like, um ...

96. 16:26

    Will Bryk GitHub. Let me add my GitHub. So okay, so here I would want to make a keyword query. So you just change the keyword.

97. 16:40

    Search. So it's gonna get information from my, from my GitHub using keyword, uh, search, 'cause this is a very typical like Google like search that would work well, right?

98. 16:47

    Oh, God, I'm running this wrong one again. Okay.

99. 16:52

    Cool. That's information about Will, Will Bryk's GitHub. Um, and then, okay, so when you're actually building an agent, you're gonna be combining lots of different types of searches. So neural searches and keyword searches, uh, and all sorts of other searches that Exa exposes.

100. 17:04

     So like the right agent in the future is going to be this system that decides what type of search it needs, uh, for, uh, whatev-whatever the user, uh, says.

101. 17:12

     Like, it'll, it'll be like, "Oh, okay, I'm gonna make like a neural search to get a list of things, and then for each one I'm gonna do a keyword search," right?

102. 17:17

     You wanna give the, uh, agent like just full access to the world's information in however way it wants. Uh, not just keyword search, but also all these other things.

103. 17:26

     Um, and so here I one-shotted with o3 a GitHub agent which combines these two queries. So first it'll-- It, 'cause I wanna, you know, I wanna get the GitHub of every, uh, engineer in San Francisco who likes information retrieval.

104. 17:40

     Uh, so the agent will make, uh, a neural search to get a list of people, extract the names, and then search those o-using a keyword search to get their GitHubs.

105. 17:49

     And then if you run that ... Here, it's just getting ten results, but we could, you know, with Exa, we could do a hundred or a thousand if you're on an enterprise plan.

106. 17:59

     Uh, [chuckles] so now it's getting all the GitHub info.

107. 18:07

     Cool. So that's just a cl- an example. Um, and yeah, I mean, there are lots of other things that you could do with Exa, like, um, we actually just today r- uh, launched this research endpoint, um, where it will actually do like as much searches in the-- and LLM calls in the background to get you, uh, that

108. 18:24

     perfect report or that perfect structured output for the thing you asked for. So it's kinda like a deep research API, and a state-of-the-art deep resea- deep research API. Um, cool.

109. 18:33

     That is the talk. I hope that was interesting. Thank you. [laughs] [upbeat music]