1
02:59:08,220 --> 02:59:09,300
All right, hello everyone.

2
02:59:09,820 --> 02:59:11,000
My name is Jeff Pinner.

3
02:59:11,140 --> 02:59:12,100
I'm the CTO of EXA.

4
02:59:12,320 --> 02:59:13,640
Thank you guys for having us here.

5
02:59:13,840 --> 02:59:19,400
And we're going to talk a little bit today about things your agent doesn't know.

6
02:59:22,620 --> 02:59:27,880
So Web Search has been around now for about 30 years, long enough where at this point

7
02:59:27,880 --> 02:59:31,280
we're kind of used to it and we don't really question the status quo or how it could be

8
02:59:31,280 --> 02:59:32,900
different with Agents.

9
02:59:34,500 --> 02:59:39,680
Over the last 18 months or so, Search has picked up Agents as sort of a new customer.

10
02:59:39,920 --> 02:59:41,220
They're searching on our behalf.

11
02:59:41,860 --> 02:59:46,420
And so at Exo, we specifically build Web Search as a knowledge layer for Agents.

12
02:59:47,200 --> 02:59:52,800
We've gone from about 12.5 million requests a month two years ago

13
02:59:52,800 --> 02:59:56,240
to a little over 1.3 billion this past October,

14
02:59:56,380 --> 02:59:57,880
so about 100x over two years.

15
02:59:58,020 --> 02:59:59,800
And it's not unique to Exa.

16
03:00:00,520 --> 03:00:10,700
If you talk, OpenAI just announced that ChatGPT handles about a fifth of Google search volumes, and agents in general make up just around half the volume of web traffic now.

17
03:00:11,720 --> 03:00:15,760
And so I think the natural question is, why do agents need search at all?

18
03:00:15,860 --> 03:00:21,360
So let's start with trying to answer this before we talk about how we do search for agents.

19
03:00:24,060 --> 03:00:28,840
So frontier models can do a lot of tasks from parametric knowledge.

20
03:00:30,060 --> 03:00:34,220
But unfortunately for real-world tasks, that simply isn't enough.

21
03:00:35,680 --> 03:00:40,060
Generally, frontier models, no matter how capable they are, are always out of date.

22
03:00:40,400 --> 03:00:46,540
Model launches take two to five months from training cutoff to launch.

23
03:00:46,720 --> 03:00:50,680
Then there's generally a few months after that before the next frontier model is released.

24
03:00:50,680 --> 03:00:56,600
And so taking that together, you're about four to eight months out of date with whatever the latest knowledge base is.

25
03:00:56,720 --> 03:01:04,520
And in the meantime, the world doesn't stop changing just because a model has moved from training to inference.

26
03:01:05,940 --> 03:01:11,960
Libraries launch, APIs change, prices change, new regulations appear, court rulings happen.

27
03:01:12,920 --> 03:01:14,480
The world continues to go.

28
03:01:16,840 --> 03:01:22,460
And even if that weren't true, training data itself is actually just a small amount of

29
03:01:22,460 --> 03:01:23,900
knowledge that the model is trained on.

30
03:01:25,240 --> 03:01:29,600
The biggest models out there are trained on around 50 trillion tokens.

31
03:01:30,080 --> 03:01:32,860
The web itself is around 3,100 trillion.

32
03:01:33,180 --> 03:01:36,800
And that includes the deep web behind things like login and apps.

33
03:01:37,000 --> 03:01:40,920
If you focus it down to just the web that is indexed, the thing that search engines

34
03:01:40,920 --> 03:01:44,360
can see, that's still about 510 trillion tokens.

35
03:01:44,480 --> 03:01:48,700
which means even if the world didn't continue to evolve

36
03:01:48,700 --> 03:01:50,040
after the model cutoff happened,

37
03:01:50,300 --> 03:01:54,000
you would still only be indexing a very small amount

38
03:01:54,000 --> 03:01:57,280
of the knowledge that's out there that models have to go and find.

39
03:02:00,640 --> 03:02:04,040
And for the type of tasks we want models to do,

40
03:02:04,180 --> 03:02:05,340
we want agents to do,

41
03:02:05,560 --> 03:02:08,360
there is valuable work that happens

42
03:02:08,360 --> 03:02:10,080
that is based on outside knowledge,

43
03:02:10,080 --> 03:02:14,680
whether it's across coding, legal work, due diligence work.

44
03:02:15,180 --> 03:02:20,860
Generally, all these tasks require correct, complete knowledge across multiple sources.

45
03:02:21,620 --> 03:02:25,040
And so this is what we're trying to, the gap we're trying to close.

46
03:02:27,620 --> 03:02:30,960
Now, the first thing to do when we're trying to close any gap is to try to measure it.

47
03:02:31,080 --> 03:02:35,860
And so the first thing we do is create a benchmark called Atlas,

48
03:02:36,040 --> 03:02:39,920
which tries to test how well harnesses can leverage real-world knowledge.

49
03:02:42,880 --> 03:02:49,200
So we created the Atlas benchmark to try to test agents in these tasks.

50
03:02:50,440 --> 03:02:54,320
First, we require agents to find entities that match a condition.

51
03:02:56,840 --> 03:03:00,820
Then we require agents to enrich the knowledge with as much information as they can find.

52
03:03:03,600 --> 03:03:07,280
And finally, we score them against the golden answer key.

53
03:03:10,080 --> 03:03:15,700
and we only count rows as correct that correctly identify all aspects of what we're looking for.

54
03:03:16,320 --> 03:03:19,440
And so to succeed at Atlas, you need to be consistently right

55
03:03:19,980 --> 03:03:22,740
and not just have a decent hit rate for individual searches.

56
03:03:26,600 --> 03:03:29,480
So we're going to use this as a benchmark for how we measure

57
03:03:30,220 --> 03:03:32,600
agent performance when it comes to knowledge retrieval.

58
03:03:33,420 --> 03:03:38,500
And the first thing that we find is that model memory falls off with recency.

59
03:03:40,280 --> 03:03:51,500
When we test agents on Atlas, we see that model success rates fall off sharply as soon as they enter, as soon as they encounter data that's not covered in the training set.

60
03:03:51,980 --> 03:04:01,100
The newer the fact is, the less the models know, and models require, or I should say agents require data to try to achieve these tasks.

61
03:04:01,300 --> 03:04:06,080
And so the first thing that we can see is that memory alone, parametric memory isn't sufficient.

62
03:04:07,840 --> 03:04:10,160
So when we run agents with just parametric memory

63
03:04:10,160 --> 03:04:13,280
without web search, they score around 9% on Atlas,

64
03:04:13,540 --> 03:04:16,180
and so we'll use this to try to understand quantitatively

65
03:04:16,680 --> 03:04:18,200
how web search improves performance.

66
03:04:22,060 --> 03:04:23,800
So the next natural question is,

67
03:04:23,960 --> 03:04:25,300
why don't they just Google it?

68
03:04:28,360 --> 03:04:31,460
And the first reason is agents don't search like people.

69
03:04:32,700 --> 03:04:35,000
Generally, people type about three words into Google,

70
03:04:35,000 --> 03:04:38,180
and that's up from about two words six years ago.

71
03:04:39,200 --> 03:04:40,480
Agents, on the other hand,

72
03:04:41,140 --> 03:04:43,160
search with more or less full sentences.

73
03:04:43,520 --> 03:04:45,840
The mean search on Exa is about 12 words,

74
03:04:46,320 --> 03:04:49,100
about 10% of queries on Exa are 17 words or higher

75
03:04:49,100 --> 03:04:51,380
and these words add constraints,

76
03:04:51,700 --> 03:04:54,000
versioning information, dates, places,

77
03:04:54,360 --> 03:04:56,440
all different types of conditions on the search

78
03:04:56,440 --> 03:04:59,340
that keyword ranking wasn't really designed to handle.

79
03:04:59,580 --> 03:05:01,580
And so we'll take a look at an example of that.

80
03:05:03,440 --> 03:05:05,540
So this is an actual query from EXA.

81
03:05:05,680 --> 03:05:06,920
It's a request for companies.

82
03:05:06,920 --> 03:05:07,820
It's a full sentence.

83
03:05:07,840 --> 03:05:09,140
It adds a bunch of constraints.

84
03:05:09,760 --> 03:05:13,300
And when you feed this into a keyword retrieval algorithm,

85
03:05:13,300 --> 03:05:15,260
what you really get is a handful of results

86
03:05:15,260 --> 03:05:18,260
that don't really answer what the query asked for.

87
03:05:18,840 --> 03:05:20,480
The search didn't understand the query.

88
03:05:21,080 --> 03:05:24,540
The result, the third one, is specifically in the U.S.,

89
03:05:24,540 --> 03:05:26,560
and the result asked for data outside the U.S.

90
03:05:29,980 --> 03:05:32,020
And so we're going to come back to this in a few minutes

91
03:05:33,360 --> 03:05:35,140
Because the second thing I want to focus on

92
03:05:35,140 --> 03:05:37,080
Is the fact that for a lot of knowledge on the web

93
03:05:37,560 --> 03:05:39,280
Search just doesn't have access to it

94
03:05:39,740 --> 03:05:42,340
Most of the valuable information isn't on the open web

95
03:05:42,340 --> 03:05:45,680
Reported news, people data, financial data, research

96
03:05:45,680 --> 03:05:46,760
They're all paywalled off

97
03:05:46,760 --> 03:05:49,500
And so search engines that crawl the open internet

98
03:05:50,080 --> 03:05:51,740
Don't ever see these pages

99
03:05:54,080 --> 03:05:57,860
And then lastly, even when knowledge is on the open web

100
03:05:57,860 --> 03:05:59,740
and your agents can access it,

101
03:05:59,980 --> 03:06:01,600
there's a third problem, which is efficiency.

102
03:06:02,980 --> 03:06:05,840
You know, if an agent makes 10 searches,

103
03:06:06,300 --> 03:06:08,280
reads five pages from that search,

104
03:06:08,660 --> 03:06:11,840
gets about 16K of raw HTML from each page,

105
03:06:12,060 --> 03:06:16,140
that's about 860,000 tokens just in processing results.

106
03:06:16,380 --> 03:06:18,220
And if you remember from the first slide,

107
03:06:18,380 --> 03:06:21,540
we serve about 1.3 billion requests a month.

108
03:06:21,740 --> 03:06:25,700
So that's a lot of tokens that are just churning to extract information.

109
03:06:28,600 --> 03:06:31,040
And then finally, when you add keyword search,

110
03:06:31,180 --> 03:06:33,240
and we look at what happens to our Atlas benchmark,

111
03:06:33,600 --> 03:06:35,080
the score definitely improves.

112
03:06:35,100 --> 03:06:36,380
It's added recent knowledge.

113
03:06:37,020 --> 03:06:39,000
But we've gone up to about 28%

114
03:06:39,000 --> 03:06:42,500
because most of the rows still come back with incomplete data

115
03:06:42,500 --> 03:06:43,740
or just wrong values.

116
03:06:45,620 --> 03:06:47,560
And so the challenge when we look at search

117
03:06:47,560 --> 03:06:50,920
is how do we actually provide the knowledge to agents

118
03:06:50,920 --> 03:06:52,260
that they actually require.

119
03:06:56,440 --> 03:06:57,500
So the first thing,

120
03:06:57,640 --> 03:06:58,920
I told you we'd come back to this query,

121
03:06:59,220 --> 03:07:01,020
is, at least at EXA,

122
03:07:01,020 --> 03:07:04,260
the first thing we do to try to tackle this task

123
03:07:04,260 --> 03:07:06,320
is to extract meaning out of information

124
03:07:06,320 --> 03:07:08,240
as opposed to just syntactical matches

125
03:07:08,240 --> 03:07:09,040
or keyword matches.

126
03:07:09,500 --> 03:07:11,600
And so we look through pages,

127
03:07:11,620 --> 03:07:13,540
we understand the intent of queries,

128
03:07:14,500 --> 03:07:16,620
and here we give you results

129
03:07:16,620 --> 03:07:18,560
that specifically answer the intent

130
03:07:18,560 --> 03:07:20,480
with all of the conditionals attached.

131
03:07:22,260 --> 03:07:27,920
And so in order to do this, we do what you would for a modern web crawler.

132
03:07:28,000 --> 03:07:33,640
We crawl the internet, we do a semantic extraction, we embed it in the vector space, we index it.

133
03:07:34,020 --> 03:07:38,940
When your query comes in, we embed the query in the same vector space, and we find the nearest matches.

134
03:07:39,460 --> 03:07:49,060
And this lets us move from syntactical understanding and keyword matching to a semantic understanding of the query in the web.

135
03:07:49,060 --> 03:07:53,380
and we do this at a volume of up to a million pages a second

136
03:07:54,300 --> 03:07:57,860
with a latency of about three minutes from crawl to index.

137
03:08:00,420 --> 03:08:01,840
Now for the second problem,

138
03:08:02,000 --> 03:08:04,920
we need to extend it just beyond the open web.

139
03:08:07,080 --> 03:08:10,880
And so here we have to extend what's on the public web

140
03:08:10,880 --> 03:08:15,180
with both proprietary data that's partnered

141
03:08:17,660 --> 03:08:20,720
and knowledge graph data that we extract and represent.

142
03:08:21,080 --> 03:08:23,660
And this way, the agent can retrieve questions

143
03:08:23,660 --> 03:08:25,540
and answers that it needs to its questions

144
03:08:25,540 --> 03:08:27,300
without having to know where to look for them,

145
03:08:27,660 --> 03:08:29,320
doing this behind a single API.

146
03:08:33,760 --> 03:08:35,340
And for the last challenge,

147
03:08:39,320 --> 03:08:42,960
in order to support the volume of queries that agents make,

148
03:08:42,960 --> 03:08:45,320
We can't simply point them to webpages.

149
03:08:45,460 --> 03:08:47,380
We have to extract the answers from the page

150
03:08:47,380 --> 03:08:49,260
and create a token-dense representation

151
03:08:49,260 --> 03:08:50,840
for agents to process.

152
03:08:51,020 --> 03:08:53,420
It's not sufficient to say the answer to this query

153
03:08:53,420 --> 03:08:54,840
lives on this web page.

154
03:08:56,220 --> 03:08:58,300
You could take, for example, this question

155
03:08:58,300 --> 03:09:00,620
of who won the Nobel Prize for lithium-ion batteries.

156
03:09:00,940 --> 03:09:03,140
If you just point to the Wikipedia page,

157
03:09:03,240 --> 03:09:04,780
you'll get thousands of characters.

158
03:09:05,700 --> 03:09:07,740
The actual excerpt that matters

159
03:09:07,740 --> 03:09:09,720
for citation and grounding is about 500.

160
03:09:09,720 --> 03:09:15,120
So there's about 16x more tokens when you have to process data directly from Google

161
03:09:15,120 --> 03:09:17,180
than when you can extract that data.

162
03:09:17,360 --> 03:09:19,880
And those extractions have to be semantically rich.

163
03:09:20,020 --> 03:09:24,920
You have to pick a segment of the text that answers the question, and so the query matters.

164
03:09:27,040 --> 03:09:28,640
And so what does this actually get us?

165
03:09:30,080 --> 03:09:33,760
When we measure with Atlas, again, to ground ourselves,

166
03:09:34,660 --> 03:09:36,980
we've seen that we've improved from about 9%

167
03:09:36,980 --> 03:09:38,760
on our closed book test

168
03:09:38,760 --> 03:09:41,600
to about 56% when we do a semantic extraction

169
03:09:41,600 --> 03:09:43,860
over the internet and extended knowledge.

170
03:09:45,200 --> 03:09:48,020
So not only do we get about 5x higher of a score,

171
03:09:49,320 --> 03:09:50,100
excuse me,

172
03:09:50,700 --> 03:09:53,060
but we also reduce the cost by about 28%

173
03:09:53,060 --> 03:09:54,900
due to the token efficiency of the extraction.

174
03:09:55,620 --> 03:10:00,000
And so we start to demonstrate quantitatively

175
03:10:00,060 --> 03:10:04,260
that improving access to knowledge actually improves performance of the underlying agent.

176
03:10:06,360 --> 03:10:20,280
And so, yeah, in general, I think the main takeaway that we've hopefully expressed is that while reasoning capabilities on our frontier models are relatively fixed,

177
03:10:20,280 --> 03:10:23,520
A lot of improvement can be done in agentic performance

178
03:10:23,520 --> 03:10:26,000
by extending the harness and giving it access

179
03:10:26,000 --> 03:10:27,480
to semantic meaning across the Internet.

180
03:10:28,520 --> 03:10:29,740
That's it. Thank you.

181
03:10:31,020 --> 03:10:31,580
Thank