1
00:00:03,206 --> 00:00:03,686
Okay.

2
00:00:04,646 --> 00:00:05,766
Pleasure to be here.

3
00:00:06,446 --> 00:00:09,166
Another round of applause for everyone at AI Engineering.

4
00:00:09,366 --> 00:00:12,406
This has been a really, really exceptional conference so far.

5
00:00:12,766 --> 00:00:13,486
Thank you.

6
00:00:14,806 --> 00:00:15,306
All right.

7
00:00:15,466 --> 00:00:19,946
So I want to begin by asking you to think about that person in your company who just

8
00:00:19,946 --> 00:00:21,206
seems to know everything.

9
00:00:21,646 --> 00:00:23,546
They've been through 17 reorgs.

10
00:00:23,626 --> 00:00:25,266
They know all the gossip, all the politics.

11
00:00:25,266 --> 00:00:30,226
and somehow they always seem to predict what's going to come out of these big strategic moves.

12
00:00:31,086 --> 00:00:35,306
That person is a company's world model, essentially.

13
00:00:35,566 --> 00:00:39,426
And the problem is that they don't scale and nobody really writes down what they say.

14
00:00:41,506 --> 00:00:44,446
I'm João Wenzel with Citadel Securities

15
00:00:44,446 --> 00:00:49,186
and I'm going to talk a little bit about how we try to approximate that with an agentic system.

16
00:00:51,166 --> 00:00:52,946
So a little bit about Citadel Securities.

17
00:00:52,946 --> 00:00:56,046
we build our own technology down to the silicon.

18
00:00:56,506 --> 00:00:59,886
At this point, over two-thirds of our employees write code.

19
00:01:00,126 --> 00:01:02,766
So naturally, we are completely AI-pilled,

20
00:01:03,346 --> 00:01:07,346
and agents already help us with a lot of things at the organization.

21
00:01:08,306 --> 00:01:12,446
So naturally, leadership started asking questions about strategy.

22
00:01:12,726 --> 00:01:16,686
These harnesses are really good at producing high-quality code,

23
00:01:16,846 --> 00:01:21,906
but can they actually help define and refine the strategy of our company?

24
00:01:22,646 --> 00:01:25,486
So this is one of my favorite visuals I've borrowed from Tim Urban.

25
00:01:26,126 --> 00:01:28,466
Strategy really comes down to three questions, right?

26
00:01:29,006 --> 00:01:30,026
Where do we stand

27
00:01:30,026 --> 00:01:50,426
right now? What's going to happen next? And how did we get here? And to answer any of those questions, an agent has to have a really rich profile of the company that it's operating in. So this is broadly a talk about engineering an institutional memory layer. Let's call it that.

28
00:01:51,086 --> 00:01:54,626
The agents need to have information about relationships, right?

29
00:01:54,806 --> 00:01:59,146
Two teams that used to work together but no longer talk because there was a reorg.

30
00:01:59,466 --> 00:02:03,946
Or a project story and why it was re-scoped several times.

31
00:02:04,186 --> 00:02:05,866
Or the politics, right?

32
00:02:05,966 --> 00:02:06,946
Like, who's in the room?

33
00:02:07,026 --> 00:02:07,826
What are they saying?

34
00:02:08,006 --> 00:02:09,486
And what are they not saying?

35
00:02:10,766 --> 00:02:12,366
There's two ways in, right?

36
00:02:12,486 --> 00:02:14,326
There's top-down and there's bottom-up.

37
00:02:14,926 --> 00:02:18,346
Top down, you have the decks, you have your strategy docs,

38
00:02:18,586 --> 00:02:20,446
your meeting and status reports,

39
00:02:20,686 --> 00:02:23,206
and they're tidy and they're pretty clean,

40
00:02:23,366 --> 00:02:24,766
but there's a unique human twist.

41
00:02:25,106 --> 00:02:28,046
People generally like to infuse their thoughts and opinions

42
00:02:28,046 --> 00:02:29,726
into these things, sometimes by design,

43
00:02:29,866 --> 00:02:32,746
so you end up with things I call watermelon projects.

44
00:02:32,926 --> 00:02:35,566
They're green on the outside and red on the inside.

45
00:02:36,526 --> 00:02:39,486
Then you have, on the bottom, you have the really noisy stuff,

46
00:02:39,646 --> 00:02:41,726
which has a surprising amount of signal,

47
00:02:41,906 --> 00:02:43,966
but it's hard to uncover.

48
00:02:44,326 --> 00:02:48,106
You have meeting notes, and you have slack and coat changes,

49
00:02:48,366 --> 00:02:50,746
and even your session traces with your favorite harness.

50
00:02:51,226 --> 00:02:53,366
And the model is where they meet.

51
00:02:53,546 --> 00:02:55,546
The evidence is synthesized at that level.

52
00:02:56,786 --> 00:02:59,486
Now, the evidence piece sometimes is problematic

53
00:02:59,486 --> 00:03:01,686
because it's either partial or wrong,

54
00:03:01,866 --> 00:03:03,506
but it turns out that in robotics,

55
00:03:03,706 --> 00:03:05,426
you actually have the solution for this,

56
00:03:05,626 --> 00:03:07,106
and it's called a state estimator.

57
00:03:07,646 --> 00:03:11,366
What you do is you have some sort of belief

58
00:03:11,366 --> 00:03:12,686
about something you care about,

59
00:03:12,686 --> 00:03:17,006
and the longer it's been since you've seen or observed evidence

60
00:03:17,006 --> 00:03:20,526
that supports it, the less you trust it, right?

61
00:03:20,746 --> 00:03:21,886
The fuzzier it gets.

62
00:03:22,366 --> 00:03:26,606
Until you have a deck or some thread or some meeting note,

63
00:03:26,826 --> 00:03:30,486
that sort of collapses that and you now have new evidence, right?

64
00:03:30,746 --> 00:03:32,666
And now you're more certain of it.

65
00:03:32,966 --> 00:03:34,666
And therefore you produce the update.

66
00:03:35,226 --> 00:03:38,426
And the things that I care about when producing an update

67
00:03:38,426 --> 00:03:41,066
or that the agent should care about, there's three things.

68
00:03:41,066 --> 00:03:45,546
There's how long has it been since some signal has been received.

69
00:03:45,966 --> 00:03:50,186
There's where did that signal come from, and is it still trustworthy?

70
00:03:50,546 --> 00:03:52,886
And then does it broadly agree with other sensors?

71
00:03:53,306 --> 00:03:57,286
There's a lot of parallels here with self-driving, for example, that I'll come back to.

72
00:03:58,286 --> 00:03:59,526
So let's start at the top.

73
00:03:59,706 --> 00:04:06,206
A very simple example is something like a slide that is produced already by someone in BizOps,

74
00:04:06,326 --> 00:04:08,166
and they produce a monthly status deck.

75
00:04:08,686 --> 00:04:11,246
and the question is, how do you get data out of this slide?

76
00:04:12,346 --> 00:04:13,826
Many of you, I'm sure, have tried that.

77
00:04:14,046 --> 00:04:18,746
It's deceptive, actually, because let's say you try to parse the file.

78
00:04:19,166 --> 00:04:21,726
You unpack the PowerPoint file and you get an XML

79
00:04:22,166 --> 00:04:25,746
and all of a sudden you get into this really interesting conceptual question.

80
00:04:25,906 --> 00:04:28,326
What is green? What is the hex code for green?

81
00:04:30,126 --> 00:04:32,546
And things break down when it comes to this type of script.

82
00:04:33,306 --> 00:04:36,586
Then the solution is to say, well, I'm just going to take a screenshot of it

83
00:04:36,586 --> 00:04:37,646
and throw it at a vision model.

84
00:04:38,166 --> 00:04:42,286
But at that point, even if it's pretty high res, you're dealing with someone who copy-pasted

85
00:04:42,286 --> 00:04:47,446
a screenshot back into a presentation, and essentially you have Claude trying to squint

86
00:04:47,446 --> 00:04:50,166
and trying to figure out if it's a 31 or an 81.

87
00:04:50,806 --> 00:04:56,926
And so the theme of the talk really is about elegance comes from just synthesizing things

88
00:04:56,926 --> 00:04:58,046
and adding more information.

89
00:04:58,306 --> 00:05:03,726
And in this example, it's very basic, but you get the true picture by taking both the

90
00:05:03,726 --> 00:05:06,206
XML and the output of a vision model.

91
00:05:06,826 --> 00:05:08,526
Now, that's pretty basic, right?

92
00:05:08,866 --> 00:05:13,206
But the issue is that you want to establish some continuity over time, right?

93
00:05:13,286 --> 00:05:18,406
Like this collateral that your company accrues is very much sort of always building up.

94
00:05:18,526 --> 00:05:24,926
And so in this particular example, you have a project that just spontaneously changed names back in March.

95
00:05:25,226 --> 00:05:30,506
And so, you know, you have to establish that sort of longitudinal traceable audit layer

96
00:05:30,506 --> 00:05:36,046
as you create this shared context layer about the sort of institutional memory of your company.

97
00:05:37,446 --> 00:05:39,166
And the irony, there's two things.

98
00:05:39,266 --> 00:05:42,026
The irony here is that even if you do this really well,

99
00:05:42,306 --> 00:05:44,346
you're still missing half the picture, right?

100
00:05:44,426 --> 00:05:47,326
You're looking at a deck and your Slack says otherwise

101
00:05:47,326 --> 00:05:49,146
because that was five days ago.

102
00:05:49,226 --> 00:05:50,586
And now you have someone on Slack saying,

103
00:05:50,686 --> 00:05:52,946
hey, by the way, I think we're on track for this.

104
00:05:53,066 --> 00:05:55,166
But then someone replies with a picture of a dashboard

105
00:05:55,166 --> 00:05:56,706
and it's no longer the case.

106
00:05:56,826 --> 00:05:58,186
So things just start breaking down.

107
00:05:58,826 --> 00:06:01,106
And so we need to do a better job

108
00:06:01,106 --> 00:06:04,066
of integrating all these things, right?

109
00:06:04,066 --> 00:06:06,226
Your company has no model of itself.

110
00:06:07,146 --> 00:06:09,826
So I'm proposing just a brief plan here.

111
00:06:10,126 --> 00:06:13,266
The idea is you take the three things that I described earlier

112
00:06:13,266 --> 00:06:14,646
about the state estimator.

113
00:06:14,726 --> 00:06:18,866
You have the update, age, and observe stages,

114
00:06:18,906 --> 00:06:19,926
and then you have grade.

115
00:06:20,346 --> 00:06:23,766
The grade stage is basically comparing what you observed

116
00:06:23,766 --> 00:06:24,846
versus what you expected.

117
00:06:25,386 --> 00:06:29,026
And then the correct stage is basically a way for people

118
00:06:29,026 --> 00:06:31,066
to allow the system to get new feedback

119
00:06:31,066 --> 00:06:33,086
and steer the behavior of the system.

120
00:06:35,446 --> 00:06:41,126
At Citadel Securities, the system that we've built has agents at every part of this particular

121
00:06:41,126 --> 00:06:41,666
loop.

122
00:06:42,106 --> 00:06:46,466
But the key thing is that the model is proposing these changes, and then you have code and

123
00:06:46,466 --> 00:06:48,646
people deciding how the records are updated.

124
00:06:49,786 --> 00:06:52,466
And of course you're probably asking yourself, why not just RAG?

125
00:06:53,186 --> 00:06:54,566
Probably RAG would do the trick.

126
00:06:54,786 --> 00:06:56,526
And in fact, it really doesn't.

127
00:06:57,186 --> 00:07:00,686
In this example, I asked, hey, what's the status of this project?

128
00:07:00,806 --> 00:07:01,366
Is it on track?

129
00:07:01,366 --> 00:07:07,786
And as of March 16th, some model would happily pick up a couple of Slack messages and some

130
00:07:07,786 --> 00:07:09,286
slides and say, yep, it's green.

131
00:07:09,486 --> 00:07:11,546
But it's missing a lot of the context, right?

132
00:07:11,626 --> 00:07:12,266
It's not enough.

133
00:07:13,146 --> 00:07:16,166
So here's how we start building this world model.

134
00:07:16,366 --> 00:07:20,766
The first stage is about updating information consistently and robustly.

135
00:07:21,126 --> 00:07:26,986
In this simple example, where I'm taking some slides and some decks, we are basically just

136
00:07:26,986 --> 00:07:28,406
looking at every month.

137
00:07:28,406 --> 00:07:32,906
A new month comes around, we take last month's record and this month's slide,

138
00:07:33,066 --> 00:07:36,546
and you try to construct a sort of longitudinal record.

139
00:07:37,246 --> 00:07:39,706
And we've been doing this now for three years.

140
00:07:39,866 --> 00:07:43,266
We have a pretty large historical database of these records

141
00:07:43,266 --> 00:07:48,666
that now allow us to construct this robust so-called institutional or enterprise world model.

142
00:07:49,866 --> 00:07:51,186
You start with the basics.

143
00:07:52,386 --> 00:07:56,146
Sometimes these models struggle with the most basic things, but they're normal.

144
00:07:56,146 --> 00:08:01,066
The actual name of a particular project changed spontaneously,

145
00:08:01,106 --> 00:08:04,326
but by actually looking at that historical set of records,

146
00:08:04,446 --> 00:08:08,186
it can see and infer that it actually refers to Project North Star

147
00:08:08,186 --> 00:08:13,046
by virtue of who was involved, by virtue of some continuity in the types of goals,

148
00:08:13,386 --> 00:08:14,366
those kinds of things.

149
00:08:15,446 --> 00:08:19,146
But more interestingly, what you want to do is you have the agentic system

150
00:08:19,146 --> 00:08:24,446
try to discriminate between what's being reported and actual evidence for what it's saying.

151
00:08:24,446 --> 00:08:38,486
So you look at some status deck, for example, and here it's identifying that while it is green on the deck, in fact it should be highlighted as a yellow and should be flagged as such because there's a comment someone left talking about how there's a dependency still outstanding.

152
00:08:40,006 --> 00:08:43,766
The main thing here is that you don't touch previous months, right?

153
00:08:43,846 --> 00:08:47,426
You just sort of heal in the sort of feed-forward mechanism.

154
00:08:47,666 --> 00:08:52,286
And what that allows you to do is to actually do some robust evals down the line, right?

155
00:08:53,666 --> 00:08:56,266
So here's what the actual update step writes.

156
00:08:56,566 --> 00:08:57,286
Very simple.

157
00:08:57,526 --> 00:08:58,666
This is what it was reported.

158
00:08:59,026 --> 00:09:02,846
This is where the evidence points, and this is why.

159
00:09:04,606 --> 00:09:08,106
And then you fold in the second component, age.

160
00:09:08,386 --> 00:09:10,786
The reality is every fact has a half-life.

161
00:09:10,986 --> 00:09:13,766
We've all seen, you know, you go back at some slides,

162
00:09:13,966 --> 00:09:18,566
and you see that the go-live date for a particular project keeps slipping.

163
00:09:18,566 --> 00:09:22,446
It was Q2 in January, then it was Q3, now it's October.

164
00:09:22,846 --> 00:09:24,946
And you actually observe this.

165
00:09:25,186 --> 00:09:26,726
By actually collecting all this data,

166
00:09:26,806 --> 00:09:29,826
you can have some pretty interesting beginnings of analytical insights.

167
00:09:30,466 --> 00:09:34,106
For example, in the last three years, you look at these delivery dates,

168
00:09:34,126 --> 00:09:37,346
and on average, half the delivery dates change within two months,

169
00:09:37,426 --> 00:09:42,066
and 90% of those change into a date that is after what was originally proposed.

170
00:09:42,706 --> 00:09:44,346
So far, so good. Pretty basic.

171
00:09:44,346 --> 00:09:52,566
But what this forces you to do is then introduce this notion of a date that is associated to a particular claim

172
00:09:52,566 --> 00:10:02,366
so that the agent can introduce some sort of recency bias when you ask about the status of a particular project, person, or the corporate intent, let's say.

173
00:10:04,506 --> 00:10:06,066
Then the observe section.

174
00:10:06,226 --> 00:10:12,226
The observe section here, I'm going to pretty quickly gloss over it, but just kind of give you something very tactical.

175
00:10:12,226 --> 00:10:16,546
In the beginning, I was pretty seductive to just basically say, well, let's just YOLO this.

176
00:10:16,726 --> 00:10:22,906
Let's have Claude or some other model take a look at the entire corpus of things, and let's see what it does.

177
00:10:23,146 --> 00:10:29,446
And as it turns out, it's pretty clear that a lot of information is lost.

178
00:10:29,626 --> 00:10:31,786
It loses some attention to detail.

179
00:10:32,026 --> 00:10:37,026
And so what ultimately we had to do was engineer a simple agentic workflow

180
00:10:37,026 --> 00:10:41,446
where you have a series of specialists break down the collateral of these assets

181
00:10:41,946 --> 00:10:46,586
look for specific things, be reviewed by other agents, and so on.

182
00:10:46,946 --> 00:10:49,146
And what you see here on the right-hand side

183
00:10:49,146 --> 00:10:52,426
is that actually the amount of insightful updates

184
00:10:52,426 --> 00:10:55,806
to a particular project more than doubled after doing this.

185
00:10:56,046 --> 00:10:58,506
So I guess the takeaway here is be intentional

186
00:10:58,506 --> 00:11:00,326
about how you build these workflows

187
00:11:00,326 --> 00:11:03,546
and don't just assume that a YOLO method is going to work out.

188
00:11:04,206 --> 00:11:06,586
But now you get, I think, to the really interesting stuff.

189
00:11:06,766 --> 00:11:09,126
A world model should tell you what happens next.

190
00:11:09,426 --> 00:11:11,186
What project is about to slip,

191
00:11:11,186 --> 00:11:13,566
or which green turns red the next month.

192
00:11:13,766 --> 00:11:16,626
And it turns out that you can actually do this, right?

193
00:11:16,926 --> 00:11:19,946
Because you can look back and establish a snapshot

194
00:11:19,946 --> 00:11:21,546
as of some sort of past date

195
00:11:21,546 --> 00:11:23,186
and essentially run a backtest.

196
00:11:23,446 --> 00:11:25,926
So for example, if we take a snapshot as of February,

197
00:11:25,966 --> 00:11:29,086
we can look at the future labels that came after that

198
00:11:29,086 --> 00:11:30,026
and see and

199
00:11:30,106 --> 00:11:35,226
compare what the agent said was going to slip or flagged versus what actually happened.

200
00:11:35,846 --> 00:11:46,106
And you can see that as the agent analyzes all this information, that for statuses that were green but flagged, they were twice as likely to actually turn yellow or red.

201
00:11:46,586 --> 00:11:48,786
And so, again, this is encouraging

202
00:11:48,786 --> 00:11:50,866
because we're starting to see some beginning

203
00:11:50,866 --> 00:11:52,026
of the inability to say,

204
00:11:52,126 --> 00:11:55,026
well, it's not predictive of exactly what's going to go wrong,

205
00:11:55,046 --> 00:11:57,086
but at least it should tell you what to look at.

206
00:11:58,966 --> 00:12:01,486
So, you know, you have 18 different cases

207
00:12:01,486 --> 00:12:04,466
where you have a verdict from an agent that's flagged,

208
00:12:04,526 --> 00:12:05,946
but what about those where you don't?

209
00:12:06,046 --> 00:12:07,946
And this is where some product intuition comes in.

210
00:12:08,286 --> 00:12:10,006
First of all, how would you incorporate

211
00:12:10,006 --> 00:12:11,706
a very elegant sort of loop

212
00:12:11,706 --> 00:12:15,886
where you include an actual user, a stakeholder, a human?

213
00:12:16,546 --> 00:12:18,926
What do you ask them and where do you meet them?

214
00:12:19,206 --> 00:12:24,946
I put a nontrivial amount of work into finding out how best to actually capture this information.

215
00:12:25,226 --> 00:12:30,126
It's meeting the users where they are, whether it's the app they're using, Slack, email,

216
00:12:30,486 --> 00:12:30,846
whatever.

217
00:12:31,466 --> 00:12:34,946
The closer you can get to them and the simpler you can make that feedback collection mechanism,

218
00:12:35,246 --> 00:12:38,426
the more scalable and robust the process is going to be.

219
00:12:39,066 --> 00:12:40,806
It also matters what you ask.

220
00:12:40,806 --> 00:12:46,746
You want to understand whether you're asking a fact which only fixes one output,

221
00:12:46,746 --> 00:12:48,746
or if you're asking about a rule.

222
00:12:48,966 --> 00:12:50,726
And once an agent understands that pattern,

223
00:12:50,906 --> 00:12:53,326
it does a much better job of fixing every output it covers,

224
00:12:53,526 --> 00:12:55,926
both now and into the future.

225
00:12:56,866 --> 00:13:02,246
So we have this notion now of our record containing not just an element

226
00:13:02,246 --> 00:13:04,246
of what was inferred by the agent,

227
00:13:04,406 --> 00:13:08,586
but more importantly confirmed after someone actually provided that signal.

228
00:13:09,806 --> 00:13:11,966
Okay, so now bottom up, right?

229
00:13:12,046 --> 00:13:15,566
You have meetings with really strong signal, what people say out loud.

230
00:13:15,766 --> 00:13:17,166
You have your Slack.

231
00:13:17,406 --> 00:13:18,306
Obviously, it's constant.

232
00:13:18,326 --> 00:13:19,046
It's noisy.

233
00:13:19,346 --> 00:13:20,886
It's very close to the truth.

234
00:13:21,166 --> 00:13:23,466
Code changes describe what shipped.

235
00:13:23,826 --> 00:13:26,786
And then, of course, the Asian sessions, which is how the sausage was made.

236
00:13:26,986 --> 00:13:30,926
And I think I encourage you all to explore all of these disparate data sources to kind

237
00:13:30,926 --> 00:13:32,406
of build such a model up.

238
00:13:32,986 --> 00:13:35,106
With Slack, it's relatively simple, right?

239
00:13:35,106 --> 00:13:37,626
like you're just looking at every thread and trying to understand,

240
00:13:37,826 --> 00:13:41,006
based on attachments and participants and content,

241
00:13:41,326 --> 00:13:45,006
the claims that you want the agent to record or update.

242
00:13:45,326 --> 00:13:47,966
When was this? Who was responsible?

243
00:13:48,246 --> 00:13:50,926
Or who does this relate to? What project?

244
00:13:51,246 --> 00:13:54,646
What is the sort of link to the evidence? And so on.

245
00:13:56,166 --> 00:13:58,746
With PRs, of course, it shows you what changed.

246
00:13:58,826 --> 00:14:02,126
And it's a really nice way to see how a project is evolving.

247
00:14:02,126 --> 00:14:09,526
But when it comes to the actual subtleties of how projects are run and deployed, it's

248
00:14:09,526 --> 00:14:14,406
really interesting to look at the actual traces for users and engineers who opt into that.

249
00:14:14,886 --> 00:14:23,006
You understand how the ask evolved over time, test failures, or particular decisions from

250
00:14:23,006 --> 00:14:24,946
the developer, those kinds of things.

251
00:14:25,186 --> 00:14:28,886
Ultimately, these culminate in a pull request, but the synthesis of these things, I think,

252
00:14:28,926 --> 00:14:29,606
makes a big difference.

253
00:14:30,806 --> 00:14:36,106
The other gotcha I just want to share is to just be mindful of how you think about the

254
00:14:36,106 --> 00:14:38,746
validity of signal coming from different so-called sensors.

255
00:14:39,046 --> 00:14:46,046
If you're driving down a highway, for example, and your speedometer just disappears, it doesn't

256
00:14:46,046 --> 00:14:48,286
mean you're now going at zero miles an hour.

257
00:14:48,686 --> 00:14:50,266
It's just like a faulty signal.

258
00:14:50,546 --> 00:14:54,646
So for example, here we ran into a couple of issues sometimes when the bottoms-up approach,

259
00:14:54,646 --> 00:14:57,166
you had some semblance of a claim saying,

260
00:14:57,306 --> 00:15:01,006
hey, this particular channel, this thing is going quiet, what's going on?

261
00:15:01,206 --> 00:15:03,706
But in reality, we were treating it as a zero,

262
00:15:03,786 --> 00:15:04,786
and really it shouldn't have been.

263
00:15:04,966 --> 00:15:08,066
And so we sort of iterated over time to understand

264
00:15:08,626 --> 00:15:12,926
how to incorporate when we last tried to get information from that sensor,

265
00:15:13,386 --> 00:15:16,706
when it was last okay, and what do you do when these diverge?

266
00:15:18,186 --> 00:15:21,366
So now, if you look at the actual record that we constructed,

267
00:15:22,046 --> 00:15:24,386
the agents are going to be much more effective

268
00:15:24,386 --> 00:15:26,246
in understanding the true state of the world.

269
00:15:27,826 --> 00:15:28,766
It's not green.

270
00:15:29,526 --> 00:15:32,606
It's the go-live, slip-twice, the evidence says yellow,

271
00:15:32,806 --> 00:15:34,326
people are talking about failing tests.

272
00:15:34,526 --> 00:15:35,606
It's going to be tricky.

273
00:15:36,986 --> 00:15:38,326
So what do we use this for?

274
00:15:38,646 --> 00:15:40,726
We use it to steer leadership conversations.

275
00:15:40,726 --> 00:15:43,486
Instead of circulating a deck to everyone

276
00:15:43,486 --> 00:15:45,486
and forcing everyone to just go through some slides

277
00:15:45,486 --> 00:15:46,446
everyone glosses over,

278
00:15:46,686 --> 00:15:49,166
you start with the stuff that actually matters.

279
00:15:49,166 --> 00:15:51,566
You start where the deck and the evidence disagree.

280
00:15:52,646 --> 00:15:55,806
You also produce a weekly read for managers and leaders

281
00:15:55,806 --> 00:15:57,386
on what stuck and what went quiet.

282
00:15:57,726 --> 00:15:59,326
But I think also, most importantly,

283
00:15:59,526 --> 00:16:01,526
you have the sort of beginning of a context layer

284
00:16:01,526 --> 00:16:03,706
that every agent can rely on.

285
00:16:04,186 --> 00:16:06,546
And we're using this pattern and constructing it

286
00:16:06,546 --> 00:16:09,086
so that it's extendable to other different use cases.

287
00:16:09,306 --> 00:16:11,126
Right in our client franchise, for example,

288
00:16:11,346 --> 00:16:13,506
we're curious about how we can build similar models

289
00:16:13,506 --> 00:16:14,606
to help serve our clients.

290
00:16:15,006 --> 00:16:17,006
We know we have some client,

291
00:16:17,006 --> 00:16:19,546
and we want to understand who holds what,

292
00:16:19,846 --> 00:16:21,286
what are they interested in,

293
00:16:21,506 --> 00:16:24,046
how can we anticipate their needs as best as possible

294
00:16:24,046 --> 00:16:25,886
by folding in information

295
00:16:25,886 --> 00:16:29,086
from all sorts of disparate internal and external sources.

296
00:16:30,046 --> 00:16:31,986
So a couple of things that are difficult to do,

297
00:16:32,326 --> 00:16:35,786
just to kind of, maybe we can chat outside about your solutions,

298
00:16:35,846 --> 00:16:36,866
I'd be very curious to hear,

299
00:16:37,426 --> 00:16:39,866
are when you have a lot of, you know, for example,

300
00:16:40,046 --> 00:16:42,366
one client with four records and four different systems

301
00:16:42,366 --> 00:16:44,006
with no canonical identifier,

302
00:16:44,006 --> 00:16:48,006
how do you go about mapping those in a very elegant way?

303
00:16:48,666 --> 00:16:51,326
That's one thing that I don't think we have a clear answer to.

304
00:16:51,946 --> 00:16:58,046
The second thing is, how can we be most effective at capturing signal from stakeholders and users?

305
00:16:58,806 --> 00:17:03,046
Where the model is sure, and the stakes are pretty low, you should just apply it automatically.

306
00:17:03,826 --> 00:17:08,426
When those stakes are low, but the model isn't sure, just wait for more evidence.

307
00:17:08,746 --> 00:17:13,606
But if the stakes are high, it really matters who you ask, when you ask it,

308
00:17:13,606 --> 00:17:15,626
and kind of the whole mechanism there.

309
00:17:16,626 --> 00:17:18,586
So here's the loop again.

310
00:17:19,206 --> 00:17:22,866
Update, age, observe, grade, and correct.

311
00:17:23,286 --> 00:17:25,346
And three things I want you to take home.

312
00:17:25,566 --> 00:17:28,646
The model that your company has,

313
00:17:29,366 --> 00:17:32,046
you should think about it as a state estimator, right?

314
00:17:32,486 --> 00:17:35,866
Don't assume that just like indexing everything

315
00:17:35,866 --> 00:17:37,786
and cataloging everything is enough

316
00:17:37,786 --> 00:17:41,806
because by definition, these things are extremely nuanced

317
00:17:41,806 --> 00:17:44,966
and this intrinsic information that lives almost between the lines

318
00:17:44,966 --> 00:17:47,786
is hard for even the best state-of-the-art models to capture.

319
00:17:48,486 --> 00:17:50,806
Grade it against what happens next, right?

320
00:17:50,926 --> 00:17:53,326
So be intentional and think through how you can integrate

321
00:17:53,326 --> 00:17:55,586
and implement evals from the get-go.

322
00:17:55,986 --> 00:17:59,386
And think about an elegant solution that allows your models

323
00:17:59,386 --> 00:18:04,726
and your agents and your stakeholders to kind of play nice together.

324
00:18:06,026 --> 00:18:08,686
Your company can have a model of itself.

325
00:18:09,086 --> 00:18:10,966
I encourage you to start collecting now

326
00:18:10,966 --> 00:18:12,106
if you're not collecting already.

327
00:18:12,666 --> 00:18:14,586
And certainly if you have, if you're at a bigger company

328
00:18:14,586 --> 00:18:17,846
and like me you had the fortune or misfortune

329
00:18:17,846 --> 00:18:19,186
of dealing with thousands of decks,

330
00:18:19,726 --> 00:18:21,586
definitely something you should start thinking about

331
00:18:21,586 --> 00:18:24,106
because I've definitely seen material improvements

332
00:18:24,106 --> 00:18:26,686
in how these agents are able to reconstruct

333
00:18:26,686 --> 00:18:29,106
that sort of corporate memory

334
00:18:29,106 --> 00:18:32,186
and apply it in all sorts of daily tasks

335
00:18:32,686 --> 00:18:35,146
without sort of being all confused about things.

336
00:18:36,166 --> 00:18:37,766
Okay, thank you very much.

337
00:18:37,846 --> 00:18:39,866
I'm João and I'll be outside for any questions if you want.

338
00:18:40,246 --> 00:18:40,666
Cheers.

339
00:18:40,966 --> 00:18:41,666
Thank you.