1
06:57:39,940 --> 06:57:44,420
Hello and welcome to our AIA Engineer San Francisco talk.

2
06:57:44,580 --> 06:57:45,280
My name is Stephen Chin.

3
06:57:45,280 --> 06:57:49,600
I'm joined by Cassandra Chen, who's the creator of Tracetune framework.

4
06:57:51,920 --> 06:57:59,340
Hi, I'm Cassandra Chen. I built Tracetune because I wanted to improve PRs. I see a lot of developers

5
06:57:59,340 --> 06:58:05,000
doing great things. The speed of innovation is as fast as it has ever been with AI technologies,

6
06:58:05,680 --> 06:58:11,140
but a lot of pull requests are getting rejected. And it's not because the code is bad, it's because

7
06:58:11,140 --> 06:58:14,660
because the PRs often contain incorrect information

8
06:58:14,660 --> 06:58:16,180
or are written poorly.

9
06:58:17,520 --> 06:58:20,040
So hopefully with Tracetune, we can improve that.

10
06:58:20,280 --> 06:58:22,880
It also has the option to add a fun comic

11
06:58:22,880 --> 06:58:25,920
to help explain your pull requests in a funny way.

12
06:58:28,660 --> 06:58:29,880
What we're gonna do today

13
06:58:29,880 --> 06:58:31,340
is we're gonna go through a little bit

14
06:58:31,340 --> 06:58:34,460
about kind of the common issues of PRs,

15
06:58:34,480 --> 06:58:35,500
how to improve the quality,

16
06:58:35,740 --> 06:58:37,560
how to improve token efficiency on this,

17
06:58:37,560 --> 06:58:43,280
and hopefully help everyone to be a better AI engineer for your organization,

18
06:58:43,280 --> 06:58:51,660
like our main character, Alex, who all he wants in life is to get an AI engineer hoodie.

19
06:58:53,060 --> 06:58:54,400
That's a really neat hoodie.

20
06:58:54,820 --> 06:58:59,080
I have a lot of AI engineer t-shirts, but I don't have the hoodie yet.

21
06:59:00,940 --> 06:59:03,640
Get Swix to make this AI engineer hoodie, everyone.

22
06:59:06,660 --> 06:59:17,780
And this is enough motivation that hopefully he's going to do an amazing talk at the conference about his use case study on what he's doing in his company to improve the quality of PRs.

23
06:59:20,320 --> 06:59:25,080
Here is a picture of the token factory where Alex works.

24
06:59:25,460 --> 06:59:30,300
So Alex works together with a bunch of agents.

25
06:59:30,920 --> 06:59:33,480
He has, of course, finance looking over his shoulder

26
06:59:33,480 --> 06:59:35,300
Because we all spend too much tokens

27
06:59:35,300 --> 06:59:37,720
And they want to make sure we're keeping up with costs

28
06:59:38,860 --> 06:59:40,940
His buddy there, Pip, he's walking with

29
06:59:40,940 --> 06:59:42,380
Carries all the evidence

30
06:59:42,380 --> 06:59:44,600
And looks into the archives

31
06:59:44,600 --> 06:59:45,640
And what's actually happening

32
06:59:46,200 --> 06:59:48,200
And then Mira does all the reviews

33
06:59:48,200 --> 06:59:50,220
So she's keeping track of things

34
06:59:51,280 --> 06:59:53,520
It looks like Pip left his station empty

35
06:59:58,640 --> 07:00:00,000
Part of the factory

36
07:00:02,500 --> 07:00:05,140
Until he sees what Alex produced for a PR.

37
07:00:05,380 --> 07:00:05,560
Oh, no.

38
07:00:07,300 --> 07:00:09,340
That's a very long ledger.

39
07:00:10,120 --> 07:00:11,220
A lot to review.

40
07:00:14,080 --> 07:00:15,520
So it's hard.

41
07:00:16,180 --> 07:00:19,500
And Mira's going through everything for the fine-tuned comb.

42
07:00:19,660 --> 07:00:21,040
She actually is red.

43
07:00:22,640 --> 07:00:24,480
Found some issues with the PR.

44
07:00:26,080 --> 07:00:47,060
And this is a real PR that I think is actually one of your Tracetune PRs where it needs to have exact preview links for the URLs, command attribution, make sure the earlier work is qualified as well, and explain all of this in detail in the PR record to be accepted by Mira's very critical review.

45
07:00:47,060 --> 07:00:55,640
This was interesting because as I was building Tracetune, I was actually, like, as I was building it, I was using Tracetune to test it.

46
07:00:56,000 --> 07:00:58,400
So I would have Tracetune build the PRs.

47
07:00:58,520 --> 07:01:01,600
And as I improved the PRs, it would, like, get better.

48
07:01:03,060 --> 07:01:08,580
You got amazing comics along the way, too, just to have fun while you were building stuff.

49
07:01:08,840 --> 07:01:08,980
Yeah.

50
07:01:09,980 --> 07:01:10,460
Effective.

51
07:01:15,240 --> 07:01:23,600
So you can see that the repeated PRs, which Alex is doing to try to get good quality results, are not working out.

52
07:01:23,740 --> 07:01:26,720
Well, it's working out for his token billionaire card.

53
07:01:26,880 --> 07:01:28,740
Do you know anything about the token billionaire program?

54
07:01:29,140 --> 07:01:31,780
Yeah, so actually I am a token billionaire.

55
07:01:32,280 --> 07:01:36,720
I spent lots and lots of tokens to build Tracetune.

56
07:01:39,400 --> 07:01:42,760
That's kind of unfortunate. Was that all my money?

57
07:01:44,160 --> 07:01:45,340
Don't sweat the details.

58
07:01:47,460 --> 07:01:57,640
Okay. So I think there's a lot of good uses for tokens and for spending more AI work. Maybe doing repeated PRs is not where you want to spend your token.

59
07:01:58,780 --> 07:02:01,660
And definitely the finance department is not happy with this.

60
07:02:01,760 --> 07:02:14,940
So he wants a result, and he wants to make sure that the PRs are accurate, are working, and they don't have to keep getting repeated again and again by reviewing the logs and reconstructing the evidence repeatedly.

61
07:02:15,700 --> 07:02:25,180
And it's interesting because normally when you're just coding on your agent, you're in your session and you ask it to write a PR.

62
07:02:25,180 --> 07:02:34,220
it just reads your session log to try to figure out what it should have done using its inflated

63
07:02:34,220 --> 07:02:40,880
context window. It actually often like as it's churning it compresses its context window

64
07:02:40,880 --> 07:02:47,140
so it gets more and more inaccurate as it goes along. So instead what we're doing here is with

65
07:02:47,140 --> 07:02:54,240
Tracetune we're having it read through different sessions on your computer to try to really figure

66
07:02:54,240 --> 07:03:00,900
out what you built and what happened. Yeah, and I think like that concept of

67
07:03:01,640 --> 07:03:04,860
looking through the audit trails, looking through like what the agents have actually done

68
07:03:05,620 --> 07:03:09,880
is a lot of what you were doing with the Showtale project you were doing for education.

69
07:03:11,120 --> 07:03:17,980
Yeah, I actually started this project with this Showtale project because there was this like

70
07:03:17,980 --> 07:03:24,200
academic research study I was supporting, but basically the issue they hit is students were

71
07:03:24,200 --> 07:03:29,720
creating programs, but there was no way to trace back the actual work and effort they did.

72
07:03:29,980 --> 07:03:36,940
Because with AI coding, what you submit is the final work, the program, but most of the work

73
07:03:36,940 --> 07:03:42,720
is the prompting and everything that came before, and that doesn't get submitted with the program.

74
07:03:43,400 --> 07:03:52,200
And I use the technology and theory behind that to be like, well, I can also improve pull requests with all this data collection.

75
07:03:54,400 --> 07:04:11,760
Cool. And I think that improvement in pull requests is something that benefits everybody because now you can find the evidence, like logging the change directory or the CWD before the CD runs, explaining why maybe other test data is leaking into other repos.

76
07:04:11,760 --> 07:04:16,200
and explaining what's happening behind the scenes on the PR

77
07:04:16,200 --> 07:04:19,540
rather than what most AI PRs read like,

78
07:04:19,700 --> 07:04:21,900
which is a line-by-line explanation of the diffs

79
07:04:21,900 --> 07:04:24,320
since that's what they build the PRs off of.

80
07:04:28,880 --> 07:04:32,260
But incorrect facts can still pollute the information,

81
07:04:32,880 --> 07:04:35,880
can cause a problem with conflicting PRs,

82
07:04:35,880 --> 07:04:39,500
and then still give you back inaccurate results.

83
07:04:40,600 --> 07:04:45,620
so maybe we have a solution for this what do you what do you think you have a solution cassandra

84
07:04:45,620 --> 07:04:53,160
yeah everyone should use trace tune because of the comics so because the comics not because

85
07:04:53,160 --> 07:04:59,720
it gives you better prs right well i started with the comics and i kind of went backwards and like

86
07:04:59,720 --> 07:05:08,160
oh i need to actually make the prs better well hopefully better prs better comics and

87
07:05:08,160 --> 07:05:12,080
and maybe better token savings by the end of this.

88
07:05:12,600 --> 07:05:12,680
Maybe.

89
07:05:13,280 --> 07:05:13,940
It should be.

90
07:05:16,420 --> 07:05:18,460
So TraceDude is an open source project.

91
07:05:18,760 --> 07:05:23,100
It lets you improve the quality of your PRs by pulling in agent context,

92
07:05:23,280 --> 07:05:26,520
by pulling in information about the code,

93
07:05:26,800 --> 07:05:29,260
and then having an independent writer, independent reviewer,

94
07:05:30,020 --> 07:05:35,340
build better quality PRs, plus optionally giving you an amazing comic,

95
07:05:35,340 --> 07:05:38,700
which, what do you think the best feature is here, Cassandra?

96
07:05:39,480 --> 07:05:41,640
I think it's obviously the comics.

97
07:05:42,280 --> 07:05:47,380
This is one of the earlier comics from when I was using Trace Tune to build it.

98
07:05:50,440 --> 07:05:51,100
That's good.

99
07:05:51,180 --> 07:05:54,060
And then you made sure that there's new characters every episode,

100
07:05:54,220 --> 07:05:55,640
none of this repeating characters.

101
07:05:56,280 --> 07:05:58,820
Yeah, that's what this PR was about.

102
07:06:01,800 --> 07:06:05,780
Okay, and then this is a description of the original trace student flow.

103
07:06:06,040 --> 07:06:09,280
So it finds all the traces for a particular branch.

104
07:06:09,400 --> 07:06:13,120
It researches them, reviews the description, adds more context,

105
07:06:13,180 --> 07:06:19,220
and kind of goes in a loop until it gets all of the evidence it needs to make a quality PR.

106
07:06:19,620 --> 07:06:25,340
And subjectively but also through PRBench, which is a testing framework,

107
07:06:26,640 --> 07:06:29,060
gives you better quality, better research,

108
07:06:29,340 --> 07:06:33,080
lower rate of inaccuracies in the PRs.

109
07:06:33,260 --> 07:06:36,400
I think at this point, the PRs were already better

110
07:06:36,400 --> 07:06:37,920
than if you don't use Tracedoon.

111
07:06:38,140 --> 07:06:41,540
But I found that it still makes some mistakes occasionally.

112
07:06:42,540 --> 07:06:45,640
You can see at the end that it tries to revise with more context.

113
07:06:46,000 --> 07:06:47,940
I think that's where the mistakes come in.

114
07:06:50,620 --> 07:06:52,840
Okay, well, Alex thinks it's good enough.

115
07:06:52,980 --> 07:06:54,900
He's going to try to solve the token crisis.

116
07:06:54,900 --> 07:06:58,100
the factory with Trace2 and see if this will solve all of his problems.

117
07:06:58,980 --> 07:07:06,180
Polish checked. Alex looks very, very happy. But Mira doesn't look that happy.

118
07:07:06,500 --> 07:07:10,760
It looks like we still hit a mistake here, like I was saying earlier.

119
07:07:11,160 --> 07:07:15,140
So the original PR claim was no later passing run is recorded.

120
07:07:16,340 --> 07:07:24,020
And the problem here is in the actual Trace, it shows it running a button test

121
07:07:24,020 --> 07:07:25,980
at the same time as the

122
07:07:25,980 --> 07:07:27,660
bun x tsc

123
07:07:28,400 --> 07:07:30,080
and you can't run those two

124
07:07:30,080 --> 07:07:31,700
at the same time they have to be

125
07:07:31,700 --> 07:07:34,300
run sequentially for the test to pass

126
07:07:34,300 --> 07:07:36,000
yeah exactly

127
07:07:36,000 --> 07:07:38,240
so he claims test passing

128
07:07:38,240 --> 07:07:39,680
and all the checks were run

129
07:07:39,680 --> 07:07:42,100
and actually that's not what the

130
07:07:42,100 --> 07:07:43,140
code did so

131
07:07:44,080 --> 07:07:45,920
poor Alex another

132
07:07:45,920 --> 07:07:48,060
failed PR very angry

133
07:07:48,060 --> 07:07:49,860
badger with all the token cost

134
07:07:51,680 --> 07:07:53,540
kind of makes me feel like

135
07:07:53,540 --> 07:07:57,760
Monday morning with the Neo4j finance department looking at my token spend each week.

136
07:07:57,920 --> 07:08:05,020
But they're very forgiving, especially when it's an important project like this.

137
07:08:06,560 --> 07:08:09,800
So maybe Alex can save some costs by using cheaper writers.

138
07:08:09,820 --> 07:08:11,860
This is the number one thing finance has me to do.

139
07:08:12,000 --> 07:08:16,580
They're like, hey, no more opus, sonnet, haiku, do something cheaper, please.

140
07:08:17,340 --> 07:08:18,240
Does this work?

141
07:08:19,800 --> 07:08:22,180
I feel like the quality is a bit lower

142
07:08:22,180 --> 07:08:23,520
when you do it like that

143
07:08:23,520 --> 07:08:26,100
just reducing the cost doesn't quite help

144
07:08:26,780 --> 07:08:28,600
yeah so we were trying to get better quality

145
07:08:28,600 --> 07:08:30,760
and we ended up getting worse quality

146
07:08:30,760 --> 07:08:32,200
by just using cheaper writers

147
07:08:34,880 --> 07:08:37,260
and obviously Alex is in a bind here right

148
07:08:37,260 --> 07:08:39,100
he's spent too much on revisions

149
07:08:39,980 --> 07:08:42,480
he doesn't have a solution for reducing token costs

150
07:08:42,480 --> 07:08:44,660
he's not getting the results he wants out of his PRs

151
07:08:45,280 --> 07:08:45,720
but

152
07:08:47,560 --> 07:08:48,560
maybe there's

153
07:08:48,560 --> 07:08:50,440
something Pip can help us with.

154
07:08:50,560 --> 07:08:51,540
Pip has a friend.

155
07:08:52,580 --> 07:08:54,840
I wonder who this friend could be.

156
07:08:55,440 --> 07:08:56,000
Oh.

157
07:08:56,640 --> 07:08:58,680
Well, I think you know from

158
07:08:58,680 --> 07:09:00,780
my previous talks that this friend

159
07:09:00,780 --> 07:09:02,840
is definitely going to be

160
07:09:02,840 --> 07:09:03,220
a graph.

161
07:09:04,640 --> 07:09:06,920
You're the graph guy. We're doing another

162
07:09:06,920 --> 07:09:07,560
graph again.

163
07:09:08,120 --> 07:09:09,820
Yeah, there's always a new graph.

164
07:09:10,020 --> 07:09:12,800
But this is not the same old graph.

165
07:09:12,940 --> 07:09:14,640
This is not the Neo4j

166
07:09:15,440 --> 07:09:18,360
database. This is our new friend, Archie,

167
07:09:19,560 --> 07:09:23,320
who is a embedded database written in Rust

168
07:09:23,320 --> 07:09:28,320
called Relate that actually is being launched this week

169
07:09:29,420 --> 07:09:32,520
at the Nodes Conference and also this talk at A Engineer

170
07:09:32,520 --> 07:09:36,360
in San Francisco. And it's an open source

171
07:09:37,400 --> 07:09:40,240
database which allows us to get a lot of the benefits

172
07:09:40,880 --> 07:09:44,320
of more accuracy, multi-hop queries,

173
07:09:44,700 --> 07:09:46,080
more explainable traces,

174
07:09:46,720 --> 07:09:48,980
without having the heavyweight approach.

175
07:09:49,160 --> 07:09:52,120
So it's kind of an agent-focused graph database

176
07:09:52,120 --> 07:09:55,360
specifically for these sort of embedded use cases.

177
07:09:58,220 --> 07:10:00,000
Okay, so let's see how Archie can

178
07:10:00,060 --> 07:10:24,900
help us out. So before we were relying a lot on agents and loops of agents to resolve the problem, but graph databases can store relationships, can find different things which are clustered, build a brief which connects all this information together, and then give the agents, even a cheaper agent, better information and more context to give a good drafting decision

179
07:10:24,900 --> 07:10:28,780
And the good thing about Archie is he has zero LLM costs.

180
07:10:28,800 --> 07:10:32,020
It's all local compute costs and storage,

181
07:10:32,100 --> 07:10:34,380
and it's a fully in-memory database and REST,

182
07:10:34,440 --> 07:10:35,200
so it's highly optimized.

183
07:10:35,700 --> 07:10:40,340
So we get this providence about all these submitted changes

184
07:10:40,340 --> 07:10:43,340
and this evidence graph basically for free

185
07:10:43,340 --> 07:10:45,840
compared to all the LLM and token costs,

186
07:10:45,860 --> 07:10:46,940
which we were doing before.

187
07:10:47,400 --> 07:10:50,760
And this means our buddy Alex doesn't need to research.

188
07:10:50,940 --> 07:10:53,080
He doesn't need to go through and find all the facts again

189
07:10:53,080 --> 07:10:54,060
and go through the sources.

190
07:10:55,700 --> 07:10:59,280
because relate is going to do all that hard work for him

191
07:10:59,280 --> 07:11:02,800
and build an evidence packet that he can use for trace tune.

192
07:11:03,660 --> 07:11:04,680
It all sounds really great.

193
07:11:04,900 --> 07:11:06,400
Maybe we do need more graphs.

194
07:11:08,580 --> 07:11:10,200
I'm surprised to hear that from you

195
07:11:10,200 --> 07:11:12,220
after all the graph talks you've had to put up with.

196
07:11:12,860 --> 07:11:15,780
Okay, but we can do this

197
07:11:15,780 --> 07:11:18,200
and make it even better with a neighborhood brief.

198
07:11:18,400 --> 07:11:21,640
So basically taking all of the why, the code, the checks,

199
07:11:21,640 --> 07:11:24,780
putting attributions and relationships between those

200
07:11:24,780 --> 07:11:28,000
and then returning back a connected set of information as a packet.

201
07:11:28,840 --> 07:11:32,820
by doing this, then when the agent is drafting up the PR,

202
07:11:33,380 --> 07:11:37,860
it has all the connected facts with their sources together in a single source.

203
07:11:38,100 --> 07:11:43,720
And that makes the writer draft more accurate with fewer passes.

204
07:11:44,000 --> 07:11:47,260
And it also makes the reviewer's job much easier.

205
07:11:47,260 --> 07:11:49,120
So we don't need an expensive reviewer.

206
07:11:49,120 --> 07:12:11,100
We can save by using a few haiku reviewers instead of a sonnet reviewer or a Luna in the case of Codex to come up with the findings, check those against the draft, redo the writing, and then we retain the higher quality writer, a sonnet or tarot class writer to make sure we have a high quality brief that will pass human approval.

207
07:12:13,060 --> 07:12:19,180
When we were building this, it felt very counterintuitive to have the writer be higher quality than the reviewers.

208
07:12:20,960 --> 07:12:28,280
Actually, it's common that people always want to have the planner, the reviewer, the checker to be the highest quality model.

209
07:12:28,580 --> 07:12:36,700
But if you're doing high quality writing, a lower model, no matter how many iterations and how much feedback you give it, can't do it.

210
07:12:36,760 --> 07:12:39,560
It just turns things into a checklist and makes it very mechanical.

211
07:12:39,560 --> 07:12:44,340
So to give writing which feels human-like, you do need a higher quality model.

212
07:12:44,580 --> 07:12:48,800
And it's actually the reviewing and the cross-checking step, which is more mechanical.

213
07:12:48,980 --> 07:12:52,360
So it's a little counterintuitive, but that works out quite well.

214
07:12:52,540 --> 07:13:01,720
It's also a little counterintuitive that our badger has a thumbs up coming out of his chest, but let's ignore the high-quality AI artwork.

215
07:13:05,060 --> 07:13:08,000
Okay, so we've checked all the claims against the evidence.

216
07:13:08,000 --> 07:13:11,760
We've looked for missing findings with our cheaper reviewer.

217
07:13:11,900 --> 07:13:13,460
Did the focus work?

218
07:13:13,700 --> 07:13:17,920
And then found the things which were connected information

219
07:13:17,920 --> 07:13:22,200
to give a better draft and review for Mira.

220
07:13:23,900 --> 07:13:25,200
And here's the output.

221
07:13:25,400 --> 07:13:26,940
So it has the Y.

222
07:13:27,460 --> 07:13:31,660
So log CWD, and that describes the shell before the TD runs.

223
07:13:31,860 --> 07:13:32,780
What's the exception?

224
07:13:33,680 --> 07:13:37,700
So the CD targets in repo, or the command mentions a root.

225
07:13:37,700 --> 07:13:43,100
what passed. It has type check plus 71 tests, checked on submitted content, not before.

226
07:13:44,280 --> 07:13:48,620
And then it starts in correlate.ts inspecting the leading CD exception.

227
07:13:49,740 --> 07:13:53,160
And Mira looks pretty happy with these results, right?

228
07:13:54,540 --> 07:13:57,720
Yeah, this might be the first scene where she's genuinely happy.

229
07:13:59,080 --> 07:14:05,100
I think she was kind of happy with the AI engineer sweatshirt before she realized that Alex actually

230
07:14:05,100 --> 07:14:12,800
was a was a goof okay so here are some of the actual data and results from the side-by-side

231
07:14:12,800 --> 07:14:21,380
analysis using PR bench which is a pull request benchmark which uses audit traces plus code and

232
07:14:21,380 --> 07:14:27,520
LM judges to review the quality of the work of course reviewed by human as well for accuracy

233
07:14:27,520 --> 07:14:33,800
and you can see that on both Claude and Codex you get a significant quality improvements on a scale

234
07:14:33,800 --> 07:14:41,680
one to 100 the judges by and large almost unilaterally liked the trace tune plus relate

235
07:14:41,680 --> 07:14:49,920
quality better um on a one to 100 scale it was like 74 to 76 82 to 86 so plus two plus four point

236
07:14:49,920 --> 07:14:56,220
these tests contributed to my token billionaire card because we were running lots and lots of

237
07:14:56,220 --> 07:15:02,240
tests we created this side project called pr bench just for testing and we would have different

238
07:15:02,240 --> 07:15:09,680
agents test over different pull requests and compare the results. Initially, I had it running

239
07:15:10,980 --> 07:15:16,480
sequentially. It'll do one test and finish it, then do the next. But then we decided to run them

240
07:15:16,480 --> 07:15:24,140
all in parallel. It's a very quick way to reach high spend. Yeah, that's kind of funny. At the

241
07:15:24,140 --> 07:15:30,700
current testing rate with the number of use cases and tests on the benchmark, we can fully saturate

242
07:15:30,700 --> 07:15:36,840
my high-end gaming computer with as many threads as Linux can handle simultaneously.

243
07:15:37,740 --> 07:15:42,980
And of course, the token spend is slightly higher than what we'd like.

244
07:15:43,080 --> 07:15:47,100
But it's all for the good of improving the quality of pull requests for the world.

245
07:15:47,600 --> 07:15:51,780
So you all get the benefit of all these test runs and having the highest quality improvements.

246
07:15:52,920 --> 07:15:55,020
And you save on tokens.

247
07:15:55,020 --> 07:16:10,580
So we've spent the tokens for you in advance because Tracetune with the graph solution takes about half as many tokens as it did previously, which is an amazing result from the graph because now you're pre-computing.

248
07:16:10,680 --> 07:16:13,460
You're getting all this evidence graph constructed in advance.

249
07:16:14,260 --> 07:16:15,720
The writing's cheaper.

250
07:16:16,060 --> 07:16:20,420
The reviewing's cheaper with a higher quality result.

251
07:16:20,420 --> 07:16:31,560
I wasn't sure about the whole graph thing initially, but after seeing the cost tank almost in half, like it's not even a question to making this program work.

252
07:16:33,480 --> 07:16:36,020
and this is something we did for the TraceTune project

253
07:16:36,020 --> 07:16:38,660
but you can do this for pretty much any project

254
07:16:38,660 --> 07:16:41,200
where you have data, you have relationships

255
07:16:41,200 --> 07:16:44,620
you have evidence, you have things which you're trying to compare

256
07:16:44,620 --> 07:16:47,680
because agents are not that good

257
07:16:47,680 --> 07:16:49,300
at doing the data analysis

258
07:16:49,300 --> 07:16:51,980
at doing all of the evaluation and the curation

259
07:16:51,980 --> 07:16:54,860
so if you give them a help at the front end

260
07:16:54,860 --> 07:16:57,960
by creating a custom graph layer

261
07:16:57,960 --> 07:16:59,660
with an in-memory database like Relate

262
07:17:00,400 --> 07:17:03,780
suddenly your costs go down, your quality goes up, and

263
07:17:04,380 --> 07:17:08,720
you too can be a token billionaire like our friend Alex.

264
07:17:09,220 --> 07:17:11,840
Well, I guess now he's using it as a bookmark.

265
07:17:12,840 --> 07:17:16,600
I think that's a good use for it. It's a really high-quality bookmark.

266
07:17:16,600 --> 07:17:18,380
It's heavier than a credit card.

267
07:17:20,420 --> 07:17:24,620
It is a nice prize for spending all those tokens on PR bench.

268
07:17:26,240 --> 07:17:27,820
so finance is happy

269
07:17:27,820 --> 07:17:31,120
lower drafting costs that they can reasonably fund

270
07:17:31,760 --> 07:17:34,240
Alex is happy he has his new bookmark

271
07:17:34,240 --> 07:17:36,060
he now has a great case study

272
07:17:37,520 --> 07:17:41,160
with the connections doing neighborhood retrievable facts

273
07:17:41,160 --> 07:17:43,020
so that it's a better brief

274
07:17:43,020 --> 07:17:46,880
focusing the reviews so the reviewers get the same compact brief

275
07:17:46,880 --> 07:17:49,700
Mira gets exactly the evidence she needs

276
07:17:49,700 --> 07:17:51,780
and then here's Alex on stage at

277
07:17:52,740 --> 07:17:54,960
AI engineer. Somewhere in the world,

278
07:17:55,440 --> 07:17:57,060
better PRs, lower drafting

279
07:17:57,060 --> 07:17:58,540
costs with

280
07:17:58,540 --> 07:18:01,060
a graph plus open

281
07:18:01,060 --> 07:18:02,900
source. Trace to

282
07:18:02,900 --> 07:18:05,060
build, which also gives you

283
07:18:05,060 --> 07:18:05,840
amazing comics.

284
07:18:10,080 --> 07:18:10,820
Okay.

285
07:18:11,220 --> 07:18:13,000
So, big round of applause for our

286
07:18:13,000 --> 07:18:14,680
friend Alex, who's now

287
07:18:14,680 --> 07:18:16,960
earned the AI engineer

288
07:18:16,960 --> 07:18:17,280
hoodie.

289
07:18:18,200 --> 07:18:20,160
I think that's definitely a

290
07:18:20,160 --> 07:18:26,280
wardrobe upgrade for him. Yeah, I don't think the pocket protector look was that great for him. So

291
07:18:27,160 --> 07:18:33,660
good for him. I think we'll see Swix on stage with one of these hoodies soon. He can't not do it now

292
07:18:33,660 --> 07:18:39,280
that we shamed into doing it at his own conference. The token O'Billion program is amazing. A big

293
07:18:39,280 --> 07:18:44,160
kudos to the team for coming up with ways to incentivize and reward people who are in the

294
07:18:44,160 --> 07:18:49,840
leading edge of AI usage and learning how to make it really effective for the rest of us.

295
07:18:52,260 --> 07:19:03,220
And now you too can have better PRs impress your coworkers by having well-researched, effectively written, and lower-cost PRs.

296
07:19:03,320 --> 07:19:04,080
With a comic.

297
07:19:05,080 --> 07:19:06,060
With a comic.

298
07:19:06,420 --> 07:19:08,920
So for more information, go to tracetune.com.

299
07:19:09,640 --> 07:19:11,600
Learn about Trace2 and give it a try.

300
07:19:11,900 --> 07:19:12,580
It works with Quad.

301
07:19:12,640 --> 07:19:13,480
It works with Codex.

302
07:19:13,500 --> 07:19:15,080
It works with all your favorite AI tools.

303
07:19:15,080 --> 07:19:18,620
and if you want to do these same sort of amazing results

304
07:19:18,620 --> 07:19:21,100
on your own project, try out Relate,

305
07:19:21,800 --> 07:19:25,000
our new Rust-based embedded open source graph database,

306
07:19:25,340 --> 07:19:27,820
which brings context to your agents

307
07:19:27,820 --> 07:19:29,820
and helps you build more efficient applications

308
07:19:29,820 --> 07:19:32,820
that just work better or cheaper

309
07:19:32,820 --> 07:19:34,340
and give you better results.

310
07:19:34,680 --> 07:19:36,120
So thank you all for joining our

311
07:19:36,120 --> 07:19:38,420
AI Engineer San Francisco talk

312
07:19:38,420 --> 07:19:41,360
and if you're watching this talk this week

313
07:19:41,360 --> 07:19:42,440
at San Francisco,

314
07:19:43,780 --> 07:19:47,600
what should you do this weekend with your kids on Saturday, Cassandra?

315
07:19:48,280 --> 07:19:50,940
We are running an AI Engineer Kids Day.

316
07:19:51,280 --> 07:19:54,660
So anyone who's available, just sign your kids up.

317
07:19:55,040 --> 07:19:56,240
We're teaching kids workshops.

318
07:19:56,580 --> 07:20:00,000
Kids will be learning how to do AI programming to make their own

319
07:20:00,060 --> 07:20:00,500
games.

320
07:20:00,760 --> 07:20:04,540
And I think we're also doing a, no, it's just the gaming one.

321
07:20:06,200 --> 07:20:07,840
No, we'll do the robots too.

322
07:20:08,200 --> 07:20:14,660
Okay. Yeah, we'll also be doing robots. So kids will get to control these little soda can robots using AI.

323
07:20:15,580 --> 07:20:20,400
Yeah. So a lot of fun. Physical AI, gaming AI, all in one place.

324
07:20:21,100 --> 07:20:23,700
sign up at AI Engineer Kids Day

325
07:20:23,700 --> 07:20:27,500
and bring your kids, bring your nieces, bring your nephews

326
07:20:27,500 --> 07:20:30,900
and let's train the next generation of AI engineers.

327
07:20:32,780 --> 07:20:35,400
All right, thanks so much and enjoy the rest of the conference.