1
01:59:07,260 --> 01:59:08,200
Thanks for coming to my talk.

2
01:59:08,500 --> 01:59:15,380
I'm BDougie, Brian Douglas, and I'm going to talk about if you are going to verbally abuse your agents, at least label the outcome.

3
01:59:15,700 --> 01:59:24,120
Now, this is a talk I'm going to quickly just run through one example, and this will be a short talk, mainly because there's actually not too much to this.

4
01:59:24,120 --> 01:59:29,800
But outside of that, I want to talk about some of the work I've been doing for the past six to eight months.

5
01:59:30,720 --> 01:59:33,060
And it actually came out with this idea early this year.

6
01:59:33,700 --> 01:59:36,140
Everyone definitely got AI pilled when it comes to agents.

7
01:59:36,900 --> 01:59:40,400
So obviously, Cloud Codes got to be one of the most popular ones out there.

8
01:59:40,540 --> 01:59:44,080
We've got quite a few other ones that you can leverage, like the Codex or ChatGPT.

9
01:59:45,480 --> 01:59:52,840
And there was a book I read years ago, well, quite a handful of years ago, from Rick Rubin about the creative act, a way of being.

10
01:59:52,840 --> 02:00:00,000
And this was actually around the idea that Rick Rubin produced the BC Boys and a number of other acts, but he's actually never played music.

11
02:00:00,640 --> 02:00:07,680
And a lot of folks have made this correlation with vibe coding, where Rick Rubin can produce the best music without actually playing the piano.

12
02:00:08,300 --> 02:00:15,000
We're now having, we're in a world where folks can vibe code a bunch of Rust code without never writing Rust.

13
02:00:15,220 --> 02:00:16,640
And that's exactly where I'm at.

14
02:00:17,260 --> 02:00:21,180
There was another article months ago from Addy Osmone,

15
02:00:21,300 --> 02:00:24,880
who's now currently at the Anthropic team,

16
02:00:25,040 --> 02:00:26,420
the Quadco team specifically,

17
02:00:27,080 --> 02:00:31,320
and was talking about the opportunity for juniors,

18
02:00:31,440 --> 02:00:33,080
or basically it's gone.

19
02:00:33,340 --> 02:00:35,480
When it comes to junior engineering,

20
02:00:35,880 --> 02:00:38,880
it was all about growing from junior to senior was about reps.

21
02:00:39,060 --> 02:00:40,140
So the more reps you got,

22
02:00:40,340 --> 02:00:44,420
the more engagement you got opportunity to basically just level up as an

23
02:00:44,420 --> 02:00:44,780
engineer.

24
02:00:45,200 --> 02:00:48,640
and without those reps, we're handing those reps specifically to agents,

25
02:00:49,060 --> 02:00:50,720
we're losing the opportunity to level up.

26
02:00:52,060 --> 02:00:54,920
So this comes out of something I was doing early this year,

27
02:00:54,960 --> 02:00:57,800
which built this open source project called Tapes.

28
02:00:57,860 --> 02:00:59,700
Tapes.dev is open source Apache 2.

29
02:01:00,600 --> 02:01:02,380
There's a lot of other similar stuff out there,

30
02:01:02,460 --> 02:01:04,680
but what I'm getting at is it's not about Tapes.

31
02:01:04,780 --> 02:01:07,400
It's about the idea of collecting your agent's session data,

32
02:01:07,460 --> 02:01:08,180
and that's what this does.

33
02:01:08,360 --> 02:01:11,880
So a lot of hotel collectors, a lot of session capture services,

34
02:01:12,460 --> 02:01:14,680
a lot of the harnesses are to do this for you out of the box.

35
02:01:14,780 --> 02:01:16,940
I just want to mention that this is what I use.

36
02:01:17,100 --> 02:01:18,280
You can use anything else you would like.

37
02:01:19,160 --> 02:01:23,980
I've also, in addition to tapes, me and my team, we've developed this thing called stereos,

38
02:01:23,980 --> 02:01:25,500
and this is the runtime for your agent.

39
02:01:25,880 --> 02:01:30,460
Again, sandboxes, not a concept that everyone needs to hear about today,

40
02:01:30,520 --> 02:01:34,060
but just knowing that in all the work I've been doing, I've been capturing these sessions

41
02:01:34,060 --> 02:01:38,400
and recording them by having a reproducible environment.

42
02:01:39,420 --> 02:01:42,660
And the environment themselves, they have these signals.

43
02:01:42,660 --> 02:01:47,560
sin. The one thing I've been doing it with these, this little setup of tapes and stereos is I've

44
02:01:47,560 --> 02:01:51,600
been building self-healing infrastructure. So this is a little project called Sweeper Agent

45
02:01:51,600 --> 02:01:57,740
and Sweeper Agent itself is a way for me to build parallel agents. I can scale up to 10 about 10

46
02:01:57,740 --> 02:02:02,380
of these at this point. I've done quite a few like Daytona sandboxes as well. I've done like

47
02:02:02,380 --> 02:02:07,920
about 200 at the same time that will just solve a specific problem. So documentation, test writing,

48
02:02:07,920 --> 02:02:13,080
code cleanup stuff that you want to scale that's like very very defined work that you want to spin

49
02:02:13,080 --> 02:02:19,260
off to multiple agents in tandem now a lot of this has been cleaning up most of my vibe coding

50
02:02:20,340 --> 02:02:26,700
software that i built in the last 18 months at this point but just pointing out that like there's

51
02:02:26,700 --> 02:02:32,100
a lot of work we can do to basically solve these problems but it doesn't change the fact that i

52
02:02:32,100 --> 02:02:37,180
still don't know how to write rust code i've never written write rust code specifically myself

53
02:02:37,860 --> 02:02:38,600
professionally rather.

54
02:02:39,100 --> 02:02:40,700
Ten years ago, I did write Rust code,

55
02:02:40,780 --> 02:02:43,020
but it was mostly for little side projects to learn Rust.

56
02:02:43,760 --> 02:02:46,440
But we use Rust day-to-day at the company I work for,

57
02:02:46,540 --> 02:02:47,380
which is Paper Compute.

58
02:02:47,820 --> 02:02:49,140
And what we've been doing is,

59
02:02:49,380 --> 02:02:52,760
because of this technology underneath this called Tapes,

60
02:02:52,780 --> 02:02:54,440
we've been capturing traces.

61
02:02:54,760 --> 02:02:56,780
And Tapes is our trace capture service.

62
02:02:57,180 --> 02:02:59,680
Telemetry capture and paper, we have a console.

63
02:02:59,940 --> 02:03:02,540
So for the moment, I'll move away from the slides

64
02:03:02,540 --> 02:03:03,980
and show you the paper console.

65
02:03:04,240 --> 02:03:05,840
Now, paper console itself,

66
02:03:05,840 --> 02:03:07,740
just showing that I can capture sessions.

67
02:03:07,760 --> 02:03:10,820
The difference of doing this local and my own sessions

68
02:03:10,820 --> 02:03:13,200
is that I can have a centralized data store

69
02:03:13,200 --> 02:03:15,040
of not just me but also my teammate.

70
02:03:15,340 --> 02:03:18,120
And what's cool about this is I can also label sessions.

71
02:03:18,400 --> 02:03:20,980
So this is a session that...

72
02:03:22,320 --> 02:03:23,160
Oh, interesting.

73
02:03:23,800 --> 02:03:24,620
Yeah, sorry.

74
02:03:24,940 --> 02:03:26,200
I've got to zoom out to see the labels.

75
02:03:26,380 --> 02:03:29,140
But yeah, so here I can see I've got a session here

76
02:03:29,140 --> 02:03:32,360
which is working with some paper cloud export memory limits

77
02:03:32,360 --> 02:03:34,460
and we've got this label called pushback.

78
02:03:34,460 --> 02:03:43,240
Now, I actually went to specifically a session that had pushback in it, and it's specifically every time an engineer pushes back on the agent.

79
02:03:43,500 --> 02:03:58,780
Now, there's a thing that I think recently Anthropic, or I guess rumors, I don't know if this is actually grounded by any facts, but Anthropic was pushing back on if you're being rude to the agent, you could potentially lose your account or get blocked.

80
02:03:58,780 --> 02:04:05,000
but what I find specifically when pushed back when I'm specifically looking for any Rust code

81
02:04:05,000 --> 02:04:11,340
or Go code that Matt has written and whenever he pushes back it's a good signal that he has

82
02:04:11,340 --> 02:04:17,140
knowledge that the agent does not and this usually comes in the form of we have decided our API

83
02:04:17,140 --> 02:04:22,620
structure a certain way despite the fact that the Rust code compiler compiles we have made decisions

84
02:04:22,620 --> 02:04:26,860
on how that code should be set up and how that code should interact with different parts of the

85
02:04:26,860 --> 02:04:31,420
code base also pushing back on things like duplicating uh services that made the agent

86
02:04:31,420 --> 02:04:36,540
didn't have context around there's a lot of other like a bunch of reasons but what i'm getting at

87
02:04:36,540 --> 02:04:42,000
me as like quote unquote a junior rust engineer i don't have enough context to be able to push

88
02:04:42,000 --> 02:04:45,360
back to the agent to say hey you're doing this wrong or you're making like weird choices

89
02:04:45,360 --> 02:04:50,360
i usually take everything that the agent writes in rust for face value and what that ends up

90
02:04:50,360 --> 02:04:55,440
turning into is when i open up pr i get a lot of pushback from the engineers the human engineers

91
02:04:55,440 --> 02:05:00,080
at review time. This is mainly because we know I don't write rest code. There's not a lot of

92
02:05:00,080 --> 02:05:05,260
pushback I can provide to the code base or even at the time of review. So by the time it gets to

93
02:05:05,260 --> 02:05:11,080
greptile, it's a five out of five review. I just let it slide. So how do I prevent that pushback?

94
02:05:11,320 --> 02:05:15,220
What I've done is actually started collecting every single one of these labels and the traces.

95
02:05:15,700 --> 02:05:21,980
And then one thing I'll point out is that the console itself, I capture every single session.

96
02:05:21,980 --> 02:05:23,800
I've got all the specific details.

97
02:05:23,940 --> 02:05:25,560
This is actually just a command line version of that.

98
02:05:26,080 --> 02:05:29,540
What I've been doing is I've been capturing all these sessions

99
02:05:29,540 --> 02:05:31,880
and then dropping into a data set that's been labeled.

100
02:05:32,300 --> 02:05:35,680
So now I can just run a little command line tool called TwinReview,

101
02:05:36,480 --> 02:05:41,060
and then I'll grab a PR number, 381, and I'll run that.

102
02:05:41,160 --> 02:05:46,100
And I'll actually run through and essentially review the code as if it was Matt.

103
02:05:46,360 --> 02:05:48,320
Now, the way I'm doing this is I'm capturing traces.

104
02:05:48,500 --> 02:05:49,380
I created the data set.

105
02:05:49,520 --> 02:05:50,920
I've also done some post training.

106
02:05:51,460 --> 02:05:53,940
and this is like where, you know, choose your own adventure,

107
02:05:54,080 --> 02:05:56,720
but I've done some post training specifically

108
02:05:56,720 --> 02:06:00,280
on every time Matt pushes back within Go or Rust code

109
02:06:00,280 --> 02:06:03,340
and I asked for it to provide me comments

110
02:06:03,340 --> 02:06:05,720
as a viewer, Matt, on my GitHub PR.

111
02:06:06,040 --> 02:06:07,480
So this is all data.

112
02:06:07,620 --> 02:06:09,360
It's only specifically within the traces.

113
02:06:09,360 --> 02:06:11,700
So this is not taking any GitHub PR review data,

114
02:06:11,900 --> 02:06:14,560
which I think you potentially could also follow that pattern.

115
02:06:14,940 --> 02:06:17,140
But what I find is like the majority of our team

116
02:06:17,140 --> 02:06:20,000
have started reviewing code on the command line.

117
02:06:20,000 --> 02:06:29,240
So you'll have Claude review the code or the PyHarness review code, or you review this PR, but you don't do a lot of pushback or a lot of commentary on GitHub itself.

118
02:06:29,240 --> 02:06:50,500
So what we're seeing right now, we're shifting from code being reviewed specifically on GitHub by agents, well, previously humans and now agents, to now codes is specifically just getting reviewed in your harness without any sort of oversight or commentary or anything that sort of navigates directly to your shared code base.

119
02:06:50,860 --> 02:06:55,740
So with that being said, like what I've navigated is I've created this thing called a twin reviewer.

120
02:06:55,820 --> 02:06:58,120
And this as well is open sourced.

121
02:06:59,260 --> 02:07:06,660
So it is, and I did not have this open, but, oh, actually, TwinReview is not open source.

122
02:07:06,920 --> 02:07:09,740
I realize I never open source it because I got sensitive data.

123
02:07:10,600 --> 02:07:16,360
But what I've done is actually set up that same pattern using Fireworks and using Fireworks to post-train specifically.

124
02:07:16,800 --> 02:07:22,280
And what's cool about this is, like, I've looked for any sort of pushbacks, apologies, any sort of golden sessions,

125
02:07:22,280 --> 02:07:28,360
sessions that actually have been marked as you should probably pay attention to this and also

126
02:07:28,360 --> 02:07:33,240
regressions for any things that usually agents will just fall over. A lot of times it turns into

127
02:07:33,760 --> 02:07:40,220
like an agent will start reinventing Go libraries that already exist. That usually is granted to be

128
02:07:40,220 --> 02:07:44,460
pushed back. Ideally, you should push back on that. But again, if you don't know these libraries

129
02:07:44,460 --> 02:07:48,720
exist, you probably just take it for face value and then you're reinventing things like Kubernetes,

130
02:07:48,720 --> 02:07:55,000
which sounds like an overstatement, but we've seen it reinvent things like mock libraries for testing, etc.

131
02:07:56,200 --> 02:07:59,780
And very popular code base, or very popular open source projects.

132
02:08:00,020 --> 02:08:01,600
So that's essentially it.

133
02:08:01,700 --> 02:08:09,360
This is basically, if you're going to push back and you're going to have some really, really valuable insight under reviews in your code base,

134
02:08:09,360 --> 02:08:19,760
my recommendation is take a look at opportunities within the traces to maybe train or post-train a model or create your own harness for reviews locally.

135
02:08:20,260 --> 02:08:23,260
Choose your own adventure. I think you could actually do a lot of this by just building your own harness.

136
02:08:24,120 --> 02:08:32,540
But if you did want to use a Quinn 3-8B, which I've done specifically in this situation, you'd also approach that.

137
02:08:32,980 --> 02:08:35,940
A lot of the stuff is available to you, a lot of tools.

138
02:08:35,940 --> 02:08:40,100
What I did actually to start is I started taking the paper traces and creating skills.

139
02:08:40,900 --> 02:08:44,520
Skills that I can then specifically do things like check the tapes.

140
02:08:44,700 --> 02:08:49,540
So if you have a database, which tapes specifically is a Postgres database local on my machine.

141
02:08:49,860 --> 02:08:51,380
So you don't have to use the paper console.

142
02:08:51,580 --> 02:08:53,580
Actually, paper console is currently waitlisted.

143
02:08:53,800 --> 02:09:00,200
So probably best use case is going to use something like tapes or a brain trust or something similar to capture your traces.

144
02:09:01,300 --> 02:09:04,220
And then what's cool about this is once you have skills, you can create evals.

145
02:09:04,220 --> 02:09:09,960
Once you have evals, you can start post training on a Quinn model and for specific reasons,

146
02:09:10,140 --> 02:09:16,280
for identifying if rush code is up to speed to how you're writing code across your organization.

147
02:09:17,460 --> 02:09:20,620
So I did write a blog post around training a model instead of writing a skill.

148
02:09:20,840 --> 02:09:22,740
My recommendation is probably write a skill first.

149
02:09:22,960 --> 02:09:26,280
You can probably get most mileage out of a skill today.

150
02:09:28,080 --> 02:09:34,380
and I specifically across my team we had about almost 80,000 sessions that we could create

151
02:09:34,380 --> 02:09:38,680
skills from which is pretty cool. Cool thing about this is also you do specialized fine tuning or

152
02:09:38,680 --> 02:09:43,300
direct preference optimization. The latter is going to take a lot more compute or a larger

153
02:09:43,300 --> 02:09:47,660
some large hardware to do that so probably specialized fine tuning. If you know about

154
02:09:47,660 --> 02:09:52,680
QuickQ or that's how I'm doing this I'm quantizing it on smaller GPUs like a 5090

155
02:09:53,600 --> 02:09:57,020
and if you don't know about specialized fine-tuning this is like writing notes in the

156
02:09:57,020 --> 02:09:59,980
margin of a book if your book is your model the notes in the

157
02:10:00,000 --> 02:10:03,540
margin are your weights or your adapters, rather.

158
02:10:03,800 --> 02:10:13,400
So the weights you generate on top of the model that's already been pre-trained. Cool. And I did this, as you saw, on a Quintin 4B model. It works pretty good.

159
02:10:13,400 --> 02:10:20,560
I did try to do this on some larger models so specifically for direct preference optimization

160
02:10:20,560 --> 02:10:26,500
it took a lot of compute the only time I could make this work is by using NVIDIA GPUs on the

161
02:10:26,500 --> 02:10:32,480
Brev service or actually I never got access to some of the stuff on Fireworks so I only did

162
02:10:32,480 --> 02:10:40,460
smaller six thousands but if you if you care to know direct preference optimization if the margin

163
02:10:40,460 --> 02:10:46,420
notes are your specialized fine-tuning dpo is going to be like your cliff notes uh not not

164
02:10:46,420 --> 02:10:52,840
possible doing a small language model or a small 4b model uh you would need like 70b plus uh which

165
02:10:52,840 --> 02:10:57,540
is going to take some hardware uh me specifically i didn't have any as hardware local so i had to use

166
02:10:57,540 --> 02:11:02,580
um the services that i mentioned fireworks and well not even fireworks brev was the only one i

167
02:11:02,580 --> 02:11:08,720
can get access to at the time good luck if you can get access to larger uh well you got to prepay a

168
02:11:08,720 --> 02:11:09,420
for this stuff.

169
02:11:09,600 --> 02:11:10,700
So best of luck on that one.

170
02:11:11,400 --> 02:11:13,600
But my suggestion is create skills.

171
02:11:13,880 --> 02:11:15,560
Try out tapes.dev if you're interested

172
02:11:15,560 --> 02:11:18,020
or leverage the tool that you have at year already.

173
02:11:18,880 --> 02:11:22,040
And reminder, you should probably read this book.

174
02:11:22,140 --> 02:11:23,900
This is a great book if you think about, you know,

175
02:11:24,080 --> 02:11:25,440
how we're looking forward into,

176
02:11:26,240 --> 02:11:27,460
how we're approaching this world

177
02:11:27,460 --> 02:11:29,440
when it comes to AI and vibe coding

178
02:11:29,440 --> 02:11:32,780
and leveraging agents to basically build your software factory.

179
02:11:33,200 --> 02:11:37,060
And I do recommend, please check out Adios Mani's blog post.

180
02:11:37,700 --> 02:11:38,940
has a series of these on the X.

181
02:11:39,260 --> 02:11:40,320
I think they're all great.

182
02:11:40,520 --> 02:11:41,480
Definitely take a look at them.

183
02:11:41,580 --> 02:11:44,980
He's got some clear-minded dots to check out.

184
02:11:45,540 --> 02:11:48,200
I'm BDougie, and I do appreciate you watching this video.

185
02:11:48,400 --> 02:11:50,660
I've tried to keep it brief because it was pretty straightforward.

186
02:11:50,960 --> 02:11:51,660
Hopefully it was.

187
02:11:51,920 --> 02:11:52,300
Cheers.