If You Are Going to Verbally Abuse Your Agents, at Least Label the Outcome

1:59:07Thanks for coming to my talk.

1:59:08I'm BDougie, Brian Douglas, and I'm going to talk about if you are going to verbally abuse your agents, at least label the outcome.

1:59:15Now, this is a talk I'm going to quickly just run through one example, and this will be a short talk, mainly because there's actually not too much to this.

1:59:24But outside of that, I want to talk about some of the work I've been doing for the past six to eight months.

1:59:30And it actually came out with this idea early this year.

1:59:33Everyone definitely got AI pilled when it comes to agents.

1:59:36So obviously, Cloud Codes got to be one of the most popular ones out there.

1:59:40We've got quite a few other ones that you can leverage, like the Codex or ChatGPT.

1:59:45And there was a book I read years ago, well, quite a handful of years ago, from Rick Rubin about the creative act, a way of being.

1:59:52And this was actually around the idea that Rick Rubin produced the BC Boys and a number of other acts, but he's actually never played music.

2:00:00And a lot of folks have made this correlation with vibe coding, where Rick Rubin can produce the best music without actually playing the piano.

2:00:08We're now having, we're in a world where folks can vibe code a bunch of Rust code without never writing Rust.

2:00:15And that's exactly where I'm at.

2:00:17There was another article months ago from Addy Osmone,

2:00:21who's now currently at the Anthropic team,

2:00:25the Quadco team specifically,

2:00:27and was talking about the opportunity for juniors,

2:00:31or basically it's gone.

2:00:33When it comes to junior engineering,

2:00:35it was all about growing from junior to senior was about reps.

2:00:39So the more reps you got,

2:00:40the more engagement you got opportunity to basically just level up as an

2:00:44engineer.

2:00:45and without those reps, we're handing those reps specifically to agents,

2:00:49we're losing the opportunity to level up.

2:00:52So this comes out of something I was doing early this year,

2:00:54which built this open source project called Tapes.

2:00:57Tapes.dev is open source Apache 2.

2:01:00There's a lot of other similar stuff out there,

2:01:02but what I'm getting at is it's not about Tapes.

2:01:04It's about the idea of collecting your agent's session data,

2:01:07and that's what this does.

2:01:08So a lot of hotel collectors, a lot of session capture services,

2:01:12a lot of the harnesses are to do this for you out of the box.

2:01:14I just want to mention that this is what I use.

2:01:17You can use anything else you would like.

2:01:19I've also, in addition to tapes, me and my team, we've developed this thing called stereos,

2:01:23and this is the runtime for your agent.

2:01:25Again, sandboxes, not a concept that everyone needs to hear about today,

2:01:30but just knowing that in all the work I've been doing, I've been capturing these sessions

2:01:34and recording them by having a reproducible environment.

2:01:39And the environment themselves, they have these signals.

2:01:42sin. The one thing I've been doing it with these, this little setup of tapes and stereos is I've

2:01:47been building self-healing infrastructure. So this is a little project called Sweeper Agent

2:01:51and Sweeper Agent itself is a way for me to build parallel agents. I can scale up to 10 about 10

2:01:57of these at this point. I've done quite a few like Daytona sandboxes as well. I've done like

2:02:02about 200 at the same time that will just solve a specific problem. So documentation, test writing,

2:02:07code cleanup stuff that you want to scale that's like very very defined work that you want to spin

2:02:13off to multiple agents in tandem now a lot of this has been cleaning up most of my vibe coding

2:02:20software that i built in the last 18 months at this point but just pointing out that like there's

2:02:26a lot of work we can do to basically solve these problems but it doesn't change the fact that i

2:02:32still don't know how to write rust code i've never written write rust code specifically myself

2:02:37professionally rather.

2:02:39Ten years ago, I did write Rust code,

2:02:40but it was mostly for little side projects to learn Rust.

2:02:43But we use Rust day-to-day at the company I work for,

2:02:46which is Paper Compute.

2:02:47And what we've been doing is,

2:02:49because of this technology underneath this called Tapes,

2:02:52we've been capturing traces.

2:02:54And Tapes is our trace capture service.

2:02:57Telemetry capture and paper, we have a console.

2:02:59So for the moment, I'll move away from the slides

2:03:02and show you the paper console.

2:03:04Now, paper console itself,

2:03:05just showing that I can capture sessions.

2:03:07The difference of doing this local and my own sessions

2:03:10is that I can have a centralized data store

2:03:13of not just me but also my teammate.

2:03:15And what's cool about this is I can also label sessions.

2:03:18So this is a session that...

2:03:22Oh, interesting.

2:03:23Yeah, sorry.

2:03:24I've got to zoom out to see the labels.

2:03:26But yeah, so here I can see I've got a session here

2:03:29which is working with some paper cloud export memory limits

2:03:32and we've got this label called pushback.

2:03:34Now, I actually went to specifically a session that had pushback in it, and it's specifically every time an engineer pushes back on the agent.

2:03:43Now, there's a thing that I think recently Anthropic, or I guess rumors, I don't know if this is actually grounded by any facts, but Anthropic was pushing back on if you're being rude to the agent, you could potentially lose your account or get blocked.

2:03:58but what I find specifically when pushed back when I'm specifically looking for any Rust code

2:04:05or Go code that Matt has written and whenever he pushes back it's a good signal that he has

2:04:11knowledge that the agent does not and this usually comes in the form of we have decided our API

2:04:17structure a certain way despite the fact that the Rust code compiler compiles we have made decisions

2:04:22on how that code should be set up and how that code should interact with different parts of the

2:04:26code base also pushing back on things like duplicating uh services that made the agent

2:04:31didn't have context around there's a lot of other like a bunch of reasons but what i'm getting at

2:04:36me as like quote unquote a junior rust engineer i don't have enough context to be able to push

2:04:42back to the agent to say hey you're doing this wrong or you're making like weird choices

2:04:45i usually take everything that the agent writes in rust for face value and what that ends up

2:04:50turning into is when i open up pr i get a lot of pushback from the engineers the human engineers

2:04:55at review time. This is mainly because we know I don't write rest code. There's not a lot of

2:05:00pushback I can provide to the code base or even at the time of review. So by the time it gets to

2:05:05greptile, it's a five out of five review. I just let it slide. So how do I prevent that pushback?

2:05:11What I've done is actually started collecting every single one of these labels and the traces.

2:05:15And then one thing I'll point out is that the console itself, I capture every single session.

2:05:21I've got all the specific details.

2:05:23This is actually just a command line version of that.

2:05:26What I've been doing is I've been capturing all these sessions

2:05:29and then dropping into a data set that's been labeled.

2:05:32So now I can just run a little command line tool called TwinReview,

2:05:36and then I'll grab a PR number, 381, and I'll run that.

2:05:41And I'll actually run through and essentially review the code as if it was Matt.

2:05:46Now, the way I'm doing this is I'm capturing traces.

2:05:48I created the data set.

2:05:49I've also done some post training.

2:05:51and this is like where, you know, choose your own adventure,

2:05:54but I've done some post training specifically

2:05:56on every time Matt pushes back within Go or Rust code

2:06:00and I asked for it to provide me comments

2:06:03as a viewer, Matt, on my GitHub PR.

2:06:06So this is all data.

2:06:07It's only specifically within the traces.

2:06:09So this is not taking any GitHub PR review data,

2:06:11which I think you potentially could also follow that pattern.

2:06:14But what I find is like the majority of our team

2:06:17have started reviewing code on the command line.

2:06:20So you'll have Claude review the code or the PyHarness review code, or you review this PR, but you don't do a lot of pushback or a lot of commentary on GitHub itself.

2:06:29So what we're seeing right now, we're shifting from code being reviewed specifically on GitHub by agents, well, previously humans and now agents, to now codes is specifically just getting reviewed in your harness without any sort of oversight or commentary or anything that sort of navigates directly to your shared code base.

2:06:50So with that being said, like what I've navigated is I've created this thing called a twin reviewer.

2:06:55And this as well is open sourced.

2:06:59So it is, and I did not have this open, but, oh, actually, TwinReview is not open source.

2:07:06I realize I never open source it because I got sensitive data.

2:07:10But what I've done is actually set up that same pattern using Fireworks and using Fireworks to post-train specifically.

2:07:16And what's cool about this is, like, I've looked for any sort of pushbacks, apologies, any sort of golden sessions,

2:07:22sessions that actually have been marked as you should probably pay attention to this and also

2:07:28regressions for any things that usually agents will just fall over. A lot of times it turns into

2:07:33like an agent will start reinventing Go libraries that already exist. That usually is granted to be

2:07:40pushed back. Ideally, you should push back on that. But again, if you don't know these libraries

2:07:44exist, you probably just take it for face value and then you're reinventing things like Kubernetes,

2:07:48which sounds like an overstatement, but we've seen it reinvent things like mock libraries for testing, etc.

2:07:56And very popular code base, or very popular open source projects.

2:08:00So that's essentially it.

2:08:01This is basically, if you're going to push back and you're going to have some really, really valuable insight under reviews in your code base,

2:08:09my recommendation is take a look at opportunities within the traces to maybe train or post-train a model or create your own harness for reviews locally.

2:08:20Choose your own adventure. I think you could actually do a lot of this by just building your own harness.

2:08:24But if you did want to use a Quinn 3-8B, which I've done specifically in this situation, you'd also approach that.

2:08:32A lot of the stuff is available to you, a lot of tools.

2:08:35What I did actually to start is I started taking the paper traces and creating skills.

2:08:40Skills that I can then specifically do things like check the tapes.

2:08:44So if you have a database, which tapes specifically is a Postgres database local on my machine.

2:08:49So you don't have to use the paper console.

2:08:51Actually, paper console is currently waitlisted.

2:08:53So probably best use case is going to use something like tapes or a brain trust or something similar to capture your traces.

2:09:01And then what's cool about this is once you have skills, you can create evals.

2:09:04Once you have evals, you can start post training on a Quinn model and for specific reasons,

2:09:10for identifying if rush code is up to speed to how you're writing code across your organization.

2:09:17So I did write a blog post around training a model instead of writing a skill.

2:09:20My recommendation is probably write a skill first.

2:09:22You can probably get most mileage out of a skill today.

2:09:28and I specifically across my team we had about almost 80,000 sessions that we could create

2:09:34skills from which is pretty cool. Cool thing about this is also you do specialized fine tuning or

2:09:38direct preference optimization. The latter is going to take a lot more compute or a larger

2:09:43some large hardware to do that so probably specialized fine tuning. If you know about

2:09:47QuickQ or that's how I'm doing this I'm quantizing it on smaller GPUs like a 5090

2:09:53and if you don't know about specialized fine-tuning this is like writing notes in the

2:09:57margin of a book if your book is your model the notes in the

2:10:00margin are your weights or your adapters, rather.

2:10:03So the weights you generate on top of the model that's already been pre-trained. Cool. And I did this, as you saw, on a Quintin 4B model. It works pretty good.

2:10:13I did try to do this on some larger models so specifically for direct preference optimization

2:10:20it took a lot of compute the only time I could make this work is by using NVIDIA GPUs on the

2:10:26Brev service or actually I never got access to some of the stuff on Fireworks so I only did

2:10:32smaller six thousands but if you if you care to know direct preference optimization if the margin

2:10:40notes are your specialized fine-tuning dpo is going to be like your cliff notes uh not not

2:10:46possible doing a small language model or a small 4b model uh you would need like 70b plus uh which

2:10:52is going to take some hardware uh me specifically i didn't have any as hardware local so i had to use

2:10:57um the services that i mentioned fireworks and well not even fireworks brev was the only one i

2:11:02can get access to at the time good luck if you can get access to larger uh well you got to prepay a

2:11:08for this stuff.

2:11:09So best of luck on that one.

2:11:11But my suggestion is create skills.

2:11:13Try out tapes.dev if you're interested

2:11:15or leverage the tool that you have at year already.

2:11:18And reminder, you should probably read this book.

2:11:22This is a great book if you think about, you know,

2:11:24how we're looking forward into,

2:11:26how we're approaching this world

2:11:27when it comes to AI and vibe coding

2:11:29and leveraging agents to basically build your software factory.

2:11:33And I do recommend, please check out Adios Mani's blog post.

2:11:37has a series of these on the X.

2:11:39I think they're all great.

2:11:40Definitely take a look at them.

2:11:41He's got some clear-minded dots to check out.

2:11:45I'm BDougie, and I do appreciate you watching this video.

2:11:48I've tried to keep it brief because it was pretty straightforward.

2:11:50Hopefully it was.

2:11:51Cheers.