AI Engineer World's Fair 2026
Everything Is a Rollout — Alex Shaw + Ryan Marten, Terminal-Bench, Harbor, Laude Institute
Read the talk
Everything Is a Rollout
Harbor connects agent evaluation, production work, and training through one execution loop: give an agent a task, preserve its trajectory, and grade the resulting state.
From a talk by Alex Shaw and Ryan Marten
Before you start: Familiarity with coding agents, command-line tools, and basic model evaluation will help; no reinforcement-learning background is required.
When reading the code was enough
A coworker sends you a 400-line pull request. You get coffee, sit down, admire the small refactors and clear method names, and leave a comment about an API that needs to be more extensible. This is the 2018 code-review scene Alex Shaw borrows from StaySaaSY to introduce Harbor, the agent evaluation and reinforcement-learning environment framework he works on at Laude Institute. The surrounding nostalgia includes Avengers: Infinity War, the arrival of GPT-1, Musical.ly becoming TikTok, and the novelty of cordless AirPods. The familiar engineering ritual is the important part: reading the implementation gives the reviewer a working explanation of what it will do.
That workflow already feels distant to Shaw, even though he places the change only six to eighteen months earlier, depending on when someone adopted coding agents. His deliberately sharp characterization of the earlier era is that software engineering meant knowing what the code would do before running it. François Chollet supplies the next step: generated code should be treated as a black-box artifact whose behavior and generalization must be established through empirical evaluation. Once confidence depends on observing behavior, agent development needs tools for experiments, not just tools for inspecting implementations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A phone number reveals the trade-off
Consider a program that extracts phone numbers from text using a regular expression. Its accepted format is explicit in the implementation. Shaw says he can predict the regex program’s behavior with 100% confidence across one million executions. A small illustration makes the boundary visible:
python
import re
texts = [
"Call 415-555-0132.",
"Call (415) 555 0132.",
]
pattern = re.compile(r"\b\d{3}-\d{3}-\d{4}\b")
for text in texts:
print(pattern.findall(text))
The first input matches; the second does not. Predictability does not mean broad coverage—it means the rule’s limitations are also predictable.
Shaw then replaces the regex with a model call asking GPT-5.5 to extract the phone number. He expects this version to handle unusual formatting that the regex would miss. But he cannot promise identical, known output over the same hypothetical million executions. This is a simple task that he expects to work almost every time; the uncertainty becomes more consequential as tasks grow more complex.
The change is broader than generated code. Shaw edits Chollet’s statement twice: agentic coding becomes agents in general, and generated code becomes agent performance. The behavior of the whole agent is what must be evaluated empirically. Inspecting its prompt, tools, or source code does not establish how reliably it will accomplish a task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The machine-learning loop, translated
Treating agent development as machine learning exposes both the tools it needs and the mistakes it can repeat. Shaw maps familiar ML concepts onto the parts an agent builder can change:
| Machine learning | Agent development |
|---|---|
| Training data | Environments |
| Test and validation sets | Evals, also built from environments |
| Model weights | Skills, prompts, tools, model choice or updates |
| Loss function | Environment rewards and feedback |
| Optimizer | Text optimization or a coding-agent loop |
| Gradient descent step | A pull request into the repository |
| Overfitting | Reward hacking or ordinary overfitting |
An optimizer such as GEPA, or simply a coding agent running in a loop, can propose changes to the agent’s behavior. The repository change plays the role of an update step; evaluation determines whether that update helped. The analogy also preserves the warning: optimizing against a reward can produce reward hacking, and repeated improvement on familiar tasks can still overfit.
Traditional machine learning accumulated libraries, platforms, and companies around these operations. Harbor and other emerging frameworks are beginning to provide corresponding infrastructure for agents. Shaw frames the opportunity expansively: he describes Databricks as worth hundreds of billions of dollars and suggests that agent users and builders may already outnumber historical ML practitioners by orders of magnitude. These are his market estimates; the concrete engineering requirement is empirical evaluation. How do you actually measure an agent doing work?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an environment to an evaluation result
An environment needs three things:
- Instruction: What the agent is supposed to accomplish.
- Sandbox: A virtual computer in which it can attempt the work.
- Verifier: A way to decide whether the requested outcome was achieved, using programmatic tests, a rubric, or another agent inspecting the result.
A time limit or another stopping condition bounds the attempt. Harbor packages the specification in a directory so that it can move between people and systems. Shaw uses environment for this whole bundle; Harbor’s documentation calls the bundle a task and distinguishes it from the execution environment. The directory format makes the instruction, starting conditions, and grading procedure portable together.
A rollout turns that specification into an observed attempt:
- Start the task’s sandbox and give the agent access to it.
- Run the agent, either outside the sandbox while executing commands into it, or inside the sandbox itself.
- Continue until a stopping condition, retaining the agent’s trajectory.
- Give the resulting sandbox state to the verifier for grading.
- Stop the sandbox and retain the verifier’s reward or set of rewards.
- Aggregate rewards across the dataset’s rollouts to produce the agent’s evaluation result.
The trajectory records the attempt; the reward grades its outcome. Keeping both matters because a score tells you whether an attempt succeeded, while the trajectory gives you material for understanding and improving the behavior that produced it.
This lifecycle is a minimal version of the execution machinery. Harbor also supports multistep rollouts, verification in a separate sandbox, artifact collection, and simulated users. Those variations change what happens inside the loop without removing its central sequence: attempt the task, preserve evidence, and assess the result.
Harbor’s open-source framework combines three roles: an environment format, a parallel rollout runner, and a registry of training and evaluation environment sets. Its intended execution contract is any agent, any model, any sandbox, and any task. At the time of the talk, Shaw estimates that the registry contains 300–400 evaluation sets and says two or three Harbor-based benchmarks appeared that day. The purpose of a shared format is to make these tasks interoperable, increasing the speed at which people can exchange and use evaluation or training data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Own the eval before choosing the model
Shaw’s proposed audience is every company that uses computers: if a workflow happens on a computer, investigate whether an agent can automate part or all of it. He cites Satya Nadella’s appearance on the Applied Compute podcast as expressing the right order of operations: identify the outcome that matters, establish how to grade it, and then welcome all models.
Build the eval first, then optimize against it. Owning the grading procedure lets a company compare models on its own work rather than relying on a brand, a public benchmark, or another organization’s priorities. It also makes the cost–performance trade-off explicit: the most capable option is not automatically the best choice at every price, and the cheapest option is useful only if it achieves the required outcome.
Shaw identifies four targets from the evaluations he sees people building:
- Agents building your product. Internal repositories supply tasks that reflect the work your engineers actually do. Ramp SWE-Bench is his example of an internal-codebase benchmark that supports informed agent and model selection instead of maximizing token spend.
- Agents using your product. Evaluate whether agents can accomplish tasks through the product, then improve the product’s usability for them. Headless access becomes part of the product surface; Shaw points to HubSpot and Stripe as examples of developer-oriented products.
- Agents powering product features. Measure the behavior of agent capabilities delivered to customers.
- Agents automating internal processes. Grade the work agents perform inside the business.
A company can need several of these evaluations at once. They answer different questions: whether an agent can change the software, operate it, deliver one of its features, or carry out the surrounding business work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Running evaluations in parallel
A person—or an agent—can start an evaluation through harbor run. Shaw’s recorded example uses Terminal-Bench 2.1 with Codex and GPT-5.5, configured for 64 parallel rollouts. He then demonstrates a launch command intended to move orchestration to Harbor’s servers, describing that hosted capability as rolling out soon. This is the talk’s interface and configuration; current dataset syntax differs, so the current evaluation guide is the appropriate reference for a new run.
Hosted execution enables the workflow Shaw wants: submit a large job, leave it running, and inspect the results later. His example of waking up to ten thousand completed rollouts is an illustration of that workflow, not measured overnight throughput. The demonstrated evaluation does not finish within the video. That wait is itself a reason to parallelize: long individual attempts make sequential execution a slow way to test an agent change.
The tasks themselves can also become a product. Experts specify work they want automated and sell those tasks to labs training models on the corresponding capabilities. Shaw describes the market around Harbor data and other data types together as probably worth multiple billions of dollars. The mechanism is straightforward: a portable task gives a buyer a repeatable situation in which to train or measure a desired skill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production work uses the same machinery
Running an agent with a model in a sandbox on a task does not require evaluation or training to be the end goal. The output can be useful production work. Shaw calls this agentic MapReduce: distribute inputs across many sandboxed agents, run them in parallel, and aggregate their results, potentially with another agent.
The inputs might be trajectories to inspect for reward hacking, receipts to process for reimbursements, Obsidian notes to search semantically, or pull requests to examine while answering a question. In each case, the map stage handles separate pieces of work and the reduce stage combines their findings. Shaw describes this as an emergent use of Harbor rather than its original purpose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn session corrections into the next evaluation
Harbor Exec makes that pattern concrete. Shaw says the feature was built for this usage and would go away soon, without naming a replacement. In the recorded harbor exec demonstration, the inputs are his Codex sessions from the June twenties. The map instruction asks an agent to find occasions when he corrected the coding agent and write an analysis.json containing the mistake, reason, and correction.
The reduce instruction asks for recurring mistakes and failure categories to be summarized in a concise feedback.md. The two stages deliberately use different agents or models. Cursor CLI handles the map work because Shaw values its speed and cost there; he names Fable 5 for reduction because he cares about the quality of the combined analysis. Modal supplies the cloud sandboxes.
Shaw limits the demonstration to 32 input sessions; that is an input limit, not a stated concurrency setting. He suggests that a larger run could process a thousand or ten thousand sessions. The demo is orchestrated locally on his computer while the agents execute in cloud sandboxes. After the parallel Cursor rollouts, the reduce step produces the recurring-mistakes report.
That report closes a development loop. Recurring mistakes can become the next set of Harbor tasks, giving Shaw a targeted way to evaluate a different agent or improve a skill. Historical user corrections are therefore more than a record of frustration: they identify behaviors worth turning into repeatable tests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rollout outputs become optimization inputs
The same preserved outputs support several forms of improvement:
| Method | Rollout material used |
|---|---|
| Supervised fine-tuning | Trajectories |
| Reinforcement learning | Token trajectories and rewards |
| Text-based or evolutionary optimization | Trajectory text and evaluation feedback |
Shaw points to Tinker’s Harbor integration for reinforcement learning. GEPA and related evolutionary methods instead use textual evidence to search for improvements to a harness or skill. The execution loop stays recognizable; what changes is the process that consumes its outputs and proposes the next update.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A shared format across different kinds of work
The closing examples show how widely the task-and-rollout structure can apply. Shaw introduces Frontier Suite as an ultra-long-horizon software-engineering benchmark, then Handshake’s BankerToolBench for investment banking. He cites swyx’s statement that Cognition migrated all its evaluations to Harbor, followed by RuneBench, which measures agents playing RuneScape.
Shaw also names Scale’s Atlas Suite and reports that Poolside uses Harbor for all its model-training evaluations. The Scale Labs announcement shown in the recording describes refreshed SWE-Atlas Codebase QnA leaderboard results, use of Harbor for all models, and a dataset available in Harbor format. These examples put the same execution infrastructure beneath both benchmark publication and repeated model-development evaluations.
The survey then moves to systems that improve agents: Kevin Gu’s AutoAgent takes a Harbor eval as the input to a self-optimizing agent, and AfterQuery used Harbor to post-train a model. Shaw names Cognition’s Frontier Code benchmark, LangChain’s integration of Deep Agents and its sandboxes, and Snorkel’s Senior SWE-bench, which he describes as released that day to measure work under ambiguity with behavioral feedback.
Harbor is also growing its team. Shaw closes by inviting interested contributors to contact him through Twitter DMs and directing viewers to the documentation link on his profile. The invitation follows the range of work just shown: building tasks, running agents, understanding failures, and feeding the resulting evidence into the next improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source framework with installation instructions and examples for running sandboxed agent evaluations.
Introduces prompt optimization through natural-language reflection on execution trajectories and evaluation feedback.
Recipe for training and evaluating models on Harbor tasks using sandboxed execution and test-based rewards.
Methodology for Ramp's private benchmark built from reviewed production engineering work, with correctness, runtime, and cost evaluation.
Investment banking benchmark requiring financial models, presentations, and reports graded against expert-authored rubrics.
Further reading
- Harbor Core ConceptsDocumentation
Definitions of tasks, datasets, agents, container environments, trials, and jobs.
Updates since the talk
Current commands for running datasets and inspecting rewards, trajectories, timing, and artifacts.
Read the complete timestamped transcript
- 0:00
[upbeat music] [audience applauding] Awesome.
- 0:16
Thank you so much. Uh, so yeah, like he said, my name's Alex Shaw. I work at Laude Institute. Um, and I'll be speaking today about Harbor, which is an agent evaluation and, uh, RL environment framework.
- 0:31
And the title of my talk is Everything is a Rollout, and I think you'll see as I get into it why we titled the talk that way. Um, but first, I want everybody to come travel back in time with me to the year 2018, so eight, eight years ago.
- 0:48
Um, and we're gonna talk about some of the things that were going on in 2018. So, uh, probably you're, you're going to see Avengers: Infinity War later today. The second one is just getting released.
- 1:01
Uh, GPT-1 was, was just released, and you probably didn't even notice, although maybe some people did. Um, musically, this random startup was about to rebrand to a product called TikTok,
- 1:17
and you might have just learned about AirPods when you s-saw somebody walking around with headphones that had no cord. Um, so what about, what about software engineering? What did software engineering look like in 2018?
- 1:30
Uh, we're gonna read this tweet from [REDACTED:username], [REDACTED:username] with two, two As, um, about what it was like to write code in 2018. So it says, "It's 2018 and your coworker just sent you a 400-line pull request.
- 1:43
You get a cup of coffee and sit down to review it. It's beautiful. Elegant micro refactors, crispy method names. You catch a few things, but that's okay. It's part of the dance.
- 1:53
They didn't consider extensibility on part of their API. Here's a comment, buddy." And this is actually just part of the tweet, so it keeps going. You should look it up if you want.
- 2:02
And it's obviously written humorously, um, but the thing is, it does feel a little bit nostalgic, um, just like some of these other things that were going on in 2018.
- 2:15
Now, the thing is, uh, this only changed maybe 6 or 12 or 18 if you're a very early adopter months ago, um, but it, but it already feels like the distant past in some ways.
- 2:27
Uh, and I think it's time to start talking about, well, what will the history books say about software engineering? And when I say software engineering, I mean that style of 2018 software engineering.
- 2:39
Um, it will probably say... The, the, the history books will probably say a lot of things, um, but I think one thing for sure that they'll say is software engineering was when you knew what the code would do before you ran it.
- 2:52
Um, so and that brings me then to a different tweet from Francois Chollet, uh, where he says, "Agentic coding is a form of machine learning. Generated code is best treated as a black box artifact whose behavior and generalization should be managed via empirical evaluation, like with any ML model."
- 3:12
Um, and that kind of brings us to the next part in this talk, which is to compare and contrast what agent development looks like versus what more traditional software engineering development looks like, and why it demands a new set of tools to really understand what's going on and have confidence and trust.
- 3:30
Um, so here's a, here's a 2018 program right here. Uh, so you can tell already the purpose of the program is to extract phone numbers from text, and we have a regex right here that, uh, looks for the phone number.
- 3:45
And I can say with 100% confidence what will happen if I run this program one million times in a row. Um, so now let's update it to the 2026 version. [laughs]
- 4:00
So, uh, I swap out my regex and instead, obviously, I throw in my model call instead, and I say, "Extract this phone number." Um, so in some ways, this is actually a more powerful program because, uh, the regex was actually a little bit brittle.
- 4:16
It would've missed any phone number that wasn't formatted exactly like how it was specified, whereas I'm pretty confident that this program with GPT-5.5 will catch a lot of the phone numbers that are formatted weirdly.
- 4:30
Um, however, if I ran this exact program one million times, I'm not 100% confident that it will print the same thing every single time or that I know exactly what it will print.
- 4:43
Um, it probably gets it right almost every time. This is a pretty simple task. Um, but, uh, this is obviously far simpler than the things we're asking these models to do, and the uncertainty only increases as the complexity of the task increases.
- 4:59
Uh, so now let's come back to Francois's tweet, and let's update it a little bit. We'll generalize it. So he says agentic coding, and my claim is, well, just agents in general are a form of machine learning.
- 5:12
And then he says, "Generated code is best treated as a black box artifact," and I say agent performance itself is best treated as a black box artifact.
- 5:22
Uh, and that brings us to our, to our new paradigm. So, uh, we know now agent development is more similar to machine learning than it is to software engineering.
- 5:32
So what are the things that we should keep an eye out for, the, the tools that you use for machine learning and also the pitfalls of machine learning, and what are the analogs for those with agent development?
- 5:43
So we have machine learning on the left, agent building on the right. Uh, so training data, that now looks like environments. And then your test and your validation set, that looks like evals, which if you take off the mask, is also actually just environments, at least the style that we'll be talking about today.
- 6:02
Um, weights, your model weights are now skills, prompts, tools, the model, whether you're picking between models or actually updating the model yourself. Uh, your loss function looks now more like environment rewards and feedback.
- 6:18
Your backprop or optimizer is now some context, text-based optimization algorithm like JEPA or even just running a coding agent in a loop.
- 6:29
Uh, your gradient descent step looks like a pull request into your repo, and overfitting looks like reward hacking or also just, uh, overfitting. That's also possible for agent development.
- 6:42
Um, so we have these things on the left. There's, you know, products and libraries and platforms that were built for these purposes. And then for everything on the right, we're just getting started.
- 6:54
So, uh, we built Harbor to answer some of these questions. We're building it to answer more of these questions, but other people are also building interesting products and interesting frameworks to help people tighten this loop of agent development.
- 7:08
Um, and then something else to consider is that, uh, as popular as machine learning was, and it spawned some of the largest companies in the world. You have, like, Databricks, which are worth, uh, hundreds of billions of dollars.
- 7:21
Um, already the number of people using and building agents is probably magnitudes larger than the number of people that ever, uh, were doing machine learning, and that trend will, will only continue.
- 7:35
Uh, and now let's look at the last piece of Francois' tweet that I wanna call out, uh, which is, uh, these, these agent-agent performance, uh, da, da, da, should be managed via empirical evaluation.
- 7:50
Uh, and that brings us to our next question, which is, how do I actually evaluate an agent? Uh, and this is when we start to get into, uh, what, what Harbor does.
- 8:02
So the answer, as we already gave away earlier, is environments. That's your evals and your training data. And what is an environment in this case? Well, we need an instruction.
- 8:12
We need some way to tell the agent what it's supposed to do, and then we need some place for the agent to try to do this thing. And right now, all of the agents run on computers, and they do stuff on computers.
- 8:24
So we'll put it into a computer, but we'll put it into a virtual computer, so a sandbox. Um, and then we need some way of telling whether or not the agent actually did the thing that we told it to do in the sandbox within some amount of time or other stopping condition.
- 8:40
And that's your verifier, which is like some programmatic tests or rubrics, um, or agent that comes in to see what happened.
- 8:48
So in Harbor, you specify this as a file directory, and this, uh, specific directory layout has become relatively standard in a lot of the environment space. So lots of people have adopted it and use it as a way to specify environments, which is useful because then it can easily pass between hands and becomes interoperable.
- 9:09
Um, and then okay, cool. We have a bunch of data now. We've implemented a bunch of these environments. Now, how do I actually use it to start understanding what my agent can and can't do and how to make it better?
- 9:21
Um, so that's a Harbor rollout. And, uh, so what you do is you start with your tasks, and then you take it, you start up your sandbox, and then step one, you pass that sandbox to your agent.
- 9:33
So you're either running your agent outside the sandbox and executing commands into it, or you're running it inside the sandbox, and it's calling whatever its commands are as part of its program.
- 9:43
Uh, and then it runs for some amount of time until it hits a stopping condition, produces some trajectory, and we'll come back to that later because that's important. And then you pass that sandbox to the verifier, which, uh, runs some verification process.
- 9:57
And then finally, you stop the sandbox. The verifier produces a reward or a set of rewards, and then you take those, you aggregate them a bunch of-- across a bunch of rollouts in a dataset with some agent, and that becomes your eval result.
- 10:10
So this process here, which looks relatively simple, is actually extremely universal. And, uh, it's also a little bit over s- overly simplified. So Harbor by now allows for a lot of different flavors of rollouts.
- 10:23
So you can do multi-step. You can run your verification in a separate sandbox. You can collect artifacts. You can simulate a user. Um, but this is kind of like the, the, the bare bones approach, uh, that they're all built off of.
- 10:37
So what is Harbor more specifically? One, it's a format for specifying agentic environments. Two, it's an open source framework for performing rollouts in parallel using any agent with any model in any sandbox on any task, and it's a registry of popular training and eval environment sets.
- 10:55
I think we have, like, three or four hundred eval sets right now. And in fact, I think two or three benchmarks even came out today that run with Harbor.
- 11:02
Um, so we're excited about that. And in general, it is trying to be a common language for environments, so a way for people to specify things that are extremely interoperable, and it allows you to maximize data velocity and just increases progress in the industry.
- 11:17
Uh, so who needs evals? Uh, now that we understand how to make them, we understand how to use them, well, now who should actually be doing this? And the answer is every single company that uses computers.
- 11:28
And I think that's probably close to all of the companies in the world. And the reason is because if you're doing something on a computer, then you should be seeing if you can use AI to automate part or all of that process that you're currently performing on a computer, uh, because that will increase the productivity of your
- 11:46
company and therefore increase the value that your company generates. Uh, so I like this quote from Satya Nadella from the Applied Compute podcast he did last week. He says, "If you want to build an agentic system, start with the eval that matters and your ability to grade that outcome, and then say, 'I welcome all models.'" So I
- 12:07
like that last line because what he goes to say is that as soon as you have an eval, the power is now in your hands. You can consider every single model.
- 12:18
You don't have to trust brand, you don't have to trust somebody else's eval, you don't have to trust a public eval, um, and you can kind of skate the Pareto however you desire to, uh, balance that cost performance trade-off.
- 12:30
So step one, build the eval. Step two, kind of optimize against it. Um, so what will people evaluate? And, uh, I'm gonna list four things, and these four things are based off of what we actually see people using Harbor to build evals for right now.
- 12:46
Uh, so s- so the first type of eval is people evaluating how well agents build their products. So everybody that's building a software product has some internal code base or set of internal code bases.
- 12:58
Um, I think Ramp recently announced, like Ramp Swebench, which is a Swebench built off of their internal code base. And what that allows them to do is pick the coding agent or the model that performs best on their internal use cases, and they don't have to, you know, maximize token spend.
- 13:14
Instead, they can make informed and educated decisions about how they build their products with agents. The second type of eval that we see people build is how to evaluate, uh, how well agents use your product.
- 13:27
So anybody that builds a software product is probably moving towards a world where they offer some sort of headless mode. So you see some companies have obvi-- uh, have always been this way.
- 13:38
They've been developer first, like, uh, HubSpot and Stripe and things like that. And the idea is, uh, if you can make an eval to see how well agents use your product and then iterate on your product to make it more usable for agents, you're gonna get more usage, and therefore your product, uh, will become more valuable to
- 13:56
agents. And then, uh, three, evaluate how well agents power product features. And then four, evaluate how well agents automate internal processes. So depending on what type of company, company you are, uh, one or more of these might apply to you, but everybody should be considering right now how they can do one of these things.
- 14:15
Um, okay. So what are the different use cases of the Harbor rollout? So we've talked about evals. That's the most popular right now. Um, oh, oh, here we go.
- 14:26
I actually have a video even of us doing an eval in Harbor. So this is, uh, how you can run it from the command line or have your agent run it from the command line.
- 14:34
So you can see here it says, "Harbor run." We're running, in this case, Terminal-Bench 2.1 with Codex and GPT-5.5, and we're saying, "Let's run 64 in parallel." And then we actually type in the launch command right here, which is something we're rolling out soon, which allows you to launch the rollouts on Harbor servers, which means you can
- 14:56
just fire and forget and go to sleep and then come back the next day and ten thousand rollouts are done for you. And you can kind of see and hear what it looks like to look at those rollouts happening.
- 15:06
Um, but they don't complete in the time span of this video because rollouts can take a while, uh, which is another case for parallelizing as much as you possibly can to tighten that, uh, that loop and maximize your throughput.
- 15:20
Um, so there are people who create tasks and sell those as data, so there's actually, like, a multi-billion dollar market right now that exists probably around Harbor data and also other types of data.
- 15:33
Um, but it's, uh, very, uh, I guess, lucrative right now where experts can specify certain types of tasks that they would like automated and then sell those to one of the labs that's training models to, uh, improve on that capability.
- 15:48
Uh, you can also do what we call prod rollouts. So remember, Harbor is evaluate any agent with any model, uh, I guess not even evaluate. We'll just say, uh, run any agent with any model in any sandbox on any task, and that actually doesn't mean you have to do it for evaluation.
- 16:08
It also doesn't mean you have to do it for training. It could literally be that you want to do what we've been calling agentic MapReduce, which is you just wanna run a ton of agents on distributed compute, so sandboxes, in parallel, and then somehow aggregate those results, probably also intelligently.
- 16:25
So things like looking over a bunch of trajectories to detect reward hacking, or I don't know, like processing a bunch of receipts for reimbursements or searching over your Obsidian files to figure out, uh, semantically what, what-- where you wrote some note or processing a bunch of PRs to ask a question about it.
- 16:43
So we see a bunch of different use cases. This is actually an emergent use case, so we didn't build Harbor for this, but we see people doing it a lot.
- 16:50
So we actually built this feature and launched it just for this. So it's Harbor Exec. Uh, it's gonna go away kind of soon, but you can see in this scenario...
- 16:58
Actually, let me see if I can pause this. So, uh, I'm actually gonna go back a little bit before we kicked it off. So I'm saying Harbor Exec input is all of my Codex sessions from the June 20s, and then I say, "Prompt, if I corrected the agent, write an analysis.json with mistake, reason, and correction."
- 17:20
And then my reduce prompt is, "Summarize recurring mistakes and failure categories into a concise feedback.md file." And you can see in this case I'm running on Modal. I do the map step with Cursor CLI because it's cheap and fast, and then I do the reduce step with Fable 5 because I want it to have, like, an accurate
- 17:38
summary and I care about intelligence in that scenario. So, and then I'm limiting it to thirty-two because I didn't actually wanna process all of my sessions for this demo, but you could-- there's no reason you couldn't do, like, ten thousand sessions or a thousand sessions, and then maybe you have to go and understand the results more deeply.
- 17:53
But, um... Oh, shoot. But, uh, yeah, you can kind of see here, I think. So we kick this off. A bunch of these rollouts are in parallel. This one is running locally on my computer with, like, cloud sandboxing, but orchestrated locally.
- 18:09
And then you can see here- All of the cursor rollouts running right now, uh, but those will take a second to finish. And then you can see the reduce step here where Fable recommended me like recurring mistakes.
- 18:26
And like these recurring mistakes, for example, could be used to inform the next batch of Harbor tasks that I create to evaluate my agent and pick like a better agent or train up a skill or something like that.
- 18:36
So, uh, another use case of Harbor, and then we see people taking the trajectories and doing SFT, and then we also see people taking the reward or rewards, the trajectory in the form of tokens, and doing actual reinforcement learning.
- 18:51
Uh, for example, Tinker launched an integration with Harbor, and then, uh, people also do other types of optimization. So we mentioned JEPA, but any of these like evolutionary methods, you take the text feedback in the form of a trajectory and the eval, and you can do some sort of like auto hill climbing with a harness or a
- 19:09
skill. Um, so in the last couple minutes, I just wanna talk about some cool things people have built with Harbor. I'll try to breeze through this. Uh, so, uh, Frontier Suite, an ultra-long horizon software engineering benchmark built on Harbor.
- 19:24
BankerToolbench, an investment banking benchmark built by Handshake on Harbor. Um, swyx saying that his team at Cognition migrated all their evals to Harbor. This RuneBench, this is one of my favorites.
- 19:36
It's like a benchmark to measure how well agents can play RuneScape. Um, Scale launched their whole suite, Atlas Suite, on Harbor. Uh, Poolside uses Harbor to do all of their evaluations for model training.
- 19:50
Uh, Kevin Gu created AutoAgent, which is a self-optimizing agent that all you have to plug in is a Harbor eval. Uh, AfterQuery post-trained a model using Harbor. Cognition released Frontier Code recently, which is a Harbor benchmark.
- 20:05
And LangChain just integrated deep agents and their sandboxes into Harbor. Uh, and then actually just today, Snorkel released Senior SWE Bench, which is a benchmark for measuring how well agents can function under ambiguity, uh, with behavioral feedback.
- 20:22
So, uh, that's kind of a brief overview of Harbor and the different things you can do with it. I would encourage everybody here to check it out. And then this is a link to my Twitter, and the reason I'm linking my Twitter, usually I link Harbor docs, but we're actually actively trying to grow the Harbor team right
- 20:38
now. So if you want to get involved, then shoot me a DM on Twitter. And if you just wanna see the docs, then click the docs link on my Twitter, and that's the best way to get there.
- 20:48
Uh, so thank you very much. [audience applauding] [outro music]