# How to Get Frontier Coding Results at 70% Lower Cost - Nouha Dziri, Cohere

Nouha Dziri, Cohere

[Talk page](https://ai.engineer/talks/maitMe9HCvw-get-frontier-coding-results-at-70-lower) · [Video](https://www.youtube.com/watch?v=maitMe9HCvw)

## On ProgramBench, 16 Cheaper Planners Came Close to an All-Frontier Team at 70% Lower Cost

In Cohere’s reverse-engineering experiments, a team that spent its worker budget probing an executable before writing code came close to the performance of an all-frontier team at 70% lower cost, while a second benchmark rewarded a different division of labor.

### Divide a long coding task before the context window fills

[0:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=0s)

A coding agent can spend hours navigating a repository, editing files, running tests and debugging its own changes. Nouha Dziri, a senior research scientist at Cohere, starts with two familiar ways to give that agent more compute at inference time. Let it reason for longer, and it can pursue a problem further, at the cost of a longer wait and an increasingly full context window. Sample several independent attempts in parallel, and you get multiple candidate solutions to aggregate. Each attempt still has to understand and solve the entire task.

A team offers a third possibility. One agent keeps responsibility for the task and assigns parts of it to workers. Each worker starts with a fresh context. It can afford to read a file closely, try several tests or dig through a log, then send back the finding the lead needs. The lead can use evidence from several investigations without carrying every intermediate observation in its own context. If the investigations are independent, they can also run at the same time.

The qualification matters. A fresh context is valuable when it holds work someone else can use. Five agents each reconstructing the same program can consume five budgets and leave the original integration problem intact. Dziri and her colleagues wanted to measure both effectiveness, meaning how much more of the task the team solves, and efficiency, meaning what it costs to obtain that improvement. Lower latency was a further possibility to investigate, conditional on how the agents divided the work.

### A group of capable agents built several versions of the program

[2:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=150s)

The researchers first gave a group of agents a difficult coding task to see how they would organize themselves. A reasonable expectation was that they would identify pieces of the problem, form groups and build one solution. Instead, individual agents went off and built their own versions. Asking them to communicate produced some discussion, including exchanges of little consequence to the implementation. They still duplicated work, interfered with one another and tended to direct their messages back to a lead.

That experience turned team design into a set of experimental choices. Who owns the final program? Can workers contact one another directly? How many should the lead start, and which model should fill each role? There is also a timing decision with consequences for all the others. A worker who investigates requirements before coding begins has a different job from one who writes part of the program or checks the finished implementation.

Cohere framed these choices as compute allocation relative to a solo agent. The efficiency measure was the gain in benchmark reward per additional dollar spent over the solo baseline. Both differences matter. A team can post the highest score while offering a poor return on the last dollars spent to get there. The researchers therefore varied the stage of delegation, the worker model, the number of workers and the lead model, asking where another dollar of inference produced the most improvement.

### One shared workspace, separate contexts and a lead that owns submission

[4:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=299s)

To compare those decisions, the researchers built a deliberately small, top-down harness. The lead receives the complete task. It can decide dynamically how many workers to create and what each should do. Every agent has a Bash tool and works in the same workspace, but a worker receives only its assigned subtask. Sharing the files gives workers a way to contribute to one program. It does not, by itself, tell them who else is working on a feature or how their assignment fits into the whole.

Delegation is an explicit tool action. The lead can request an investigation, hand off implementation or ask a worker to verify a result. Workers report back, and the lead decides how to combine their contributions. It can resume a worker, point out an error and ask for a correction, or terminate the worker. Workers can also contact one another. Exposing those decisions in the harness let the researchers change one part of the arrangement while holding the others fixed.

They implemented two ways to handle the interval between an assignment and its result. A blocking lead hands out a batch of subtasks and waits until every worker reports back. The workers can run in parallel with one another throughout that wait. An asynchronous lead can continue its own work, receive reports as they arrive, supervise workers and send feedback. The latter gives the lead more freedom, including responsibility for recognizing when its own next decision depends on a result it has yet to receive.

![Two timelines show a lead delegating to five workers. In the upper timeline, the lead waits for each group of workers to report back. In the lower timeline, the lead keeps working as reports arrive and reaches DONE earlier.](https://ai.engineer/api/talk-essay-media/15fe85362aed0ddd4e19ee91b40518d8e2fe742945d628c57b7f10480e8b4788)

A blocking lead and an asynchronous lead

### ProgramBench makes the team discover what it has to build

[6:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=419s)

A ProgramBench task starts in an otherwise empty workspace with documentation and the original program’s executable. The agents can run the executable but cannot read its implementation. They have to probe it to discover its interface, observable behavior and edge cases, then write a replacement that behaves the same way. A plausible implementation can therefore fail even when the code does exactly what its author intended. The author may have inferred the wrong requirement from the original.

![A flowchart goes from a binary and documentation through probing, rebuilding and scoring. Below, an agent probes dsq with sample commands. A grid of hidden tests beside the reconstructed code shows 60%.](https://ai.engineer/api/talk-essay-media/99da5cfc1b6e6d34ba0e4003097086d1cc29c0e08cd7a19539280c42d478eed7)

ProgramBench: rebuild a program from a binary

Dziri cited programs on the scale of SQLite and the PHP interpreter. For a sense of difficulty, she pointed to the FFmpeg task, where the best model on the leaderboard she described passed only 10% of the tests. That figure describes the challenge of reproducing FFmpeg. The team experiments measured a separate quantity, the change in reward when the researchers gave a lead workers on ProgramBench.

With minimal prompting, the blocking team beat the solo agent by almost ten points. Teams cost roughly four to seven times as much as a solo run. The blocking lead scored better than the asynchronous lead, and it also cost more and took longer to finish. Waiting improved the submitted program enough to show up in reward, even though keeping the lead busy looked like the better use of elapsed time.

### An asynchronous lead could submit while its workers were still exploring

[8:18](https://www.youtube.com/watch?v=maitMe9HCvw&t=498s)

The runs exposed a specific failure. The lead would ask workers to investigate the executable, make its own guess about the behavior under investigation and begin implementing that guess. By the time the workers reported what they had found, the lead had moved on. Sometimes it ignored the evidence. Sometimes it went back and redid its work. In the clearest cases, it declared the task complete and submitted while workers were still running. Their investigations cost money and had no opportunity to affect the submitted program.

The blocking harness removed that particular decision from the lead. A round of delegation had to finish before the lead could consolidate the reports, build on the findings and submit. That explains the advantage under the initial instructions. Those instructions described the task, the ability to create workers and the tools available. The researchers deliberately gave little guidance about coordination because they wanted to see how much of it the models could work out on their own.

They then tried detailed instructions covering planning, implementation and verification, abbreviated PIV. The prompt directed the lead to distribute exploration and planning, delegate implementation, keep edits within assigned locations and create adversarial workers to check the result. It gave the lead guidance about when to wait. Both team variants improved, and the asynchronous lead caught up with the blocking one. The ability to continue working had been useful only to the extent that the lead could tell which work was safe to start.

Better instructions also raised the solo score by almost six points. That makes the well-prompted solo agent an important control. Comparing a heavily guided team only with a minimally guided individual would credit delegation for gains available from instructions alone. Dziri also stressed how much time the researchers spent tuning those instructions. Even after that work, the guided teams cost six to seven times as much as the solo agent.

### Three experiments isolate planning, implementation and verification

[11:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=688s)

The researchers kept the lead fixed and gave it workers at only one stage at a time. In the planning condition, the lead assigned different parts of the executable to workers for exploration. Each worker probed its part and returned findings and a plan. The lead used those reports to write the replacement and verify it. Here, planning meant empirical work against a running program. Its output was an account of what the implementation needed to do.

In the implementation condition, the lead explored the executable and developed the plan on its own. It split that plan into assignments, gave the assignments to workers and received code back. The lead retained responsibility for verification. This arrangement distributed the writing, but every worker began with the requirements a single investigator had been able to discover.

In the verification condition, the lead both planned and implemented the replacement. Workers entered at the end to check the work and report their findings. The lead then decided whether to submit. All three conditions gave workers consequential jobs. What changed was the point at which several fresh contexts became available and the kind of evidence or code the lead received in return.

On ProgramBench, planning-only delegation reached nearly the reward of the full PIV team at about half its cost. Dziri put the planning advantage at roughly 1.6 times the reward per added dollar compared with spending on the other stages. Increasing the number of implementation or verification workers drove cost up while performance stayed around the solo level. The limiting resource on this task was an accurate enough account of the binary’s behavior to guide the first implementation.

![The left plot shows planning-only and all-stage delegation reaching similar rewards at roughly $250 and $500 in total inference cost. The right chart compares reward points per $100. Planning has the tallest bars for both blocking (3.8) and async push (3.4).](https://ai.engineer/api/talk-essay-media/afb1838fe9c035977644ecbc4f0d2212091daa427c41dc37d91666c4f7daec68)

Planning-only delegation matches PIV's reward at half the cost

### The delegation traces show when 830 extra calls stopped paying off

[14:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=868s)

Consider the discovery work from the perspective of one context window. A solo investigator accumulates the details of each probe while deciding what to investigate next. It can spend a great deal of context on one part of the program and still miss an edge case elsewhere. Several planners can each pursue a different part in depth. Their compact reports let the lead retain the observations needed for implementation across a wider range of behavior.

The researchers checked whether the agents actually used their workers this way. In their delegation plot, each row represented one run, time advanced from left to right, and each dot marked the lead sending work to a worker. Yellow identified planning, blue implementation and green verification. With planning guidance, the dots clustered in one early burst, before any code was written. The workers explored, the reports came back and the lead built the program.

![Four ProgramBench timelines show when the lead delegates under PIV, planning, implementation, and verification guidance. Dots cluster at the start under planning guidance and recur later under PIV guidance. Adjacent bars show the share of calls in each phase.](https://ai.engineer/api/talk-essay-media/b9da81aec4dd35944eac29beb38a07b9ee923d2dec53932b2bb952d86e27993b)

Why planning wins: one early burst of discovery

The full PIV traces began with much the same burst, but the leads kept handing out work during implementation and verification. Across the traces Dziri showed, that added 830 delegation calls. She put the cost of the full PIV configuration at roughly $500, about twice the cost of planning-only delegation, for essentially the same reward. Measured per delegation call, planning alone delivered 2.4 times as much reward as full PIV.

### Implementation workers inherited the lead’s gaps and overlapped when given more context

[16:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=987s)

In the implementation-only runs, the lead had to discover the behavior of the entire binary before dividing the coding work. Anything it missed became a gap in the plan. A worker then received a narrow slice, with limited visibility into the larger task. It could make a local implementation error, but it could also carry out its assignment faithfully and reproduce the lead’s misunderstanding. Adding writers downstream did little to improve the observations on which their assignments depended.

The researchers tried giving workers more context. That introduced a different failure. Once workers could see neighboring features, some expanded their assignments to build additional functionality. Several agents could then work on the same thing in the shared workspace. They had the means to contact one another about scope and duplicated effort, yet most communication went back to the lead. A worker could keep editing without learning that another agent was already addressing the same problem.

The cost surfaced at integration. The lead had to recognize the overlap, determine where the implementation was incomplete and stitch the contributions together. When reports accumulated, leads sometimes struggled to communicate the problem back to the worker who could fix it. The resulting runs took longer, cost more and earned less reward. Parallel writing had moved work onto the agent responsible for assembling the finished program.

### Commit0 supplies a skeleton and tests, shifting the payoff to implementation

[18:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1107s)

Commit0 changed what the agents knew at the beginning. They received a code skeleton, documentation and unit tests visible to the workers. Much of the plan was already expressed in those materials. On ProgramBench, the team had to discover the behavior it was trying to reproduce before it could sensibly divide up the writing. On Commit0, workers could start from supplied implementation targets and known tests.

Here the largest improvement came when the researchers doubled the number of agents in the implementation stage. Dziri gave the direction of the result and referred listeners to the paper’s appendix for the figures. The two benchmarks support a practical way to set up a team experiment. Inspect what the lead actually has to hand off. When the intended behavior is obscure, several independent investigations can improve every subsequent coding decision. When a skeleton and tests already specify the work, extra implementation capacity can address the remaining difficulty directly.

### Price the planners separately from the model that must use their findings

[19:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1167s)

Having narrowed ProgramBench delegation to planning, the researchers changed the models filling the two roles. They used three GPT 5.6 models, Luna, Terra and Sol, in increasing order of capability and cost. The reference planning team used Sol for the lead and its workers. The lead chose its own team size and spawned about five workers on average. This was already a more economical reference point than paying Sol workers throughout PIV.

Keeping Sol in charge and replacing its planning workers with Luna cut cost by about 60% relative to the all-Sol planning setup, with a small loss in performance. A Luna planner could pursue a bounded investigation and return what it learned. Sol still decided which investigations to commission, interpreted the results together and took responsibility for implementation and verification. Only the delegated portion of that arrangement had moved to the cheaper model.

The researchers also tried reducing the capability of the lead. Across solo runs, both performance and cost rose as they moved from Luna to Terra to Sol. Homogeneous teams, such as Luna with Luna workers and Terra with Terra workers, showed a similar capability ordering. The mixed teams revealed how much the assignment of roles mattered. Sol with Luna workers approached Sol with Sol workers. Terra with Luna workers scored 51%, trailing the 57% that Sol achieved working alone.

![A chart compares solo models, same-model teams and teams with Luna workers on a logarithmic cost axis. Sol + Luna and Sol + Sol both plot near 64%, with Sol + Luna farther left at a lower cost.](https://ai.engineer/api/talk-essay-media/9a7581e960b3e9c3924f1df866a8050a222ff82fdc9a3a73faefcc7b0069e4bd)

Can the lead be cheap?

Dziri summed up the role split this way: “The lead sets the ceiling and the workers set the price.” She connected it to Meta’s work on meta-reasoning, where a better controller got more out of the same workers. In the coding harness, the lead chooses which parts of the binary to probe and decides how the findings fit together in one implementation. Luna workers could make those investigations cheaper. With Terra directing them, the team still finished six points behind Sol working alone.

### Sixteen Luna planners cost 38% less than all-Sol planning and 70% less than full PIV

[21:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1317s)

Until this point, team size was itself a decision made by the lead. To separate the effect of worker count from the quality of that decision, the researchers fixed Sol as lead, chose Luna for the planners and imposed team sizes ranging from two to sixteen. This let them spend more of the planning budget on independent probes while preserving the model that would combine the reports.

At sixteen planners, the mixed team cost 38% less than the all-Sol planning team whose lead had chosen about five workers on average. The headline 70% saving uses a more expensive comparison, the all-frontier team that delegated across planning, implementation and verification. The sixteen-planner team approached that team’s performance by changing two things together. It moved exploration to cheaper workers and stopped paying for delegation after exploration finished.

![Two charts compare worker count and inference cost. With a Sol lead, increasing cheap planning workers from 2 to 16 raises the plotted result from just under 60% to nearly 62%. The 16-worker team costs 38% less than the all-Sol planning team and 70% less than the all-Sol team used at all stages.](https://ai.engineer/api/talk-essay-media/a90a309661748695c03c4a3a9a6d8b7e780be66a8e9887d88cc6bb364c781d2b)

Scale the team where the bottleneck is

Sixteen was the largest configuration the researchers could afford to test. The observed gains gave them reason to investigate larger planning teams, but the experiment did not establish where the gains would level off. They also increased team size in the implementation and verification variants. More workers there still failed to help. Worker count had different effects even within the same benchmark, depending on what each additional worker could contribute.

### Most messages went to the lead, and half of Luna’s failed at the interface

[23:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1407s)

Every agent had a send-message tool that could reach any other agent in the sandbox. The researchers examined where messages went, whether the harness accepted them, how much text agents sent and whether enabling the tool changed reward. Despite direct peer access, 84% of messages went from a worker to the lead. Only 4% went between workers, averaging less than one peer message per task. Leads steered workers more often than workers consulted one another. The broad routing pattern persisted across model families.

The models differed substantially in how they used the interface. About half of Luna’s messages were malformed and rejected by the harness, so the intended recipient never received them. Sol produced roughly three to four times as much message text as Luna. Under full PIV guidance, stronger teams generally sent more messages overall, including more direction from leads to workers. Counting attempted sends alone would obscure both delivery failures and the difference in how much information the models tried to transmit.

Enabling messaging moved the Sol and Terra scores only within the noise. Luna teams lost four to seven points. Many delivered messages were status reports to a lead that would receive the worker’s result through the ordinary delegation mechanism anyway. That gave the lead additional text to process without necessarily changing an assignment, resolving a conflict or improving the submission.

Dziri cautioned that the experiment gave workers little reason or opportunity to find a useful peer. The lead deliberately tried to make assignments independent. Workers knew their own assignments and had a way to send a message, but they lacked visibility into which other agents existed in the sandbox and what those agents were doing. A worker seeking help, or beginning to overlap another worker’s implementation, first had to discover whom to contact. Whether better visibility would make peer exchanges worthwhile remains open.

### Models that delegated equally often earned different returns

[27:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1633s)

The researchers ran further model comparisons with the same harness and identical tools. GPT 5.6 teams delegated frequently and gained up to seven points in reward. Dziri also reported a Claude result with a gain of almost six points. Within the Claude results, Opus 4.7 rarely delegated and gained little. Haiku and Gemini presented a different case. They delegated about as often as the GPT teams yet saw no gain.

An assignment count can identify a lead that almost never asks for help. It cannot establish that a frequent delegator benefits from its workers. To see that, follow an assignment to its returned result and then to the lead’s next decision. On ProgramBench, a probe delivered before implementation could determine which behavior the lead built. The same probe delivered after submission could do nothing for the score. Both would count as a delegation.

### Beyond sixteen workers, team design and supervision remain open

[28:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1723s)

The experiments reached sixteen workers in a hierarchy with one lead. Dziri closed by asking how teams of thousands might organize, communicate and allocate compute. One possibility is to give agents a task, a budget and a deadline, then have them design the team themselves. That would require a capability the researchers still had to supply through prompt tuning, namely recognizing what kind of coordination a particular task needs.

Communication becomes a different research problem once a project outgrows one lead’s ability to coordinate it. The observed 84% of messages reporting upward and 4% passing between peers describe a small hierarchy performing deliberately separated assignments. Dziri left open whether effective peer coordination would emerge with scale, require different ways for workers to discover one another or need to be trained. Detailed, task-specific instructions helped the asynchronous lead here, but writing a new coordination prompt for every task would demand considerable human effort.

She also raised security and alignment as open questions for teams able to operate around the clock without supervision. She referred to a Hugging Face incident without describing its mechanics. Her concern was how to secure systems and infrastructure as autonomous operation extends from individual agents to whole teams. Dziri closed by crediting Kinan and Daniel for their work on the project.

## Key takeaways

- Establish a well-prompted solo baseline before pricing delegation. Better instructions alone raised the ProgramBench solo score by almost six points.
- Test delegation one stage at a time. ProgramBench rewarded parallel discovery of an executable’s behavior. Commit0, with a skeleton and known tests, rewarded additional implementers.
- Make the lead wait for findings that its next decision depends on. Inspect whether each returned result changed the plan, code or decision to submit.
- Compare model mixes against an explicit baseline. Sol with sixteen Luna planners cost 38% less than all-Sol planning and 70% less than all-frontier delegation across all three stages.

## Slides

### Single long-running agent: Context bloats!

[0:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=57s) [A question mark sits above an agent beside a context window filled with gray blocks for logs, commands, and errors. The task accounts for 0% of the context.](https://assets.ai.engineer/sites/aiecode2025/sha256/e44fa920a9575580be866c0b83c6a3e080ea97be07fda512207454750be6654c.webp)

Single long-running agent: Context bloats!

?
agent

context window
task = 0% of the context
test log   diff   ls   error   grep   read file   traceback   diff   diff

### Lead task with five separate subtask contexts

[1:12](https://www.youtube.com/watch?v=maitMe9HCvw&t=72s) [The lead’s context contains a single orange task block. Five rows below each start with a subtask and contain separate sequences of commands and outputs. The speaker inset covers the end of the percentage label.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac28f7c9c67ed012a6ab78eb25508673898d09504c150c8d2a62ba524b07b444.webp)

lead
lead context
task = 100% of the
task

subtask   grep   pytest   traceback   read file   diff   ls   ls
subtask   read file
subtask   pytest   ls   ls   search   search   ls
subtask   grep   diff   test log   error   ls
subtask   error   diff   error   read file

### Task at 38% of the lead context

[1:16](https://www.youtube.com/watch?v=maitMe9HCvw&t=76s) [The lead’s context contains an orange task block followed by five smaller dark blocks. All five context rows below are empty. The speaker inset covers the end of the percentage label.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb4f6a83a03ea80bee8761a9db3e8dbd17786dbc3aee4bc4a4d817eafda0502f.webp)

lead
lead context
task = 38% of the
task

### A new axis for scaling inference-time compute

[1:32](https://www.youtube.com/watch?v=maitMe9HCvw&t=92s) [Three diagrams compare ways to use inference-time compute. One long bar represents longer reasoning, four parallel bars represent independent attempts, and a branching network represents agents splitting a task.](https://assets.ai.engineer/sites/aiecode2025/sha256/62bd576721908a71764ec59ef08c3374ad6edd3f35b3d16965d891fe1176ac3f.webp)

A new axis for scaling inference-time compute

Think longer
one agent, a longer chain of
reasoning
Wei et al., 2022 . OpenAI (o1), 2024
Snell et al., 2024 . DeepSeek-R1, 2025

Sample more
many independent
attempts in parallel
Wang et al., 2023 . Brown et al., 2023

Work as a team
agents split the task and coordinate

### Relative cost of three agent team structures

[1:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=103s) [An orange root connects to two nodes above one dollar sign. A second root connects to five nodes above two dollar signs. A third root connects to three nodes with four children each, above five dollar signs.](https://assets.ai.engineer/sites/aiecode2025/sha256/2348bf5a30e3c2a8c3c972f184827dfd3d0da5d63bdb9721ff64a7bf5d44fc74.webp)

$
$$
$$$$$

### Team design

[4:07](https://www.youtube.com/watch?v=maitMe9HCvw&t=247s) [Five numbered questions cover team organization, communication, team size, model roles, and which software engineering stage should receive more agents. The fifth question is centered below the first four.](https://assets.ai.engineer/sites/aiecode2025/sha256/6386d50706cbb84eb2e7cf1306b45600aa1f25cf6d2a4512cc6f6c085d803169.webp)

Team design

1   Organization
How do you organize the team?

2   Communication
How should agents communicate?

3   Team size
How big should the team be?

4   Models
Which models should play which role?

5   Stage
Which stage of software engineering should
we allocate more agents?

### Team design as a compute-allocation problem

[4:20](https://www.youtube.com/watch?v=maitMe9HCvw&t=260s) [Four variables define a team configuration on the left. A blue panel compares the team’s reward and cost with a solo agent’s. Three boxes below show the order of choices: stage, models, then size.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5e0ee7d82469c4f607d1ff7d769386e7cb94587ba15f69171139f1916768c85.webp)

Team design as a compute-allocation problem

A team configuration
c = (s, w, n, ℓ)
s   Stage the workers are directed to: P, I, V or PIV
w   Worker model
n   Number of workers
ℓ   Lead model

Ideal: maximize reward R(c) s.t. cost C(c) ≤ B, across budgets, i.e. the
reward–cost Pareto frontier. Infeasible exactly, so we ask a marginal
question.

Efficiency: reward per added dollar
η_c = (R_c − R_solo) / (C_c − C_solo)
Reward points per $100 over the solo agent. Higher is
better; negative means added compute hurts.

Tracing the frontier, one choice at a time
1   Stage
2   Models
3   Size

### Recursive hierarchy in the multi-agent harness

[5:04](https://www.youtube.com/watch?v=maitMe9HCvw&t=304s) [An orange root node branches to three outlined nodes. Each of those branches to three smaller nodes, giving nine nodes at the bottom.](https://assets.ai.engineer/sites/aiecode2025/sha256/cb7a80b60d6a524594a5c54d107a05fa2f9a9e305342511651d14d0d9c40d335.webp)

Our multi-agent harness: hierarchical

Recursive hierarchy

### A team of coding agents at work

[5:56](https://www.youtube.com/watch?v=maitMe9HCvw&t=356s) [The lead’s delegation log connects to five worker panels, all marked idle. The shared workspace shows shell commands for dsq and Python code in three files.](https://assets.ai.engineer/sites/aiecode2025/sha256/31fc0c2e380a9fb1dacf6bdf533629a4aa80c810185493e7d637e144845f3d36.webp)

Our multi-agent harness: hierarchical
A team of coding agents at work

LEAD
lead
bash
delegate_agent
wait_for_updates
list_agents
send_message

DELEGATIONS
plan        new w1
plan        new w2
plan        new w3
plan        new w4
implement   new w5
implement   resume w1
verify      resume w2
implement   resume w5
verify      resume w2

WORKERS
worker-1   idle
context
worker-2   idle
context
worker-3   idle
context
worker-4   idle
context
worker-5   idle
context

SHARED WORKSPACE
dsq
$ diff want.json got.json
$ python dsq.py -s csv < users.txt
$ python dsq.py users.csv
$ diff want.json got.json

dsq.py
1   def main(argv=None):
2       args = parse_args(argv)
3       tables = [load(p, args.format) for p in args.files]
4       if args.schema:
5           return print_schema(tables)
6       write_json(run(args.query, tables), args.pretty)

formats.py
1   def load(path, fmt=None):
2       fmt = fmt or path.rsplit(".", 1)[-1]
3       if fmt == "csv":
4           return list(csv.DictReader(open(path)))
5       if fmt == "logfmt":
6           return [parse_logfmt(l) for l in open(path)]

query.py
1   def run(sql, tables):
2       db = sqlite3.connect(":memory:")
3       for i, rows in enumerate(tables):
4           create_table(db, f"t{i}", infer_types(rows))
5       cur = db.execute(rewrite_tables(sql))
6       return [dict(zip(columns(cur), r)) for r in cur]

### A blocking lead and an asynchronous lead

[6:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=403s) [Two timelines show a lead delegating to five workers. In the upper timeline, the lead waits for each group of workers to report back. In the lower timeline, the lead keeps working as reports arrive and reaches DONE earlier.](https://assets.ai.engineer/sites/aiecode2025/sha256/15fe85362aed0ddd4e19ee91b40518d8e2fe742945d628c57b7f10480e8b4788.webp)

A blocking lead and an asynchronous lead

BLOCKING LEAD
hands out subtasks, then waits
lead   DONE
worker 1
worker 2
worker 3
worker 4
worker 5

ASYNCHRONOUS LEAD
keeps working while results arrive
lead   DONE
worker 1
worker 2
worker 3
worker 4
worker 5

LEAD   WORKERS   ↓ DELEGATION   ↑ REPORT BACK   LEAD IDLE

### ProgramBench: rebuild a program from a binary

[7:16](https://www.youtube.com/watch?v=maitMe9HCvw&t=436s) [A flowchart goes from a binary and documentation through probing, rebuilding and scoring. Below, an agent probes dsq with sample commands. A grid of hidden tests beside the reconstructed code shows 60%.](https://assets.ai.engineer/sites/aiecode2025/sha256/99da5cfc1b6e6d34ba0e4003097086d1cc29c0e08cd7a19539280c42d478eed7.webp)

ProgramBench: rebuild a program from a binary

200
programs,
from jq and ripgrep
to SQLite, FFmpeg
and PHP

Binary + docs
can run it, cannot

Probe
discover interface,
edge cases, behavior

Rebuild
a new
implementation

Score
hidden
behavioral
tests

200 PROGRAMS · NOW REBUILDING dsq · STEP score

ORIGINAL PROGRAM
dsq
source hidden
documentation

1 · PROBE THE PROGRAM
agent
$ dsq --help
$ dsq a.csv -f q.sql
$ dsq --schema a.json
$ cat a.csv | dsq
$ dsq notes.txt
INTERFACE
EDGE CASES
RUNTIME BEHAVIOR

2 · RECONSTRUCT THE CODE
reconstructed dsq

3 · SCORE
hidden tests   60%

### Teams beat a single agent, at 4–7× the cost

[7:46](https://www.youtube.com/watch?v=maitMe9HCvw&t=466s) [A chart plots reward against total inference cost for GPT5.6-Sol. The solo agent is at the lower left, the asynchronous team is higher and farther right, and the blocking team is highest and farthest right. Both team markers have vertical error bars.](https://assets.ai.engineer/sites/aiecode2025/sha256/763f4b1dbcf152443f6263103b2cec70312825c82a8b75284fc586a9e9598b90.webp)

Teams beat a single agent, at 4-7× the cost

minimal prompt   solo   team, blocking   team, async

Reward
65%
60%
55%

Total inference cost
$0   $100   $200   $300   $400   $500

Reward
+9.7 pts
team vs. solo,

Cost $
4-7×
the cost of the solo agent

Model: GPT5.6-Sol

### The prompt (heavy)

[9:50](https://www.youtube.com/watch?v=maitMe9HCvw&t=590s) [A system prompt for a lead agent fills the center of the slide. Three braces on the right label the phases Explore and Plan, Implement, and Verify. The visible verification section ends mid-list introduction.](https://assets.ai.engineer/sites/aiecode2025/sha256/1b38ec6e18107fb82bd6d41c47e5a66feadd332d2ce8343bd410250ce523aeb3.webp)

The prompt (heavy)

Asynchronous-Push Heavy PIV Lead System Prompt

Work as the lead of a team of engineers. Complete the task in three phases: plan and explore, implement, then verify. Use sub-agents during all three phases.

Sub-agents run asynchronously. Call `delegate_agent` for concurrent assignments before waiting for any one result, and continue useful non-conflicting work while they run. It returns an `agent_id`; retain that ID when the same worker’s context may be useful in a later phase or follow-up. Use `list_agents` to inspect status and unread outcomes. Use `wait_for_updates` only when no productive independent work remains or a result is required to proceed.

## 1. Plan and explore

Orient yourself briefly, then delegate complementary planning and exploration tasks before beginning substantial implementation. Useful divisions may include:

- requirements, codebase, architecture, and interface mapping;
- systematic investigation of major features and workflows;
- edge cases, failure modes, compatibility constraints, and side effects;
- implementation strategy and risk analysis grounded in evidence.

Choose scopes that fit the task and avoid duplicate broad assignments. Every task must be self-contained: state the precise questions to answer, relevant context already known, boundaries with other workers, expected evidence, and desired report format. Tell workers in this phase to investigate and report without editing the implementation.

Conduct your own non-overlapping exploration while workers run. Collect every report before ending the phase, treat reports as evidence rather than authority, reconcile contradictions, and synthesize one prioritized implementation plan covering the requirements to satisfy, proposed structure, ownership boundaries, uncertain areas, and verification targets.

## 2. Implementation

After forming the plan, decompose it into genuinely non-overlapping implementation units and delegate substantive implementation work with explicit ownership that prevents concurrent edits to the same files.

Every assignment must include:

- the behavior to implement and the evidence supporting it;
- the exact files or modules the worker owns;
- interfaces and invariants that must remain compatible with other pieces;
- files or concerns the worker must not change;
- focused tests or acceptance checks to run;
- what changes and caveats to report back.

Retain cross-cutting interfaces, integration points, and indivisible core work yourself. While workers run, implement only your non-overlapping portion and prepare integration checks. Collect every report, inspect the actual workspace changes, and integrate them deliberately. Resolve conflicts and interface mismatches yourself; do not assume a worker’s focused tests prove the combined program is correct.

## 3. Verification

Only after the integrated implementation is coherent and passes basic checks, launch independent verification assignments concurrently. Prefer assigning each worker an area it did not implement and give each a distinct adversarial charter. Useful divisions may include:

Explore and Plan
Implement
Verify

### With heavy prompting, better performance

[10:15](https://www.youtube.com/watch?v=maitMe9HCvw&t=615s) [A chart plots performance against total inference cost. Black arrows show the minimal prompt and purple arrows show the PIV prompt. With the heavy prompt, the solo score rises from 52.1% to 57.7%, and the blocking and async teams both reach about 63%.](https://assets.ai.engineer/sites/aiecode2025/sha256/565af6a286adafa220d96238dcdbae9695448b397b60207fc27c9ed08cbc1be5.webp)

With heavy prompting, better performance

minimal prompt    PIV prompt    solo    team, blocking    team, async
Inference cost
65%
60%
55%
$0    $100    $200    $300    $400    $500
Total inference cost

+5.6 pts
solo agent, just from the heavy prompt (52.1% → 57.7%)

Gap closes
blocking and async both reach ~63%

Model: GPT5.6-Sol

### Where should the extra agent go?

[12:23](https://www.youtube.com/watch?v=maitMe9HCvw&t=743s) [Three diagrams each connect one lead to five workers. In the first, workers plan while the lead implements and verifies. In the second, workers implement while the lead plans and verifies. In the third, workers verify while the lead plans and implements.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a8ae3ee6b5bed77f84ea7056903f2d368f927a9cd75fe324eba162eb60406ea.webp)

Where should the extra agent go?

Workers plan
lead
implement
verify

Workers implement
lead
plan
verify

Workers verify
lead
plan
implement

### Planning-only delegation matches PIV's reward at half the cost

[13:38](https://www.youtube.com/watch?v=maitMe9HCvw&t=818s) [The left plot shows planning-only and all-stage delegation reaching similar rewards at roughly $250 and $500 in total inference cost. The right chart compares reward points per $100. Planning has the tallest bars for both blocking (3.8) and async push (3.4).](https://assets.ai.engineer/sites/aiecode2025/sha256/afb1838fe9c035977644ecbc4f0d2212091daa427c41dc37d91666c4f7daec68.webp)

Planning is the most efficient place to spend compu

planning (P)    implementation (I)    verification (V)    all stages (PIV)
solo    all-Sol team
half the cost
65%    60%    55%
solo minimal    solo PIV    I    V    P    PIV
$100    $200    $300    $400    $500
Total inference cost

Efficiency η (reward pts / $100)
0    1    2    3    4    5
Blocking    Async push
Planning (P)    3.8    3.4
Implement. (I)    2.4    1.7
Verification (V)    1.7    0.9
All stages (PIV)    1.4    1.3

Planning-only delegation matches PIV's reward at half the cost (ηₚₗₐₙₙᵢₙɢ = 3.8 vs. ≤ 2.4 for next best)

### Why planning workers pay off?

[14:24](https://www.youtube.com/watch?v=maitMe9HCvw&t=864s) [Dashed lines connect a binary to four workers. Each worker sends a short report to one lead, which uses a single plan to implement and verify.](https://assets.ai.engineer/sites/aiecode2025/sha256/a79d6c12a643e9cd859750477277cd16cbbdd5bb02bd121e06c6d45449945f92.webp)

Why planning workers pay off?

binary
lead
plan
implement
verify

Discovery is the bottleneck
the spec is hidden in a binary

Fresh contexts cover more
each worker probes one part

Short reports back
the lead's context stays clean

One plan, one context
the lead builds and checks on it

### Why planning wins: one early burst of discovery

[15:10](https://www.youtube.com/watch?v=maitMe9HCvw&t=910s) [Four ProgramBench timelines show when the lead delegates under PIV, planning, implementation, and verification guidance. Dots cluster at the start under planning guidance and recur later under PIV guidance. Adjacent bars show the share of calls in each phase.](https://assets.ai.engineer/sites/aiecode2025/sha256/b9da81aec4dd35944eac29beb38a07b9ee923d2dec53932b2bb952d86e27993b.webp)

Why planning wins: one early burst of discovery

Planning    Implementation    Verification    Successful lead delegate_agent call

ProgramBench tasks (sorted by earliest implementation)
PIV guidance | 1,625 calls per run (avg.)
P    49%
I    32%
V    19%

P guidance | 674 calls per run (avg.)
P    100%
I    <1%
V    0%

I guidance | 429 calls per run (avg.)
P    74%
I    26%
V    <1%

V guidance | 599 calls per run (avg.)
P    0%
I    55%
V    45%

0    0.25    0.5    0.75    1
Normalized team wall-clock progress
0%    50%    100%
Mean share of delegation calls

674
P-guidance calls, all in one burst before any code is written

+830
extra PIV calls during implementation and verification, with no reward gain

2.4×
more reward per delegation call for planning

### Why implement workers fall short?

[16:45](https://www.youtube.com/watch?v=maitMe9HCvw&t=1005s) [Four bullets describe an incomplete plan, workers seeing only their own assignments, 84% of messages going to the lead, and the lead having to stitch duplicated work together.](https://assets.ai.engineer/sites/aiecode2025/sha256/b348472978931f9c43cb0b203fafc96b4413973bdc343b81d9851375a769ab7a.webp)

Why implement workers fall short?

• The plan is incomplete: only the lead explored
• Each worker sees only its slice: give it more context and it
oversteps, building features nobody asked for.
• Workers work alone: 84% of messages go to the lead, and nobody
sees who is doing what.
• The lead pays for the gaps: duplicated work and stitching, so
latency goes up and reward goes down.

### When the spec is given, implementation wins

[18:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=1110s) [Two columns compare benchmarks. Planning is the best stage to delegate on ProgramBench, where workers must discover the spec. Implementation wins in 8 of 10 setups on Commit0-Lite, where the skeleton, docs and tests specify the work.](https://assets.ai.engineer/sites/aiecode2025/sha256/1fd696343cde8ed2e6fe3d38090f0cad63e33a0ed051041c10a591f36460980e.webp)

When the spec is given, implementation wins

ProgramBench
Spec must be discovered by running the
binary

Planning
is the best stage to delegate

Commit0-Lite
Skeleton, docs and unit tests spell out what to
build

Implementation
is the best stage in 8 of 10 team setups

### Cheap planners keep about half of the gain at 60% less cost

[20:12](https://www.youtube.com/watch?v=maitMe9HCvw&t=1212s) [A chart plots percentages against total inference cost. Orange planning points show cheap workers near $100 and an all-Sol team near $250. A dashed arrow labels the cheaper team's result as half the gain at 60% less cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a568199a69876792eedff370c7ec5f9e4b392b8c2c8ad24f71d64356103d79b.webp)

Cheap planners keep about half of the gain at 60% less

planning (P)     implementation (I)     verification (V)     all stages (PIV)
solo     all-Sol team     Sol lead + cheap workers

half the gain
60% less cost

half the cost

65%
60%
55%

solo minimal
solo PIV
P     I     V     P     PIV

$100     $200     $300     $400     $500
Total inference cost

### Can the lead be cheap?

[21:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=1290s) [A chart compares solo models, same-model teams and teams with Luna workers on a logarithmic cost axis. Sol + Luna and Sol + Sol both plot near 64%, with Sol + Luna farther left at a lower cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a7581e960b3e9c3924f1df866a8050a222ff82fdc9a3a73faefcc7b0069e4bd.webp)

Can the lead be cheap?

solo     team, same model     team, strong lead + cheap workers

65%
60%
55%
50%
45%
40%
35%

Luna     Terra     Sol
Luna + Luna     Terra + Terra     Sol + Sol
Terra + Luna     Sol + Luna

$2     $10     $50     $250
Total inference cost (log scale)

The lead sets the ceiling and workers set the price!

### Scale the team where the bottleneck is

[22:06](https://www.youtube.com/watch?v=maitMe9HCvw&t=1326s) [Two charts compare worker count and inference cost. With a Sol lead, increasing cheap planning workers from 2 to 16 raises the plotted result from just under 60% to nearly 62%. The 16-worker team costs 38% less than the all-Sol planning team and 70% less than the all-Sol team used at all stages.](https://assets.ai.engineer/sites/aiecode2025/sha256/a90a309661748695c03c4a3a9a6d8b7e780be66a8e9887d88cc6bb364c781d2b.webp)

Scale the team where the bottleneck is

planning     all stages
solo     all-Sol team, sized by the lead     Sol lead + cheap workers, fixed size

Number of workers
64%
62%
60%
58%
56%
54%
solo
2     4     8     16
Number of workers

Inference cost
38% lower cost     70% lower cost
64%
62%
60%
58%
56%
54%
solo
$75     $150     $300     $600
Total inference cost (log scale)

### Workers mostly report to the lead

[24:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=1440s) [Paired horizontal bars show messages per task for seven model configurations. Worker-to-lead messages dominate. The slide reports that 84% of messages went from workers to the lead and workers exchanged fewer than one message per task with each other.](https://assets.ai.engineer/sites/aiecode2025/sha256/5bf7a5c075038d5204c9a4afbc211535eb251369560b90a5a9cf4c2e2760dc10.webp)

Workers mostly report to the lead

Messages per task (peer)

Sol
Team minimal
Team PIV

Sol + Luna
Team minimal
Team PIV

Terra
Team minimal
Team PIV

Terra + Luna
Team minimal
Team PIV

Luna
Team minimal
Team PIV

Claude Opus 4.7
Team minimal
Team PIV

Claude Haiku 4.5
Team minimal
Team PIV

0     2     4     6     8     10     12     14     16
Worker → lead     Lead → worker     Worker → worker

84%
of messages went from
workers to the lead

<1
worker-to-worker message per task

### Peer messaging didn’t raise reward in our harness

[24:41](https://www.youtube.com/watch?v=maitMe9HCvw&t=1481s) [Three bar charts compare message counts, text volume and reward changes for Luna, Terra and Sol under two prompts. Hatched bars mark rejected Luna calls. Luna loses reward under both prompts, while Terra and Sol stay near zero.](https://assets.ai.engineer/sites/aiecode2025/sha256/872724816d8e650b994914f5d79121a12cc091d679f19029d0719c2624cfda09.webp)

Peer messaging didn’t raise reward in our harness

■ minimal prompt    ■ PIV prompt    ▧ rejected worker call

Messages per task
20    10    0
Luna    Terra    Sol

Message text per task (k chars)
30    20    10    0
Luna    Terra    Sol

Δ reward with messaging (points)
0    −5    −10
Luna    Terra    Sol

~half
of Luna messages were
malformed
and rejected

3.4 x
more messages from Sol than Luna
with no gain in reward

-4 to -7
Reward points for Luna; Sol and Terra
unchanged

### Not every model knows how to lead a team

[27:05](https://www.youtube.com/watch?v=maitMe9HCvw&t=1625s) [Two side-by-side dot plots show delegations per task and reward lift over the best solo result for eight models. Filled and open diamonds and squares distinguish four conditions. A list at right describes the lead’s role.](https://assets.ai.engineer/sites/aiecode2025/sha256/4424521738303187e736bb3b35485d147f90ec7bf69704818ea13f046df07d6b.webp)

Not every model knows how to lead a team

◆ minimal, async    ◇ minimal, blocking    ■ PIV, async    □ PIV, blocking

Delegations per task                         Lift over best solo (points)
GPT-5.6 Sol
GPT-5.6 Terra
GPT-5.6 Luna
Claude Opus 4.8
Claude Opus 4.7
Claude Haiku 4.5
Gemini 3.8 Flash
Gemini 3.7 Flash
0    2    4    6    8    10                         −5    0    +5    +10

Two things have to go
right
1. The lead chooses to
   delegate
2. It makes good use of
   what comes back

### Takeaways on team scaling, planning workers and cost

[28:08](https://www.youtube.com/watch?v=maitMe9HCvw&t=1688s) [Three numbered findings compare teams with a single agent, planning workers with workers at every stage, and a strong lead paired with cheap planners. The final finding underlines “70% lower cost.”](https://assets.ai.engineer/sites/aiecode2025/sha256/e2b8f2e4df34ddd8f5946e1021eba25accaa8e3f8f82bd201692b1a943820dd6.webp)

Takeaways

1    Teams are a new axis of inference-time scaling
     Up to ~10 points over a single agent BUT at 3-7× the cost

2    Find the bottleneck and scale the team there
     Planning workers alone match workers at every stage, at half the cost

3    The team doesn’t have to be all strong
     A strong lead with cheap planners: within 1.4 points at 70% lower cost

### Open questions on agent organization, communication and security

[29:01](https://www.youtube.com/watch?v=maitMe9HCvw&t=1741s) [Three numbered blue cards ask how thousands of agents should organize and spend compute, whether coordination emerges with scale, and how to secure teams when errors propagate through delegation.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e4fec6d8221651529223d535f120ef345ae0cbf6dc865ad6f51d91700bf5bc9.webp)

Open questions

01
Hierarchy vs. self-
organization

How should teams of
thousands of agents be
organized, communicate, and
spend compute?

02
Alternative forms of
communication

As projects outgrow one lead,
will effective coordination
emerge with scale, or must it
be trained?

03
Multi-agent safety &
security

Errors propagate through
delegation; emergent
protocols may be hard to audit.
We need to urgently secure
agent societies.

## Full transcript

[0:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=0s) Hi everyone. Uh, my name is Nouha. I'm a senior researcher at Cohere, an associate industry member at Mila, and an adjunct PI at the Allen Institute, Tübingen. I'm so excited to tell you today about how to optimize your agent team and get frontier coding at seventy percent lower the cost. Um, this is a work that I did, uh, at Cohere. All right. So traditionally, there are two ways to make an agent stronger at inference time. First, you can let it think longer, meaning you scale compute sequentially

[0:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=30s) with longer chain of reasoning. And second, you can sample more. You run many independent attempts in parallel, and then you aggregate them. But as agent take on longer and longer tasks, they can navigate large repo, they can edit the code, they can debug their own work over hours. So what can happen here is that the context window may blow up. And so we can see here that, you know, the context window eventually fills up, and the longer it runs, the longer you wait. And sampling more

[1:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=60s) doesn't either fix it, because each attempt here still has to do the whole task on its own, right? So what if we split the work? Um, um, so here in this case, the key-- the lead keeps the task and hands out sub-tasks, and each worker starts with a fresh context. So it can read files, it can run tests, it can dig through logs, et cetera, et cetera, and then it can send back only a compact result. So here the lead keeps the full picture, and it can finish faster if the task is decomposable and

[1:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=90s) agents can work, uh, on independent pieces. And so here we're talking about a new axe, uh, for scaling inference time compute. You work as a team. You give a task to a team of coding agent and watch them do the task for you. But then scaling the team gets expensive very, very quickly. And so y- as you add more agents, your token budget can blow up. Um, and it can actually, um, and, and it raises a, a harder question, actually. So first, how does coordination actually

[2:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=120s) happen? How should we organize the team so that agent can work together, et cetera. And so, you know, this is the question that we set out to answer. How should a team be orchestrated to be both effective and efficient? By effective, I mean it actually solves more of the task, so we get higher reward. And by efficient, I mean it doesn't burn through your budget getting there. All right. So since these models now are extremely capable, I was wondering, what if

[2:30](https://www.youtube.com/watch?v=maitMe9HCvw&t=150s) we throw an extremely challenging coding task as a society of agent and see what they can do? And so we would expect them to automatically form clean groups, split up the work neatly, and build it together, communicate nicely, and coordinate, et cetera, et cetera. But that's not what-- not quite what happened, actually. In reality, what happened, the agent preferred to solve the task on their own. So each went off and built their own version. Even when we asked them to communicate, they mostly discussed things sometimes that didn't

[3:00](https://www.youtube.com/watch?v=maitMe9HCvw&t=180s) matter, and sometimes they kept also stepping on each other toes. They also preferred to communicate back to the lead, et cetera, et cetera. So here we were wondering, okay, how should we design a team so that, you know, we can actually get the best out of it in terms of reward and in terms of cost, and hopefully in terms of latency. And so, you know, we set out to figure out what a multi-agent harness should actually look like, and like we were wondering about the team design. So there are many design choices, and each one changes how well the

[3:29](https://www.youtube.com/watch?v=maitMe9HCvw&t=209s) team does and how much it costs. So first, the organization, right? How do you structure the team? Who leads? And at which stage of the work planning, implementation, or verification do actually workers come in? Um, the second thing, uh, is communication. Should agent talk to each other directly or only report to the lead? Third is the team size. How many workers is worth it? Like should it be one, two, sixteen, hundred, et cetera? And fourth, the models, right? Which model should

[3:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=239s) play which role? Do we need, you know, a strong model everywhere, or, you know, can cheaper ones do the same work, right? And finally here that something that we cared about is the stage of the software engineering, right? Should, should, you know, allo-- we should allocate more workers towards planning, towards implementation, toward verification, et cetera. And so we actually frame team design as a compute allocation problem. So what we cared about is reward per added dollar over the solo agent, right?

[4:29](https://www.youtube.com/watch?v=maitMe9HCvw&t=269s) So starting from a solo agent, we ask where does the next dollar of inference buy the most reward? This is what we cared about. And so here we actually define a metric called efficiency, and it's actually a function for things. First, the stage. It's the, the stage at which the worker is at, whether it's implementation, planning, or verification, or all of them together, we call it P. We also define that as a function of the worker model. Is it like a very capable model? Is there-- Is it a very, you know, uh, you know, less capable

[4:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=299s) model, the number of workers, the lead model, et cetera. Um, and all right. So basically, if we-- to answer this question, we actually design our multi-agent harness as a top-down hierarchy. We build deliberately a minimal harness, a mini suite agent, and it's hierarchical. As I mentioned here, we have a lead, and based on the task, it decides dynamically how big the team should be and, um, you know, who should do what. And so here we made sure that every part of the harness is exposed, so we could change

[5:29](https://www.youtube.com/watch?v=maitMe9HCvw&t=329s) one choice at a time. We keep everything else fixed and see how each one moves reward and cost. Right. And so here more closely, how does a team of coding agent do the work? You can see here every agent works through a bash tool in one shared workspace. So basically, the lead receives the task and hands out work with a tool we call Delegate H. And then the worker sees only its sub-task and reports back to the lead, and it can now also di-- communicate directly to the other worker. And so here, basically,

[5:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=359s) the lead can decide based on the task, "Oh, actually I need to just, you know, pow an implementation worker or a verification worker," et cetera. And so, as I mentioned also, the lead decides how many workers to spout, how to combine, also consolidate the results. It also has the capability to resume a worker, to kill a worker, and to tap in and, for example, supervise and tell the worker that you have made some errors here, fix it, et cetera. And so, yeah, and you may be wondering, okay, what is the lead

[6:29](https://www.youtube.com/watch?v=maitMe9HCvw&t=389s) doing while the other workers are working? We actually have two variation. One is what we call the blocking lead, the other one is asynchronous lead. So the blocking lead actually hands out subtask and waits until every worker reports back. It doesn't do anything, you know, in between. Uh, whereas an async lead, it keeps actually working while, you know, the results keep coming from the other workers. It can supervise what other workers are doing, it can send them feedback, et cetera, et cetera. And so how did we test this? We test this on a

[6:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=419s) benchmark called Program Bench, which is basically a reverse engineering task. So basically, the agent starts from an empty workspace with only the original program executable, which it can run but not read, and it also receives some documentation. So here the goal is to basically probe the program to discover its interface, to discover the edge cases, the behavior, and then write a new implementation that behaves in the same way. And then the task here were extremely challenging. They range from SQLite, sorry, for example, the

[7:29](https://www.youtube.com/watch?v=maitMe9HCvw&t=449s) PHP interpreter and for example, you know, on the leaderboard on the official website on the task FFmpeg, the best model only passes ten percent of the test. So this is also done by, you know, frontier model. So it was quite challenging, and it was a good test bed for, for our, you know, investigation. And so the question here is that, does a team beat a solo agent, yes or no? The answer is yes. It actually can beat a single agent, but at four to seven times

[7:59](https://www.youtube.com/watch?v=maitMe9HCvw&t=479s) the cost. And we can actually see here in the figure that, you know, with a minimal prompt, uh, a team with a blocking lead actually beats the solo agent by almost ten points, but it costs about, it costs about four to seven times as much. Now, you can see that the blocking actually perform better than the asynchronous, but cost higher and it actually takes more time to complete. So what we noticed here is that we noticed that the async lead doesn't know when to wait, when its

[8:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=508s) workers are still exploring. It starts building on its own guesses before the workers finding come back, and then when the reports finally arrive, it has already moved on. And so either it ignores them or it has to go back and redo the work. And sometimes it just decides the task is done while the workers are still actually running. So basically, it decides, okay, done. It submits the work, and so the work of these workers is simply lost. But then, you know, this problem never occurs with a blocking lead because it will

[8:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=538s) wait until the workers report back, and then it actually consolidates the work and do the submission. All right, so the next thing I want to emphasize on is the prompt, minimal. So I, I mentioned, okay, in the earlier results, I-- we used the minimal prompt. What is an earlier prompt? And I can't emphasize on this enough. This is how our agent operates. We give them prompt and instructions, and essentially they do what we ask them. And so the first one, we tried to go very, very minimal. We just told the lead about the task, you know, that it

[9:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=568s) should create, uh, you know, subworkers and agent, and it should inform them about the task and should tell them about the tools. We gave also the lead tools. We wanted to see why we started minimal because we wanted to see how the agent behave on their own. Do they need a lot of guidance or is it, is a minimal prompt enough, et cetera? And so this was very minimal. We also were wondering, okay, what happens if we give it a very heavy prompt? Meaning like, you know, what the model should exactly

[9:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=598s) do. We tell it exactly about different tricks. Follow exactly this plan of try to, you know, delegate agent and, you know, create agent throughout the whole process of the implementation from exploring and planning to implementation. Create also adversarial workers to verify the task, et cetera, et cetera. And actually, that helped a lot. Uh, we can see here, um, you know, compared to the minimal prompt, um, you know, we can see that the solo agent alone jumps almost six point just from better

[10:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=628s) instructions and telling them, "Okay, don't step on each other toes. Uh, don't, you know, uh, there is this folder right in this folder," et cetera. So lots, lots of guidance and lots of prompt tuning, which is not ideal 'cause we spent too much time there. And now these agent are supposed to be smart enough to figure out how to do it. And so we can actually see that the teams both, you know, async and, uh, blocking both go up, um, and we can see that the async lead now catches up with the blocking one, right? So once we tell the lead how to coordinate, it knows when to

[10:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=658s) wait. So this actually really helped address the problem that I talked about earlier. But again, you know, we have a team that works better than the solo, but it's really, really expensive. It costs six to seven times as much as the solo agent. So we were wondering, now that a team beats a solo agent, um, but it costs more, so if a team is a new way to spend inference time compute, where should the next dollar go, basically? Can we actually optimize our agent setup so that actually we get

[11:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=688s) very good reward, but at a very, you know, much cheaper cost? So to answer this question, actually, we took the same lead and gave it workers at only one stage of the software engineering process. So the first setup that we considered, here, the workers are only responsible for the planning and exploration, right? So basically, here, the lead will give them the task, and then each of the workers will go and try to probe the binary to discover, okay, what kind of plan I need to do, and then it will report the plan to

[11:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=718s) the lead, and the lead will take care of the implementation verification. The second one is, you know, workers are only responsible for implementation. And so the lead is the one responsible for coming up with a plan, distributing that, splitting the work between workers. They will do the implementation, report back with the code Actually to the, to the lead. And so basically, and then the lead does the verification on its own. Now, the third actually setup that we had, the lead is responsible for both the planning and

[12:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=748s) the implementation. And then here the workers are responsible to do the verification, which is the final stage, and then they will report back to the lead, and then the lead will decide whether to submit or not. And so, okay, we wanted to see where exactly extra agent will pay off the most. And here are the conclusion of our, our results, like extra agent pay off the most in program bench in planning. So here you can see, okay, going from solo with minimal prompt to solo with PIV. PIV is the heavy

[12:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=778s) prompt. Actually, we get like a jump already, and then we go with a team of heavy prompting. We get extra, you know, increase and gain in performance. But then, now when we allocate more agent towards the planning, right? We can see that we get almost the same performance as the PIV, but at half the cost. So actually, the conclusion here is that, you know, the extra agent pay off the most in planning. Um, uh, at, at, at the task that bottlenecks actually, uh, the, the, the task

[13:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=808s) which is planning in this case, because you need to figure out how exactly, what's the bi-binary behavior and come up with an, um, detailed plan on what should you implement first, et cetera. And so here, yeah, I... Actually, we computed even, you know, the efficiency ratio that I discussed earlier. And we can see that, you know, it dominates the PIV actually on the frontier. So each planning dollar buys almost one point six more reward than any other stage that we have here, whether it's implementation or verification. And actually, we, we, we

[13:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=838s) noticed that, you know, you know, doubling down on the number of agent in the implementation sta-- in the implementation or verification stage doesn't actually really help. We can see that it just increased the cost, but we are actually getting the same performance as the solo, so it's not worth it. And so again, like program bench only exposes a black box executable. So discovery here is the bottleneck. So planners paralyze this probing and it brings, and they bring lots of benefits, right? And so, yes. So this is to explain

[14:28](https://www.youtube.com/watch?v=maitMe9HCvw&t=868s) why planning workers pay off. And we were wondering, why not, for example, verification or implementation. And you know, as I mentioned, program bench give you this binary and its documentation. So before you write anything, you have to work out what the program actually does. So discovery is the real bottleneck here. And you can also think about this in terms of context, right? So we have one agent that's exploring, you know, loan files, uh, it's in its own context with probing and also still misses edge cases, right? And on the other

[14:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=898s) hand, you know, we have several workers each explore a different part of the program in a fresh context, and each one hands back a compact report to the lead. So basically here the lead, you know, learns more than it could do alone. And so now, you know, this is also to explain better this phenomena. And you can-- We actually looked at our data, and we came up with these nice figure based on what we saw. And we spe-specifically looked at, uh, when the lead delegates, right? So here we have planning, we have implementation and

[15:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=927s) verification, and we have all the stages as well. And so every row that we can see here in the figure is one run. Uh, and actually time goes left to right, right. So the yellow is planning, the blue is implementation, the green is verification. And every dot here that you see in the figure is the lead sending work to a worker. And so with planning guidance, all the delegation happens in one burst at the very, very first of, you know, the, the work before any code is written. So basically here the, the workers

[15:58](https://www.youtube.com/watch?v=maitMe9HCvw&t=958s) go explore, and then they report back, and then the lead builds. With the full, um, PIV guidance where, you know, um, there is no, um, specific guidance on where exactly the, the, the number of workers should be allocated. We get the same early burst, but then if you see here, y- actually the lead keeps delegating all the way through. Another eight hundred and thirty calls and they buy nothing. Actually, it just increased the cost. As we saw, it was around five hundred dollar, but it didn't

[16:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=987s) actually improve in performance. So it's the same reward for planning, we can see, but it's two point four times more reward per call that we, we actually measured. And so what pays off here is, is, is one early burst of discovery and not delegating all the way through, right? So why implement workers fall short? Here, first of all, as I mentioned, the plan was the bottleneck, and so if you don't do it right, it's not gonna work out. And so here we have only the lead that explored. It

[16:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1017s) had limited capacity to discover all, you know, the, the edge cases and the full binary. And so we had a weak foundation, and then basically each worker got like a narrow slice. It doesn't have visibility over the entire task. And basically, they were already doing errors by implementing each feature on, on their own. That's one problem. And on top of that, they were building on a weak foundation. And also we noticed that... So here you might ask me that, you know, why not give workers more context?

[17:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1047s) We did. But we noticed that when you-- they have more context, actually, they didn't respect the boundaries of their specific task. They get excited, and then they want to build extra features that nobody asked them for it, so they just step on each other toes, et cetera. The other issue that we noticed is that, you know, they are supposed to communicate together to figure out who's doing what to avoid conflicts and duplicated work, but they didn't do that. They actually, they worked alone and eighty-four of messages go, go to the lead directly and nobody sees

[17:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1077s) who's doing what. And sometimes here what happens is the lead gets overwhelmed, and it doesn't know, doesn't communicate well where the problem is to the other workers, et cetera. So it can be actually quite challenging. And so what happens here is that when, you know, implementation is not, uh, fully correct, you know, the, the, the lead pays for the gap. It actually tries to, you know, make up for that duplicated work and does lots of stitching. And so here what happens is that the latency goes up and also the reward goes down and,

[18:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1107s) you know, w-- it, it, it's also expensive. Uh, okay. So basically so far I've been talking about program bench, but we also run experiments also on CommitZero. And what I want to say here is that the results, like are, are different and the hierarchy and the, the allocation of the team is different according to the task. Because the bottleneck for program bench was planning, and so allocating more agents there really, really helped. But for CommitZero it was not the

[18:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1137s) case. The CommitZero, the bottleneck was not planning because basically you're given the skeleton, so the docs are known and the unit tests are actually known to the workers. So the plan is basically given to you. And the, the, the bottleneck was for the, you know, for the reward to go up was the implementation. You need to get the implementation right. And so here when we doubled the number of agent on the implementation stage, this is where we saw the biggest improvement. I'm not gonna present results on this. Please check out the

[19:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1167s) paper for, for, for these results. You'll find them in the appendix. So the conclusion here is that you need to allocate more agents at the software engineering stage that bottlenecks your task. All right, so planning already, uh, gets us the same reward at half the cost. So, uh, we were wondering, can we, uh, go even cheaper than this? Can we cut the cost even further? And so here we did some experiments, and we focused, for example, on three GPT five point six, uh, models, uh,

[19:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1197s) Luna, Terra and Sol, uh, in, uh, this, uh, increasing order of capability and cost. And so, like the question here is that what would happen if we keep Sol as the lead, like the strongest model, and then use the cheap models as the workers, right? So remember here we used to have this. This is all the P here is the old Sol, you know, model. By the way, here the, the, the Sol, the basic-- this is the version like of P where the lead dynamically decides on the size of the

[20:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1227s) team, and it ended up on average creating and spawning five agents, right? And so basically here we have this, we go from P id, we actually allocate more agent towards P. And, uh, basically when we replace the Sol, you know, sub workers with cheap ones like Luna, we actually get almost, you know, same performance but at sixty percent less the cost. Like just we lose a little bit on performance, but it's still good because we're actually

[20:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1257s) maximizing reward compared to the Solo and then we're, we're getting less cost. And so, you know, now the other way around, we were wondering like, can the lead be cheap, right? Can we just even get a, a cheaper lead? And so already here we can see that, you know, going from like a cheap lead like Solo, Luna, Terra, Sol, we keep increasing in performance and also in cost. And so can we have like this homogeneous team like Luna plus Luna, Terra plus Terra. Already we can see the scaling law

[21:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1287s) the same. As, uh, the model gets more capable, we get better results. But then actually, you know, we noticed that when we have a strong lead, Sol plus Luna, we can get the same performance as Sol plus Sol, but you know, at much lower the cost. But the other way around, like cheap workers can't make up for a we-weak lead. Basically, the lead sets the ceiling and the workers set the price. Here, Terra with Luna workers still trail Sol working alone fifty-one percent versus fifty-seven percent. So essentially here the lead

[21:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1317s) sets, you know... Basically, this also matches Meta's meta-reasoning work recently. Like they just mentioned that a better controller raised what the same workers achieved. All right. So, so far the lead decided how many workers to spare, as I mentioned, and it did on average five workers. Now we were wondering what would happen. We want to scale ourselves the number of agent at each stage. So we will impose the team size. Um, so here, uh, we're gonna, you know, work with Sol as a lead, Luna as

[22:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1347s) planners because this was the best, um, reward cost, um, um, uh, relat-- and we go from two to sixteen planners, right? Um, and so here I can actually look at the cost. You know, the sixteen planner team is thirty-eight percent cheaper than all Sol planning team and actually seventy percent, um, cheaper than workers at every stage here for the PID, which is great 'cause here we-- and it seems we, we-- due to some

[22:57](https://www.youtube.com/watch?v=maitMe9HCvw&t=1377s) budget constraint, we couldn't like scale up the, the number of agent further, but it seems like going up and up and up, obviously there is a ceiling to it, but it's really helping. And the good news is that it's not that expensive. It's not a-- it's not more expensive than the five Sols in, uh, all allocated towards planning. Now we also, uh, try to see whether, uh, scaling up the number, the size of, uh, the team, uh, for the other variants like implementation and verification, and it

[23:27](https://www.youtube.com/watch?v=maitMe9HCvw&t=1407s) didn't help at all. Uh, we already saw that, uh, you know, allocating teams', um, agents to work in these stages, uh, is not helping because of, you know, the foundation was weak and other considerations. And so the last thing we were actually curious about, how do agents communicate, right? And so we gave every agent, uh, a send message tool to reach any other agent actually in the sandbox. And we were curious whether communication, uh, happens directly or, you know, it happens through the lead and

[23:56](https://www.youtube.com/watch?v=maitMe9HCvw&t=1436s) whether communication will lead to better performance or worse. And so basically the first thing is that are they communicating together? Not really. They are actually essentially communi-communicating through the lead. Like eighty-four percent of all messages went from a worker to the lead. Worker to worker, you can see is actually very little, less than one message per task. So even when they can talk to each other, they actually behave like a hierarchy. Like do your piece, you report up, expect the lead to communicate back to the right sub

[24:26](https://www.youtube.com/watch?v=maitMe9HCvw&t=1466s) agent, et cetera. And you can also see this pattern consistent across all models. The lead takes much more, talks, sorry, talks much more to workers than they talk to, to each other. And this is also true for Claude Opus four point seven. Also, we did experiments on Gemini

[24:44](https://www.youtube.com/watch?v=maitMe9HCvw&t=1484s) I mentioned, we were curious about the reward. Does it, you know, does extra communication between workers help or not? In our experiments, we didn't find it helped, and I will explain why. Um, so first already you can see, you know, on the left, you can see that about half of Luna messages never even get through, like, because they're malformed and the harness reject them, et cetera. Um, and so we notice also in the middle figure that Sol writes about three to four times as much message text as, as

[25:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1513s) Luna. And under the PIV guidance, stronger teams actually, we can see that they tend to send more messages overall with leads steering their workers more, which is good. So we can see that, you know, as the, you know, capability of the model increases, so they message rather more. But then on the right figure, you can see that more talking doesn't mean more reward. For example, for Sol and Tora-- Terra, adding messaging changes nothing basically beyond noise, and Luna teams actually lose four to seven

[25:44](https://www.youtube.com/watch?v=maitMe9HCvw&t=1544s) points, so it just seems distracting to them. And we noticed that communication mostly carries status report to a lead, you know, who already receives each worker results anyway, so it, it was just like an overhead. There was nothing important to be discussed. And I want to be clear here. I don't wanna say that peer messaging is not necessary and we should not do it. I do think it's necessary, and I do think as we scale the number of agent, they should be capable to communicate efficiently together because it doesn't make sense to

[26:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1573s) all the time communicate through your manager for you to work with your team members. No, it doesn't work like that. I think that it's also a byproduct of how, uh, of the harness that we built and also of the, you know, the task itself. Because basically the lead was splitting the work and making sure that each work is independent of the others, and so they give them di-- or the lead gives them directly the task, and so they are not also aware of other, you know, agent in the same sandbox. And so we gave them

[26:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1603s) the tool only to communicate together, but they didn't have visibility who exists there. And so they have to do extra efforts, and maybe a message where would have solved the problem, and so would incentivize them more to sort of communicate and to figure out, "Oh, this actually agent is doing something similar to what I'm doing. Let me communicate." Or, "I need more help. Let, let me reach out to someone else." Anyways, we leave this to future work. And so the other thing that we also were curious about is like, you know, whether other, you know, models, frontier

[27:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1633s) models know how to lead. So we also ran five other models family with the same harness, with identical tools. And so here I'm talking about Opus four point eight point-- four point seven five, also Gemini family. We noticed that GPT five point six teams delegates often, and it gained up to seven points in reward. Claude, on the other hand, actually behaved much, you know, the same, gaining almost six points. But then Opus four point seven rarely delegated and actually

[27:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1663s) gained very little. Haiku and Gemini, you know, delegated as we-- delegate as GPT five point six, but then gained nothing. And so what I want to say here, the conclusion is that, you know, two things have basically to, to go right. The lead has to choose to delegate. This is important because it's actually deciding on who's gonna do the work, and it has to actually make use of what comes back and at which stage of the task do you also delegate. All right, so here are the takeaways. Three things to remember. The

[28:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1693s) first thing is that teams are now a real new axe for inference and compute, but the catch here is that they cost several times more. We got up to ten points, which is, you know, great, but still it's very, very expensive. Now, you know, we tried our best to optimize our teams of agent, and we were successful to some extent. Still some work to be done. But essentially here the conclusion is that, you know, where you put agent, uh, matters more

[28:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1723s) than how many.  So you need to find the bottleneck and scale the team there. This is what matters the most. And finally, the team does not have to all be strong. Like a strong lead with cheap workers can get you near frontier results at actually a fraction of the cost. And so what's next here? These are open questions that we are so curious and excited about. First, here we're excited about this hierarchy versus self-organization. We, in our work, we tested up to sixteen workers in a

[29:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1753s) hierarchical lead worker structure. But then how should teams of thousands of agent be organized, communicate, and spend compute? Can agents design the team themselves given task, budget, and deadline? We think so, and so, but that requires, you know, careful investigation of the problem. Now, the second thing is that we think that there must be alternative forms of communication. As I mentioned, we observed eighty-four percent of messages going from workers to the lead and only four percent between workers. And

[29:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1783s) so as... I think projects outgrow, like coding project, you know, they can outgo- grow one lead. Will effective communication here emerge, right, with scale or, like, should it be trained, right? How should the lead adapt to the task? It's not scalable to just... For each task that we have, we spend, uh, you know, a huge amount of time optimizing the prompt and giving very detailed instruction to the lead. That should not be the

[30:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1813s) case. The lead should figure out how to actually, you know, uh, organize the team, uh, according to the task that it has. Um, now finally, uh, multi-agent safety and security. Um, you know, uh, this is, I can't emphasize on this enough, especially given what happens with the Hugging Face incident. You know, right now we have teams of agent that can run autonomously twenty-four/seven with no supervision. They're quite capable, and we're not

[30:43](https://www.youtube.com/watch?v=maitMe9HCvw&t=1843s) talking anymore about the single agent, but we talking about teams of agent and even swarm of agent. And so here the, the key question is that how do we secure our systems and our infra? And we need to urgently basically figure out ways on how to align these models. All right. With that, I actually would like to thank you for making it to the end. And I want to give a special shout-out to the first author here, Kinan and Daniel, for their hard work on this.

[31:13](https://www.youtube.com/watch?v=maitMe9HCvw&t=1873s) And for more information, please check out the paper and the blog using these QR codes. And thank you so much for listening.
