AI Engineer World's Fair 2025
Real AI Agents Need Planning, Not Just Prompting
Read the talk
Real AI Agents Need Planning, Not Just Prompting
As instructions accumulate, reliable output becomes a search and validation problem. Planning coordinates that work; an execution engine decides how to spend the time and compute.
From a talk by Yuval Belfer
Before you start: Familiarity with LLM prompts and tool calling is helpful; no prior planning-framework experience is required.
Why is instruction following still a problem?
In 2022, InstructGPT made following natural-language instructions substantially more useful. By the release of GPT-4.1 in 2025, instruction following was still an engineering problem. Why had better models not made it disappear? Yuval Belfer opens with that question, extending it beyond OpenAI to language models generally.
Consider the early request: “Explain the moon landing to a six-year-old.” A short, accessible explanation was enough to make the capability feel remarkable. As developers adopted language models, however, requests grew into long specifications. The task remained in the prompt, but so did the context, constraints, formatting rules, and business requirements. Putting everything into one input made the model responsible for satisfying all of it in one attempt.
A task can be easy to describe and difficult to satisfy. Even instruction following can demand more than a single generation. This is the opening for an agent architecture: not merely a longer prompt, but a way to plan the work required to meet the requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From routing to reasoning and acting
Belfer works at AI21 Labs, which develops language models and an agentic framework. His approach to defining an agent is pragmatic: whether a working system is called an agent or a workflow matters less than whether it solves the problem. That leaves several useful architectures on the table.
| Pattern | What the model does |
|---|---|
| LLM router | Sends a query to a specialized language model. |
| Function calling | Selects from provided tools to interact with APIs or services such as Google Search. |
Belfer introduces MCP as standardizing these external interactions, then leaves the protocol details aside. Tool access is useful regardless of whether the surrounding system earns the agent label.
ReAct, short for reasoning and acting, adds an iterative loop: reason about the situation, take an action, observe the result, and repeat. Belfer characterizes this as choosing the next step from the current state rather than looking ahead across the entire plan. That is a distinction about the planning horizon of the pattern he describes, not an inherent prohibition on planning: the original ReAct work also discusses updating action plans.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A plan needs an execution engine
Planning means figuring out the steps needed to reach a goal. It becomes useful when the problem has no straightforward solution, particularly when work needs to run in parallel or the system needs to explain its choices. Belfer's concern with a ReAct trace is that seeing the sequence of thoughts and actions does not necessarily explain why that route was chosen.
Plans can take different forms:
- Text: Belfer points to Microsoft's Magentic-One as an example of text-based planning.
- Code: He points to Hugging Face's smolagents, where agents can express actions as executable code. This is a useful contrast in representation, rather than an exclusive product taxonomy; smolagents also supports conventional tool calling.
A dynamic plan can change during execution. Instead of committing to the initial sequence until completion, the system reassesses whether the plan is still useful and replans when necessary. That makes planning an ongoing responsibility rather than a one-time preamble to the task.
The execution engine makes the plan operational. It analyzes dependencies between steps, identifies work that can run in parallel, and manages trade-offs between speed and cost. Belfer also names branch prediction as a possible latency optimization, without specifying its implementation. AI21 Maestro brings these two responsibilities together: a planner chooses how to proceed, while a smart execution engine manages the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn requirements into a search and repair process
Maestro's simplified instruction-following example starts by separating the context and task from the requirements. The requirements include no more than ten paragraphs, a formal tone, and positive mentions of the brand. These are the same conditions a developer might otherwise bury in the prompt; making them explicit makes them easier to validate.
The paragraph limit illustrates the benefit of a separate check. For text whose paragraphs are separated by blank lines, a small Python validator can turn that requirement into a concrete result:
python
import re
def check_paragraph_limit(text: str, maximum: int = 10) -> dict:
paragraphs = [
part for part in re.split(r"\n\s*\n", text.strip())
if part.strip()
]
return {
"requirement": "paragraph_limit",
"maximum": maximum,
"observed": len(paragraphs),
"passed": len(paragraphs) <= maximum,
}
Tone and the sentiment of brand mentions require different checks: counting paragraphs does not establish that either semantic requirement was met. Separating requirements gives each condition a place in validation.
The execution graph then turns generation into a search over possible answers. At each stage, the planner and execution engine choose candidates worth continuing. Best-of-N supplies diversity: generate several high-temperature samples from one model, or obtain candidates from different models, instead of relying on a single answer.
The remaining loop controls where additional effort goes:
- Discard candidates that do not look promising.
- Continue with the best candidate or candidates within a predefined budget.
- Validate those candidates, fix failures, and repeat the improvement process.
The diagram's branches represent successive candidates and repairs, not a requirement to exhaust every possible path. Candidate selection concentrates the available budget on answers that appear worth improving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a path, then reduce the results
For a more complex query, the same architecture chooses among alternative execution tracks. The execution engine estimates expected cost, latency, and success probability for those tracks, and the planner uses the estimates to select a path. The displayed graph expands the choices across retrieval parameters, model choices, and model parameters.
At the end, a reduce step turns the accumulated results into an answer. It can select the best result or combine several results into a complete response to the original query. The highlighted routes in the final diagram converge toward validation and the complete answer: branching creates alternatives, while reduction brings the work back to one deliverable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Higher quality costs additional inference
Belfer reports improved IFEval results using Maestro's planner and execution engine with GPT-4o, Claude Sonnet 3.5, and o3-mini. IFEval evaluates instruction following through verifiable requirements. The spoken comparison gives no numerical scores, model snapshots, scoring variant, or inference budgets.
On an internal requirement-satisfaction dataset built from customer data, Belfer also reports improvement over a single LLM call, at the cost of more runtime and money. The internal evaluation is distinct from IFEval, and the talk provides no numerical cost or latency measurements. The engineering trade-off is explicit: candidate generation, validation, and repair buy quality by doing additional work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use planning when the task requires it
The practical ending is an adoption ladder, not a mandate to put a planner around every model. Start with the simplest system that works. If a small language model is sufficient, use it. If a model with tools handles the task, stop there. If ReAct works, use that.
Move to a planner and execution engine when the task's complexity requires that additional coordination. Belfer closes by inviting developers to the Maestro waitlist—the access path offered in the recording—to see whether the system fits their needs. The decision is whether explicit planning, validation, and controlled execution solve a problem that the simpler architecture cannot.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
OpenAI’s original announcement explains instruction tuning and includes the child-friendly moon-landing example.
The original paper on interleaving reasoning and actions during interactive tasks.
Explains an orchestrator that maintains task progress, delegates work, and revises plans.
Hugging Face’s introduction to agents that express actions as executable code, with examples.
Introduces IFEval and its use of automatically verifiable instructions to evaluate model outputs.
Further reading
Yuval Belfer explains Maestro’s candidate search, validation, repair, execution budgets, and evaluation methods.
Google Research’s implementation and dataset for instruction-following evaluation.
Updates since the talk
Documentation for defining requirements, setting budgets, and inspecting validation reports.
AI21’s July 2025 announcement making Maestro available to enterprise customers.
Read the complete timestamped transcript
- 0:00
The year is 2022. OpenAI just released InstructGPT, which is the first model who can take instructions and actually follow them, really do what you want it to do. They have even called their model InstructGPT.
- 0:16
So how come three and a half years later, in 2025, when they release GPT-4.1, they still have trouble to follow instruction? And it's not just them, right? Every language model has some problems with following instructions.
- 0:34
And why is that? So if you remember the good old days of InstructGPT, this was one of the examples that blew everybody's mind, right? Explain the moon landing to a [REDACTED:age].
- 0:45
We got something nice, and everybody went bananas. And then with time, more and more developers started to use language models, and now the instruction looks more something like this,
- 0:58
or this, or this. Right? And that's what everybody's doing, just taking all the information, all the context, all the constraints, all the requirements, and just put everything in one prompt and hope for the better.
- 1:13
And really, even for simple tasks such as instruction following, LLMs alone are no longer enough, and this is where AI agents come to play. And agents don't just use prompting, they need planning.
- 1:27
I'm Yuval. I work at AI21 Labs, where we developed our own language models and also an agentic framework, and I want to talk to you about planning. So let's maybe start first with what is an agent.
- 1:40
There's a philosophical debate, everybody's wondering, and the truth is, as an engineer myself, I don't care. Whatever works, just make it work, call it an agent, call it a workflow, call it whatever.
- 1:54
Some people call LLM as a router an agent, where we have some sort of a routing model that just takes any query and directs it into a specialized LLM.
- 2:05
Is, is this an AI agent? Who cares? We also have function calling, right, where essentially we're providing the LLM a list of external tools that it can use and let it interact with other APIs, with the world, Google Search, uh, where this is something that now a lot of people talk about MCP, where it standardize this and
- 2:27
everything MCP related is an agent. Again, doesn't matter, as long as it works, and there's a lot of information about MCP right now, so we will not get into it.
- 2:36
And maybe the most popular agent, uh, of it all is ReAct, Reasoning and Action, where this is a framework that can be done with any language model in the framework of thought, then act upon that thought, and observe, and so on, and so on.
- 2:52
Every time the model wants to do some sort of a step towards, uh, the solution, but this is something that happens each step at a time, and there is no look ahead to the entire plan.
- 3:04
Right? This is kind of like part of a plan. I know the situation right now, I know what the next step I want to go to, and so on, and so on.
- 3:12
And what is planning that, that we're talking about, that every AI agent essentially needs to do? So it's figuring out the steps that you have to take to reach your goal.
- 3:24
And when do you want to use it is when there is no straightforward problems, essentially complex tasks, and that it requires parallelization and explainability, right? Rather than ReAct, where you can't really understand what happened, you can just see all the thoughts that happened throughout, but you don't really know why.
- 3:43
You can talk about forms-based, uh, planners, such as text bl- based, as Agentic 1 by Microsoft, or code-based planners by Small Agents, uh, from Hugging Face.
- 3:56
What do you mean when I say dynamic planning, is that there is an option to do replan. And essentially, it's not just have one plan and go all the way in this plan.
- 4:06
In the middle, you can ask yourself, "Mm, is this plan good, or should I do replanning and go somewhere else?" And when we talk about efficiency, we want to put smart execution into that.
- 4:18
Essentially, every planner need an execution engine because an execution engine can analyze dependencies between steps, which enables paral- parallel execution, and can also have some sort of a trade-off between speed and cost.
- 4:34
For instance, you can use branch prediction for s- uh, faster systems. Let's take a look of a simplified version of AI21 Maestro, which is a system that uses both a planner and smart execution engine.
- 4:49
So you can see here the very simple case of instruction following. Really, we have a prompt, which is essentially some context and the task, and the requirements are all those, uh, shouldn't be more than 10 paragraphs, the tone should be formal, all mentions of our brand should be positive, and so on, and so on.
- 5:08
The thing that you usually just shove in the prompt, now they are separated, so it will be easier to validate. And you can see that in every step here, we have some sort of a, an execution tree or execution graph, if you will, where in every step the, uh, planner and the execution engine chooses several candidates and
- 5:27
only continue to fix and improve those who seem promising. And really, there are several techniques here that you can use for any system, such as using Best-of-N, where essentially instead of one generation, we sample from any LLM with high temperature, several generations, or maybe use different LLMs.
- 5:49
We also ditch the candidates that do not seem like they will be good and only pursue the best candidate or candidates based on predefined budget. And also, we have some sort of validation, so iterations, and keep fixing, and so on, and so on.
- 6:05
Here is how it looks in something where it may be a bit more complicated, where again we have the input, the original, uh, input of the system, and then we have those, uh, tracks where the execution engine can say what is the expected cost, the expected latency, latency, and success probability, and the planner will choose accordingly the
- 6:27
right path to go. Where at the end, we will have some sort of a reduce, where we will have a lot of, uh, results, and we want to take the best one or combine all of them for a complete answer to the query that we had in, in the beginning.
- 6:41
And you can see that it actually works. Right? You can actually see that for IFEval, for instance, uh, we get really high results for GPT-0, 4o, or Claude Sonnet 3.5, or o3-mini.
- 6:54
We're getting very, very high results by just improving them and using our planner and smart execution engine. Also, for requirement satisfaction, which is a data set that we build internally based on customer data, again, you can see that we improve upon just a single call for LLMs, and we are paying a little bit here, uh, with, right,
- 7:14
more runtime and more money, but we get higher quality.
- 7:20
So this is really the takeaways here, that LLMs alone are not always enough, even for what we call simple tasks, such as just instruction following. I will always say start simple and then use whatever work.
- 7:32
So if you can use SLMs, use SLMs. If you can just take a model and put, uh, some tools into it, have at it, that's great. Use ReAct if you can.
- 7:42
And if you really need, if your task is so complex you have to use planning and execution e- engine, this is where you should go and explore, and I do invite you to join the Maestro waitlist to check it out and see if it works for you.
- 7:57
Thank you.