# UX Design Principles for (Semi) Autonomous Multi-Agent Systems

Victor Dibia, Microsoft Research

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/fmZWvE7yDZo-ux-design-principles-semi-autonomous-multi-agent) · [Video](https://www.youtube.com/watch?v=fmZWvE7yDZo)

## UX Design Principles for (Semi) Autonomous Multi-Agent Systems

From LIDA’s visualization pipeline to live Blender control, Victor Dibia shows how to build useful agent tools, evaluate their behavior, and keep users in control.

Familiarity with LLM tool calling is helpful; no prior Blender or AutoGen experience is required.

### From a data file to a useful visualization

[0:15](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=15s)

What should happen between dropping a data file into a web interface and receiving a useful visualization? The interface must help turn an open-ended request into concrete work. Victor Dibia approaches that problem through human–AI collaboration: he introduces himself as a principal research software engineer at Microsoft Research, with experience on GitHub Copilot, AutoGen, and AutoGen Studio. Earlier work at Cloudera and IBM Research combined machine learning with human-computer interaction.

In [LIDA](https://aclanthology.org/2023.acl-demo.11.pdf), a project Dibia dates to approximately August 2022, the user drops in a CSV or JSON file. The system summarizes the data, proposes questions about it, and generates visualization code for those questions. It then executes the code, performs post-processing and error recovery, and returns charts. This is already an agentic workflow: the useful result depends on a sequence of model decisions and executable operations, rather than a single text response.

The pipeline separates **summarization**, **goal exploration**, and **visualization generation**, with a code interpreter inside the visualization stage. Once charts exist, a diffusion model produces more varied visual representations of the data. The slide’s four modules—Summarizer, Goal Explorer, Viz Generator, and Infographer—make those responsibilities visible.

![Four connected modules labeled Summarizer, Goal Explorer, Viz Generator, and Infographer, with a Cars.csv input and green chart outputs.](https://ai.engineer/api/talk-essay-media/7be9124a07758a515ba343597b1cc4d3b2858ffad172b6416683a864d7f3f17a)

LIDA’s workflow moves from data summarization to goal exploration, visualization generation, and infographics.

Dibia reports that LIDA’s error rate fell from about 20% with **DaVinci Codex** models to about 1.5% after tuning with **GPT-3.5 Turbo**. He describes the improvement as occurring roughly three months later, but does not specify the evaluation set or error definition for this comparison. The practical lesson is feasibility: a pipeline that initially required substantial recovery became more usable as its model component improved, and similar capabilities subsequently appeared in Microsoft and other products.

### From authored workflows to agent teams

[3:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=190s)

LIDA’s stages were deliberately assembled into a workflow. The next question was whether developers could instead define agents that exchange messages and organize their work around a problem. That became the premise of AutoGen, a framework for building multi-agent applications. Dibia presents AutoGen as having about 45,000 GitHub stars at the time of the talk.

![AutoGen slide showing its GitHub URL, 45k stars, MIT licensing, Azure AI Foundry support, and a repository screenshot.](https://ai.engineer/api/talk-essay-media/fbc64804e3daa0a96efb815e7a81342d86121234df87e8c5a97a9808b298eff4)

AutoGen is presented as an open-source framework for building agentic AI applications.

AutoGen Studio exposes that composition through a web interface: create a team, drag agents into it, and configure the models and tools those agents use. The developer works with explicit building blocks instead of wiring every interaction by hand. Rather than spending the rest of the talk touring Studio, Dibia switches to a new application built from scratch, so that the implementation choices and their UX consequences can be examined together.

### Why Blender needs an observe-and-act loop

[4:29](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=269s)

[BlenderLM](https://github.com/victordibia/blenderlm) is a framework-free multi-agent prototype for translating natural-language requests into 3D artifacts. A user might ask for a scene containing a ball on a table; the application handles the operations needed to produce it in Blender.

The motivation comes from learning Blender itself. Dibia estimates that working through roughly four hours of the Donut Tutorial can take 40–50 hours. The learner must discover where controls live while also learning the underlying 3D concepts. Natural language offers a way to express the desired artifact without first mastering every operation. The polished example he displays is an aspiration, however: he explicitly says the prototype has not reached that quality.

The control architecture depends on how much of the solution is known in advance.

| Approach | How work proceeds | Useful when |
| --- | --- | --- |
| Fixed workflow | Predetermined steps, function calls, structured outputs | The solution can be expressed ahead of time |
| Exploratory agent | An LLM chooses actions, inspects results, and chooses again | The next step depends on the changed environment |

Fixed workflows can be reliable and valuable in production. Blender introduces a different challenge: taking an action changes the application state, and the system must react to what actually happened. In the exploratory approach, tools and observations become part of the control loop.

That choice creates three requirements. The system needs enough autonomy to address different tasks, must handle tool calls with side effects or unexpected results, and must break complex requests into steps that may run for an extended period. These are also the reasons a chat box alone is an incomplete interface: the user needs to understand and control work that continues after the initial request.

### A live request, a changing scene, and a verifier

[7:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=470s)

The BlenderLM web application connects to a running Blender instance over a **WebSocket**. It also exposes fixed tools that a person can invoke directly. Dibia begins by clearing the scene with a button; the socket connection streams the changed scene back into the web interface. This establishes a visible connection between an operation and its effect before any agent starts working.

He then selects an example requesting two balls with a shiny, glossy silver finish. Activity starts streaming immediately: the system announces task analysis, then displays a plan. The planner proposes adding a ground plane, creating spatially separated spheres, and assigning the silver material. The first visible change is the plane appearing in Blender. **The plan describes intended work; the viewport shows executed work.**

After tool execution, the application streams another update and runs a verification step. A verifier uses a scene snapshot, a log of the scene’s contents, and a visual representation to judge whether the system is progressing or stalled. That judgment influences the next action. The demonstration reaches a ball that Dibia can inspect and manipulate in Blender; he moves on before establishing completion of the entire requested two-ball scene.

### Build the environment before the agent

[10:07](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=607s)

**Start with a goal and a working baseline, not an agent definition.** Dibia’s construction order makes each new layer depend on something already testable:

1. Define the goal.
2. Build a baseline without AI.
3. Implement the tools a person would need to accomplish the task.
4. Define an evaluation test bed.
5. Add the agent.

For BlenderLM, the goal is to translate natural-language tasks into 3D artifacts. Before attempting that translation, the application needs to prove it can create an artifact at all.

The baseline is Blender’s equivalent of Hello World: run a script that adds a single cube. In Blender’s Python environment, the essential operation can be expressed directly:

```python
import bpy

bpy.ops.mesh.primitive_cube_add(location=(0, 0, 0))
cube = bpy.context.object
cube.name = "HelloCube"
```

A Blender add-on and a client library supporting socket connections provide the surrounding integration. This separates a basic question—can the application send an operation and observe its effect?—from the later question of whether a model chooses the right operation. It also gives the developer a fast path for prototyping and testing.

Tools then determine the agent’s practical reach:

- **Task-specific tools:** Perform a bounded operation, such as creating a Blender object.
- **General-purpose tools:** Execute arbitrary code generated by the model, allowing a much broader range of Blender operations.

The broader tool expands capability, but it also expands what the system must inspect and control. Dibia recommends spending about 50% of development time on tools, because an agent’s capabilities are bounded by the tools it receives.

### Expand the loop through evaluation

[11:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=710s)

Evaluation infrastructure grows in stages. First, a Jupyter notebook lets the developer exercise the code and tools directly. Next, an interactive web UI exposes the evolving scene and the application’s behavior. A third proposed stage adds automated test suites, metrics, and an evaluation harness. The demonstrated interface belongs to the second stage; the automated suite is described as a next step.

![Eval Test Bed slide beside stacked colorful blocks, listing a Jupyter notebook, a React web UI with viewport capture and streaming, and automated test suites, metrics, and an evaluation harness.](https://ai.engineer/api/talk-essay-media/3177338231b01d5b0239c49912922c7c3f6304e46c78d9f72bcfa2ae80f5bb7a)

The evaluation test bed progresses from a notebook to an interactive web UI and automated evaluation.

Only then does the agent enter the picture. The base implementation is an LLM in a tight loop with function calls, continuing until it returns a final result. For a stateful task such as scene construction, Dibia finds that this alone is insufficient. He adds a **verifier** that examines scene contents and object lists after each step, asking whether progress has been made and whether the user’s task is complete. A **planner** then breaks the request into atomic steps for the execution loop to handle.

These roles make the application easier to reason about: execution changes the scene, verification evaluates the resulting state, and planning organizes the remaining work. They also expose the UX problem more clearly. The user is supervising an ongoing process with several responsibilities, not merely waiting for a response. Dibia offers four design principles for that situation as provisional guidance in an early field, rather than an exhaustive specification.

### Make capability, progress, and control visible

[13:45](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=825s)

**Capability discovery** starts with the distinction between everything an agent might attempt and the smaller set of tasks it can perform reliably. Expose that dependable set through capability presets or example requests, as BlenderLM does. Suggestions can also depend on context: inspect the currently open scene and propose useful next operations. The interface helps users choose a productive task before execution begins.

**Observability and provenance** make the ongoing work intelligible. Stream activity logs so users can understand what the agent is doing, and provide debugging tools when behavior goes wrong. Token usage and time taken for individual activities help users explain a slow or expensive run. A changing viewport shows the artifact; the activity trace supplies the accompanying account of the work.

**Interruptibility** gives users a way to act on that information. An agent may take the wrong route, approach a mistake, or consume resources the user did not intend to spend. The system should support pausing, checkpointing, rollback, and resuming. Those capabilities let a person stop an unwanted trajectory and recover useful work instead of treating the whole run as an indivisible operation.

**Cost-aware delegation** addresses the consequences of the next action. Dibia argues that tool calls are effectively equal from the LLM’s perspective unless the system adds a distinction. Python that adds an object to a scene and code that attempts to delete the operating system cannot receive the same treatment. An inspection module should estimate an action’s cost or risk and determine when to involve the user. Here, cost includes destructive consequences, not just tokens or elapsed time.

### When multiple agents earn their complexity

[15:46](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=946s)

More agents and more autonomy also create more opportunities for error. Before adopting them, inspect the problem and ask whether the benefit justifies the added complexity. Dibia illustrates this with a large circle representing engineering tasks and a much smaller circle representing tasks that truly benefit from a multi-agent approach. The drawing expresses selectivity, not a measured proportion.

Four suitability questions make that judgment more concrete:

- **Planning:** Can a high-level request be meaningfully decomposed into steps that move it from unsolved to solved?
- **Different perspectives:** Does the task benefit from distinct responsibilities or personas, such as planning and code execution?
- **Extensive context:** Would it help to distribute large inputs—such as application snapshots and screenshots—to separate agents that return results to a coordinator?
- **Adaptation:** Do actions change the environment enough that the system must repeatedly reconsider what to do next?

Blender scene construction supplies reasons for each: a desired artifact needs a plan, execution and verification have different responsibilities, scene observations accumulate, and every operation changes the state subsequent operations must use.

**Task-specific evaluation should decide whether those extra agents stay.** Define the task and its metrics, establish a non-agent baseline, and improve iteratively. BlenderLM followed that sequence from a simple execution loop to a verifier and then a planner. Dibia reports that his interactive evaluation tool showed improvements that justified the additions, although he supplies no quantitative comparison here. Academic benchmarks can inform a design, but they do not replace evaluations of the application’s actual task.

### Carry the design into the implementation

[18:58](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1138s)

The closing design standard is a user who can discover suitable work, follow its progress, interrupt and restart it, and receive control when an action’s risk warrants intervention. Supporting that experience requires implementation work beyond the agent loop. Building BlenderLM from scratch made the pieces visible, but Dibia describes it as labor-intensive and recommends considering a framework to reduce that burden.

For continued study, he points to work on AutoGen Studio, Magentic-One, [Magentic-UI](https://www.microsoft.com/en-us/research/blog/magentic-ui-an-experimental-human-centered-web-agent/), and Challenges in Human-Agent Communication. He closes with his then-in-progress book, [Designing Multi-Agent Systems](https://victordibia.com/book/), whose third chapter focuses on design, and notes that the BlenderLM source code is available. The example leaves readers with both an implementation to examine and a concrete question to apply to their own systems: what does the user need to see or control while the agent is acting?

![Thank You slide with a Chapter 3 book pointer, multiagentbook.com, github.com/victordibia/blenderlm, and a large QR code.](https://ai.engineer/api/talk-essay-media/e89b7b2f2b4d31079bdfbfbf4bd8667bf833c47e813fdffd0d43c8283e69e413)

Closing resources include multiagentbook.com and the BlenderLM GitHub repository.

## Resources

- [UX Design Principles — talk slides](https://victordibia.com/pdf/victordibia_agentux.pdf) — The June 5, 2025 presentation deck, including the construction sequence, UX checklist and further reading.
- [4 UX Design Principles for Autonomous Multi-Agent AI Systems](https://newsletter.victordibia.com/p/4-ux-design-principles-for-multi) — Dibia's companion explanation of BlenderLM, evaluation-first development and the four UX principles.
- [BlenderLM](https://github.com/victordibia/blenderlm) — Source for the sample application that lets language models control Blender. The author warns that it may not receive ongoing maintenance.
- [LIDA — original research paper](https://aclanthology.org/2023.acl-demo.11.pdf) — The ACL 2023 paper explains LIDA's visualization pipeline, code-generation evaluation and infographic generation.
- [Magentic-UI: a human-centered web agent](https://www.microsoft.com/en-us/research/blog/magentic-ui-an-experimental-human-centered-web-agent/) — Microsoft's launch article explains collaborative planning, user intervention, action approval and reusable plans.
- [Designing Multi-Agent Systems](https://victordibia.com/book/) — Dibia's book on agent design and implementation; chapter three covers UX principles for multi-agent systems.

## Slides

### How I got into agents

Dibia traces his entry into agents to LIDA, a project he recalls working on around August 2022 that accepted CSV or JSON data through a web interface.

[1:22](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=82s) [Section slide titled “How I got into Agents.”](https://assets.ai.engineer/sites/aiecode2025/sha256/8448f3a8c9607a1124f66fe00986d221cb47c8441f005895be0ee479759b1fe1.webp)

How I got into
Agents ..

Interface text:
AIE
Microsoft
smol ai

### LIDA data upload

LIDA shows seattle-weather.csv and is generating a summary. The speaker describes how it then generates questions, runs code and displays visualizations.

[1:56](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=116s) [LIDA demo showing visualization settings and a data upload area.](https://assets.ai.engineer/sites/aiecode2025/sha256/4da154ebb3f96833637d8955651425ea0c6dc00eb2ba02d55d54dec8c04772fb.webp)

LIDA [Beta]
Generate visualizations from data

Lida provides a conversational interface for automatic generation of grammar-agnostic visualizations (and infographics) from data!.
Currently being developed at MSR! (Work in Progress.) Learn more.

Select a visualization library/grammar
Seaborn

Generation Settings
Model: gpt-3.5-turbo-0301, n: 1, number of Temperature: 0 ...

Ready? Upload a file to begin.
Click or drag file to this area to upload
Upload .json or .csv files to generate a visualization.
seattle-weather.csv

Don't have data? Try any of the files below
cars.json
wheat.json
movies.json
seattle-weather.csv
sp500.csv

[illegible] data, generating summary ..

Interface text:
About LIDA
Gallery
Demo
AIE
Microsoft
smol ai

### LIDA visualization workflow

LIDA summarizes a dataset, proposes visualization goals, and generates and runs chart code. An infographer then turns the chart into a stylized image.

[2:03](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=123s) [Diagram of LIDA’s summarizer, goal explorer, visualization generator, and infographer.](https://assets.ai.engineer/sites/aiecode2025/sha256/9330c59709808a78edd38eb805fa8eac0e563d7f3d3a9db3b12222e7895311e7.webp)

LIDA: Automatic Generation of Visualizations
https://microsoft.github.io/lida/

Cars.csv

SUMMARIZER — RULES + LLM
Convert datasets into a rich but compact natural language representation (context).
Output:
“The cars dataset contains technical specifications for cars and has 9 fields - Name, Miles_per_Gallon, Cylinders, Displacement, Horsepower, Weight_in_lbs, Acceleration, Year, Origin ..”

GOAL EXPLORER — LLM
Generate a set of potential “goals*” given the dataset context.
Output:
• Histogram of Miles per gallon
• Plot of miles per gallon vs horse power
• Trends in miles per gallon over time
• Average horsepower per country

* goals may also be directly provided by the user. Supports multi-lingual input.

Generate code in visualization based on context and goal

VIZ GENERATOR — LLM
Generate, evaluate, repair, filter and execute visualization code to yield specifications*.
Output:
Y-axis: Count of Records
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
X-axis: Miles_per_Gallon (binned)
X-axis ticks: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50

* specification may be in any programming language or grammar.

INFOGRAPHER — IGM
Generate stylized infographics based based on visualization and style prompts.
Output:
X-axis: Miles_per_Gallon (binned)
X-axis ticks: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50

* Style prompt: line sketch art, line drawing

Interface text:
AIE
Microsoft
smol ai

[2:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=130s) [Diagram showing dataset summarization, goal exploration, visualization generation, and infographic generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/d79b8d83868f30ca8396b88b0a2db649e798ad850dd32a6dd45690d05a45bc98.webp)

LIDA: Automatic Generation of Visualizations
https://microsoft.github.io/lida/

Cars.csv

SUMMARIZER — RULES + LLM
Convert datasets into a rich but compact natural language representation (context).
Output:
“The cars dataset contains technical specifications for cars and has 9 fields - Name, Miles_per_Gallon, Cylinders, Displacement, Horsepower, Weight_in_lbs, Acceleration, Year, Origin ..”

GOAL EXPLORER — LLM
Generate a set of potential “goals*” given the dataset context.
Output:
• Histogram of Miles per gallon
• Plot of miles per gallon vs horse power
• Trends in miles per gallon over time
• Average horsepower per country

* goals may also be directly provided by the user. Supports multi-lingual input.

Generate code in visualization based on context and goal

VIZ GENERATOR — LLM
Generate, evaluate, repair, filter and execute visualization code to yield specifications*.
Output:
Y-axis: Count of Records
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100
X-axis: Miles_per_Gallon (binned)
X-axis ticks: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50

* specification may be in any programming language or grammar.

INFOGRAPHER — IGM
Generate stylized infographics based based on visualization and style prompts.
Output:
X-axis: Miles_per_Gallon (binned)
X-axis ticks: 5, 10, 15, 20, 25, 30, 35, 40, 45, 50

* Style prompt: line sketch art, line drawing

Interface text:
AIE
aws

### AutoGen

AutoGen lets developers define agents that exchange messages and self-organize to explore a problem. The slide describes it as an MIT-licensed framework for agentic AI applications, supported in Azure AI Foundry.

[3:08](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=188s) [Slide describing AutoGen as an open-source framework for agentic AI applications, alongside its repository page.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8e18a6f925c54d092549fa281e23b24b40986a13cea60df7b85e308314a3748.webp)

6
AutoGen
https://github.com/microsoft/autogen
45k stars

An open-source (MIT) framework for building agentic AI applications. Supported in Azure AI Foundry

2023

Repository screenshot:
AutoGen

Important
• (10/13/24) Interested in the standard AutoGen as a prior user? Find it at the actively-maintained AutoGen 0.2 branch and autogen-agentchat~=0.2 PyPi package.
• (10/02/24) AutoGen 0.4 is a from-the-ground-up rewrite of AutoGen. Learn more about the history, goals and future at this blog post. We're excited to work with the community to gather feedback, refine, and improve the project before we officially release 0.4. This is a big change, so AutoGen 0.2 is still available, maintained, and developed in the 0.2 branch.

AutoGen is an open-source framework for building AI agent systems. It simplifies the creation of event-driven, distributed, scalable, and resilient agentic applications. It allows you to quickly build systems where AI agents collaborate and perform tasks autonomously or with human oversight.

• Key Features
• API Layering
• Quickstart
• Roadmap
• FAQs

AutoGen streamlines AI development and research, enabling the use of multiple large language models (LLMs), integrated tools, and advanced multi-agent design patterns. You can develop and test your agent systems locall[illegible] then deploy to a distributed cloud environment as your needs grow.

Interface text:
README
Code of conduct
CC-BY-4.0 license
MIT license
Security
AG
Follow @pyautogen
AIE
Microsoft
smol ai

[3:24](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=204s) [AutoGen overview describes an open-source framework for agentic AI applications, alongside a GitHub README screenshot.](https://assets.ai.engineer/sites/aiecode2025/sha256/f275efdc2f79d2ac77dd39a181a2a12bc9b25ad91f1f0518a1e086e331f5f9b1.webp)

6
AutoGen
https://github.com/microsoft/autogen
45k stars

An open-source (MIT) framework for building agentic AI applications. Supported in Azure AI Foundry

2023

Repository screenshot:
AutoGen

Important
• (10/13/24) Interested in the standard AutoGen as a prior user? Find it at the actively-maintained AutoGen 0.2 branch and autogen-agentchat~=0.2 PyPi package.
• (10/02/24) AutoGen 0.4 is a from-the-ground-up rewrite of AutoGen. Learn more about the history, goals and future at this blog post. We're excited to work with the community to gather feedback, refine, and improve the project before we officially release 0.4. This is a big change, so AutoGen 0.2 is still available, maintained, and developed in the 0.2 branch.

AutoGen is an open-source framework for building AI agent systems. It simplifies the creation of event-driven, distributed, scalable, and resilient agentic applications. It allows you to quickly build systems where AI agents collaborate and perform tasks autonomously or with human oversight.

• Key Features
• API Layering
• Quickstart
• Roadmap
• FAQs

AutoGen streamlines AI development and research, enabling the use of multiple large language models (LLMs), integrated tools, and advanced multi-agent design patterns. You can develop and test your agent systems locall[illegible] then deploy to a distributed cloud environment as your needs grow.

Interface text:
README
Code of conduct
CC-BY-4.0 license
MIT license
Security
AG
Follow @pyautogen
AIE
Microsoft
smol ai

### AutoGen Studio component library

AutoGen Studio places a component library beside the team canvas. The speaker describes dragging agents into a team and composing them with models and tools to build multi-agent applications.

[3:56](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=236s) [AutoGen Studio component library lists agents, models and tools beside a team-building canvas.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d79538861a23d144994e22a95cc7abb92d865efaeaf239c42e38eeac3deab99.webp)

Component Library
Drag a component to add it to the team
Search components...

Agents (4)
AssistantAgent
MultimodalWebSurfer
AssistantAgent
UserProxyAgent

Models (2)
Tools (6)
[illegible]inations (3)

AGENTS (1)
weather_agent
Drop agents here

TERMINATIONS
termination
Drop termination here

TOOLS
get_weather
Drop tools [illegible]

Interface text:
team and this just gives you a basic
AIE
Microsoft
smol ai

### But this is ai.engineer

The speaker pivots from explaining AutoGen Studio to building something from scratch, echoing the slide’s hard hat and invitation to learn by building.

[4:16](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=256s) [Slide says “But this is ai.engineer” and “we are all here to learn how to build,” beside a yellow hard hat.](https://assets.ai.engineer/sites/aiecode2025/sha256/e6b1ddb7aacaa28c8a0c8bf21c7e2cfea0f9fb0018988be7f0cf24c9578bc977.webp)

But ..
this
is
ai.engineer

we are all here to learn how to build!

Interface text:
AIE
Microsoft
smol ai

### Let's build something from scratch

The speaker sets aside an AutoGen Studio walkthrough to build a multi-agent system from scratch for the talk.

[4:21](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=261s) [Slide proposes building a multi-agent system from scratch, beside a yellow hard hat.](https://assets.ai.engineer/sites/aiecode2025/sha256/7768b328e8ea7cb7df61201e0c51a49c1d90a1055fb9aaa66cadfe4db46b6086.webp)

Let’s build something from scratch!

A multi-agent system from scratch!

AIE
aws

### BlenderLM: a multi-agent system for 3D tasks

BlenderLM is a multi-agent system built from scratch without frameworks to help create 3D scenes in Blender from requests such as a ball on a table.

[4:27](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=267s) [BlenderLM title slide describes a multi-agent system for 3D tasks and displays its project link.](https://assets.ai.engineer/sites/aiecode2025/sha256/27d7cd6e33efe465d45bb205a047e953b99e430edfc825c80eac73b6464f5e72.webp)

BlenderLM - a multi-agent system for 3D tasks

https://github.com/victordibia/blenderlm

Interface text:
AIE
aws

[4:30](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=270s) [Title slide introducing BlenderLM, with a 3D cube icon.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed9fbeeeda928cbced8692f371ffd5dfab1a593a5d52ed29af6e8ffd72384753.webp)

BlenderLM - a multi-agent system for 3D tasks

https://github.com/victordibia/blenderlm

Interface text:
AIE
Microsoft
smol ai

### BlenderLM scene execution plan

BlenderLM shows a two-step plan to set up lighting and a camera, then create two glossy spheres. The adjacent Blender viewport shows two spheres above a ground plane.

[4:37](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=277s) [BlenderLM shows a two-step execution plan beside a Blender viewport with two spheres above a ground plane.](https://assets.ai.engineer/sites/aiecode2025/sha256/6ee5d1623a93296cd072bbaeabdeed602ffb977b0c3935562f89b6876d784c85.webp)

Execution Plan
5.21s  1936 tokens  plan_generated

1. Set up the scene environment with a ground plane, complete 3-point lighting (adding fill and rim lights), and a properly positioned camera for clear composition.
2. Create two spheres with correct spatial separation, assign a glossy silver material to both, and position them above the ground plane.

Event: step_start  5.22s
Step 1/2: Set up the scene environment with a ground plane, complete 3-point lighting (adding fill and rim lights), and a properly positioned camera for

Interface text:
Blender LM
Connected to Blender
Blender Controls
Chat  Presets  Projects
Blender Viewport  Refresh
AIE
Microsoft
smol ai

### What is the plan for today?

The speaker plans to demo Blender LM, explain how it was built, and discuss UX design principles and takeaways.

[4:54](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=294s) [Agenda slide showing only the heading, What is the plan for today?](https://assets.ai.engineer/sites/aiecode2025/sha256/7653f6e073d8a372996ec1d5b60d5588a3b66e3223a4444dd553c58362defaa4.webp)

What is the plan for today?

12
12

Interface text:
AIE
Microsoft
smol ai

### What is the plan for today?

The speaker starts with a demo of a tool that turns a scene request into a Blender interface.

[4:57](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=297s) [Agenda slide adds Part 0: a demo.](https://assets.ai.engineer/sites/aiecode2025/sha256/64688f66f39af892cc6a4bf69799d9294f33805a84ab5597125621b59ed415b9.webp)

What is the plan for today?

- Part 0: I’ll show you a demo

12
12

Interface text:
AIE
Microsoft
smol ai

### What is the plan for today?

The speaker will demo the Blender tool, explain how it was built, and discuss design principles and takeaways for agentic applications.

[5:01](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=301s) [Agenda slide lists a demo and a walkthrough of how the tool was built from scratch.](https://assets.ai.engineer/sites/aiecode2025/sha256/d1e97db9de84092409db30aa5ba8a518aa6b5a58942f0dcc6076c60d8ff47852.webp)

What is the plan for today?

- Part 0: I’ll show you a demo
- Part 1: I’ll walk you through how it was built from scratch

Interface text:
12
AIE
Microsoft
smol ai
12

[5:05](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=305s) [Agenda covering a demo, how it was built, design principles, and takeaways; the camera inset partly covers the final line.](https://assets.ai.engineer/sites/aiecode2025/sha256/836b2770424e051368180b3dd90587b62e2fd68ecb77d558deeeaa19a5226798.webp)

What is the plan for today?

- Part 0: I’ll show you a demo
- Part 1: I’ll walk you through how it was built from scratch
- Part 2: We will discuss a set of design principles that you can apply to your own agentic applications.
[illegible]rt End: Takeaways

Interface text:
12
AIE
aws
12

### Create a low-poly well

The speaker explores an agentic workflow that turns a natural-language prompt into a scene like the pictured well. The prototype has not yet reached that quality.

[5:58](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=358s) [A prompt to create a low-poly well points to a rendered well.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f62d04da8b816388be1a6718163391cd8749c7e857e3339b78f0f8b4f50f06b.webp)

“Create a scene that contains a low poly well”

Interface text:
AIE
Microsoft
smol ai

### Building an agentic system

The speaker asks how to use a multi-agent workflow to turn natural language into a scene, while noting that the prototype has not yet reached the illustrated quality.

[6:06](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=366s) [The slide asks how to build an agentic system and shows a colorful light bulb.](https://assets.ai.engineer/sites/aiecode2025/sha256/2cac654372d98926f1411d5c5eba4423c15d6a4780cdf3a54f4ac3aa5ff22f24.webp)

How do you build an agentic system for this? ...

Interface text:
AIE
Microsoft
smol ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=0s) [upbeat music] Hi, everyone.

[0:15](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=15s) Um, my name is Victor Dibia. I'm a principal research software engineer at Microsoft Research, and, um, my background is mostly around, uh, human-AI experiences, so that's sort of what I'm interested in right now.

[0:27](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=27s) And over the last few years, I've been sort of looking at scenarios where a human works in tandem with an AI agent to solve problems. And so one of the things I've worked on at Microsoft Research is GitHub Copilot.

[0:37](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=37s) So how many of you have used GitHub Copilot? Excellent. And so I think, in my opinion, it's the first example of an AI model working at scale, um, in an IDE helping the developer solve a problem.

[0:48](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=48s) Okay, go ahead. You can switch that. And, um, beyond that, um, more recently, I spent my time working on an open source, um, multi-agent framework, something called AutoGen. Uh, how many of you have heard of AutoGen?

[1:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=60s) Okay, great. About half of the room. And as part of that, I've, I've also helped build out AutoGen Studio, which is, um, um, um, a low-code developer tool to help build out, um, multi-agent workflows.

[1:12](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=72s) Um, previously, I worked at Cloudera as a machine learning engineer, and more-- and before that, I worked at IBM Research as a research staff member focusing on, uh, human-computer interaction.

[1:23](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=83s) Okay, so how did I get into agents? So I'm just gonna give a really brief history. And so sometime in, um, I think August twenty twenty-two, so this was about four months before ChatGPT sort of took off, um, I had worked on a project called, um, LIDA.

[1:38](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=98s) And essentially what that tool did was that it let you, in a web user interface, drag in some data, a CSV or a JSON file, and it did a few things.

[1:46](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=106s) So first, it came up with a summarization of the data. Um, next, it did things like ask a bunch of questions regarding the data. And for each of those questions, we sort of generated code, executed it, did some post-processing, uh, error recovery, and then we showed the user a bunch of visualizations.

[2:03](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=123s) And so if you sort of look at it, it actually is an agentic workflow, um, and this just quite, quite early in this time. So it had these four main categories, summarization, goal exploration, visualization generation, which is essentially a code interpreter sort of built into the entire system.

[2:20](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=140s) And then once we got these data visualizations, we did things like use a diffusion model to sort of come up with more representations, more, uh, sort of diverse representations of that data set.

[2:32](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=152s) And so an interesting thing about the system was that, uh, the first version used the DaVinci Codex models. Uh, does anybody remember the OpenAI DaVinci models? That was a really long time ago.

[2:42](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=162s) And one of the interesting things was that once I showed it, and the error rate was about twenty percent, and then about three months later, uh, there was the GPT-3.5 Tu- Turbo models, and we sort of tuned that, and the error rate sort of went down to about one point five percent.

[2:57](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=177s) The key fun fact there was that, like, um, this sort of showed that these sort of applications were possible, and today I think you see a lot of these sort of, uh, capabilities across many Microsoft products and products b- uh, even beyond Microsoft.

[3:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=190s) And so fast-forward after that, um, a few colleagues started to think about, um, how can we, as opposed to building these handheld workflows, how can we instead build multi-agent applications where you define agents and they sort of exchange messages and self-organize to sort of explore a problem, a problem space?

[3:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=205s) And that's sort of where AutoGen sort of came about. Um, I'd encourage you guys to sort of look through it. It's a framework for building multi-agent applications. It's pretty well used 45K stars.

[3:35](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=215s) And I think the more interesting thing I did there was AutoGen Studio, which is, uh, kind of a nifty tool. It's a low-code tool again. But in this case, essentially what happens is that you sort of sign into a web interface.

[3:48](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=228s) You can then compose multiple agents. So for example, you create a team. You drag in a set of agents into that team. And then for each of those teams, you have primitives like models, tools, and you can sort of compose them together to sort of build multi-agent applications.

[4:03](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=243s) And so when I started to prepare this talk, I think one of the things I wanted to do was sort of walk you through all of the capabilities of AutoGen Studio, how we built it, the design philosophy behind all of that.

[4:13](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=253s) But I thought, you know, there's just a bunch of resources out there, and in general, this is AI dot Engineer. How about we go ahead and we build something from scratch, and we sort of show that today?

[4:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=265s) And so maybe you shouldn't do that, but we're gonna do that today.

[4:29](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=269s) And the tool I'm gonna show today is something called Blender LM. Um, it's a multi-agent system built from scratch, no frameworks, nothing. And the idea is that it's supposed to help you build, uh, uh, enable 3D sort of tasks.

[4:41](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=281s) So you could go to this tool, say things like, "Build like, I don't know, a scene with a ball on a table," and essentially it'll do all of the plumbing underneath and get you a Blender interface, uh, that sort of accomplishes that.

[4:54](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=294s) Anyone here familiar with Blender? Okay, great. Awesome. So here's the plan today. I have about thirteen minutes left. I'll show you a demo. I'll walk you through how I built it.

[5:05](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=305s) And then at the end, we'll sort of discuss or synthesize a bunch of design principles that underne- underpin, um, a good user experience for a tool like this. And then finally, we'll sort of, uh, settle on a bunch of takeaways.

[5:16](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=316s) Okay, let's go. In terms of background, how did I settle on Blender LM? So about two years ago, I sort of wanted to learn how to use Blender. And of course, um, if you've tried to use Blender, there's a really popular tutorial called Donut Tutorial.

[5:31](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=331s) So that's what you're sort of looking at. And it's, it's kinda deceptive because the tutorial takes about four hours, but at the end of the day, you need about forty or fifty hours just to get through the whole thing.

[5:41](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=341s) And so you're trying to learn, you know, where things, where do things live, how do you use the tool, and then you need to learn all of the concepts underneath.

[5:48](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=348s) And so one of the things I asked myself was, "Can I build with all that I know about agents, with all my experience building AutoGen, can I create an agentic workflow that will help me take-- go from natural language to, let's say, something that looks like this?"

[6:02](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=362s) The prototype is not at this level of quality, but I think it can get there. And so the next question is: how do you express this as a multi-agent workflow?

[6:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=370s) And you have a couple of options as a multi-agent system. So do you build a workflow? And I'm sure if you've been at this conference, you've seen people debate, um, all of the pros and cons between a fixed deterministic workflow, so essentially know exactly what are all the steps.

[6:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=385s) And this is great. We sort of use a lot of that in production today. You can build reliable systems, take advantage of things like function calling, structured output, and build really, really valuable systems.

[6:35](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=395s) However, it requires that you know the exact solution to the problem, and so what you're doing is that you're expressing that solution as a workflow. But there are class of problems, like the kind of thing we wanna address here, that you don't know the exact s- solution to the problem because every time you take an action, let's

[6:49](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=409s) say click something in Blender, the entire space changes, and you have to react to that in some way.

[6:55](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=415s) And so on the o- other hand, on the other end of the spectrum, what I'm gonna focus on today is more autonomous exploratory systems. And so what that means is that we are sort of looking at a system where an LLM sort of drives the flow of control.

[7:07](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=427s) We have tools, we take actions, we inspect the results, we observe, and then we make progress.

[7:13](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=433s) Okay, so three characteristics here that we should be, sort of have at the back of our minds. The system should have a bit of autonomy, so it might not address just a single task, maybe many different tasks.

[7:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=445s) It should be able to take actions, and an action here can have side effects. And so for example, you could try something, call a tool, and it could sort of return with a result that you don't expect, and your system should be able to handle, handle that.

[7:39](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=459s) And then finally, uh, you need to have systems that, uh, are expected to sort of explore complex tasks, break them down into steps, and then run for extended periods of time.

[7:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=470s) Okay, so let's switch to a quick demo. This is the Blender LM interface. It's the web application. And essentially what's going on here is that it's connected over a WebSocket connection to an actual Blender instance.

[8:02](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=482s) So this is, uh, Blender. This is a software tool for building 3D applications. And what we have here, the first thing you'll notice is that we have a set of fixed tools that the developer can use directly, and I'll tell you why we, we kinda need that in a second.

[8:15](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=495s) So for example, I could click a button to clear the scene. And because we have a socket connection, we can stream exactly what's going on in the Blender interface.

[8:24](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=504s) The sc- the, the scene is now clear, and we can show that to the UI in the UI.

[8:30](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=510s) Next, we can, um, let's say, go onto a list of pre-selected or predetermined examples here. And so maybe I might ask this system to create two balls with a shiny, glossy silver finish.

[8:43](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=523s) And essentially what we see here is that a bunch of activities start to occur. They're streamed to the UI in real time. So first it says we're analyzing the task.

[8:52](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=532s) If we scroll up just a bit, we've come up with a plan. There's some planning done. There's a planning agent underneath. Says w- the first step, we're gonna set up the scene environment by adding a ground plane.

[9:03](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=543s) Um, we're gonna create the spheres with correct spatial separation, assign a glossy silver finish and all of that. And we can see in real time the first thing it's done is, is put on, putting that plane.

[9:13](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=553s) If we look at Blender, we see the, uh, horizontal plane here, and all of this is running live. You probably shouldn't do live demos in a talk, but hey, you know, we're trying to be brave here.

[9:23](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=563s) And then it sort of explores, um, each time-- Essentially what's happening is that each time it takes a step, calls a bunch of tools, executes those tools, um, we stream the update to the user interface, and then we have a sort of verification loop.

[9:35](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=575s) There's a verify agent that sort of takes a snapshot of the scene, um, uh, an actual log of what's in the scene, a visual representation. We use an LLM.

[9:45](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=585s) We sort of judge are we making progress? Are we stalled? And then we sort of use that information to decide what we do next.

[9:52](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=592s) And we can see that we have a ball here, which is actually what we really, really wanna do, and we can look at that in Blender. We can tweak it around.

[10:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=600s) Hey, look, look at that, guys. It works. I think we deserve a little applause here. [clapping] Come on. There we go.

[10:07](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=607s) I wouldn't explore more. I have about eight minutes left. Let's move on. Um, so how is all of this built? Let's walk through the process really, really fast. Um, most of the time when people sort of think of a system like this, the first thing they probably will say is like, "Let's define the agent."

[10:23](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=623s) Um, that's really not what you should do first. First, let's define the goal. Pretty simple. Next, we need to come up with a baseline, probably has nothing to do with agents, nothing to do with, do with AI.

[10:32](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=632s) Create, just ensure everything works correctly. Third, we build out our tools. What tools does this a- agent need? If a human was gonna do this, what tools would they need to accomplish this?

[10:42](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=642s) And then n- not the agent yet. Next, we define a test bed. How do we evaluate how this thing works? And so--

[10:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=650s) And then finally, when you have all of that, that's when you then go ahead and build the agent. The step one is really simple. What we want here is we wanna translate natural language tasks to 3D artifacts.

[11:01](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=661s) Next, we create a baseline. We want a script that can say, "Let's build the 'Hello, World!' of Blender." Create a script. We run it. It adds a single cube to the scene.

[11:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=670s) What we need here is a Blender add-on. We need a client library that can enable socket connections, all of that. And then this is really, really valuable for rapid prototyping and testing.

[11:19](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=679s) Next, we need to define a set of tools, and there are two types of tools. There could be task-specific tools. For example, something directly just create a Blender object and do nothing else.

[11:28](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=688s) And then you might have, um, let's say general purpose kinda tool, which is something to execute arbitrary code. And so in this case, you get your LLM to generate code, and you execute that, and that's what drives all the capabilities on Blender.

[11:42](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=702s) And one thing to note is your agent is always only as good as the tools you give it. So spend a lot of time, about 50% of your time, on tools.

[11:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=710s) And you can test all of this in code. This is what that looks like. Next, you wanna build an eval test bed. In this case, it's three steps. V1 is just a Jupyter Notebook.

[11:59](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=719s) We're gonna w- write all that code, test it in Jupyter Notebook. Next, we create a full interactive web UI, which is the kind of thing I just showed here.

[12:08](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=728s) Um, And then third, we probably want to create an eval- automated test suite, so things with metrics and u- and a full evaluation on this. And then finally, um, to create your agent, the first thing you wanna do is to create a base agent loop.

[12:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=745s) If you've been at this conference, you know that an agent is mostly an LLM in a tight loop with a bunch of function calls. So you create that, you get its final result, and then you're fine.

[12:35](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=755s) But typically, for s- a problem like this, this is typically not enough, and you need to it- iterate just a little bit more. [clears throat]

[12:45](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=765s) Okay. You need to iterate just a little bit more, and in this case, we have two other agents. There's one called the verifier agent. What it does is that every time this agent takes a step, it'll sort of take this, the content of the scene, it'll take a list of all the objects there, and we'll sort of

[12:59](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=779s) use an LLM to predict, are we making progress, is user task completed, and then we decide how to move forward. The second agent you want here is a planner, and so you saw earlier, um, when the task came in, the planner sort of broke it down into atomic steps, and then each of the steps sort of addressed

[13:14](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=794s) in this sort of tight loop. So what can we learn from all of these? And so the, the design principles I'm gonna give you here, um, they're not exhaustive, they're not perfect.

[13:24](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=804s) And in fact, if you met someone that told you that they knew the exact design principles for multi-agent system design, you probably shouldn't trust them because the space is just too early for that.

[13:33](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=813s) So what I'm gonna try to give you today is a set of four high-level ideas that you can sort of take, and once you build this sort of system, sort of apply them to see how, um, you can use that to improve your own systems.

[13:45](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=825s) So the first is, the first, um, principle is capability discovery. So what you wanna do is that because you have an agent, it can do a whole bunch of things, but there are a few things that it can do with high reliability.

[14:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=840s) And so you saw earlier, I had this little sort of POs that showed that here are the things the agent can do. So you wanna itemize the kind of things that your agent can do with high reliability.

[14:09](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=849s) The second thing you can do is to have proactive suggestions based on user context. Let's say we have a scene open. You can sort of parse the scene and sort of suggest to the user some high-level things they can accomplish.

[14:21](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=861s) And so this is an example of that. The second thing is observability and provenance. And so stream all of the activity logs, help the user sort of make sense of what the agent is doing.

[14:31](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=871s) And then you wanna provide tools for debugging and all of that. So, so all the little things around, um, number of tokens used, the amount of time taken for each of that, very, very valuable for the user to make sense of what the agent is doing.

[14:44](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=884s) The third is interruptibility. So at this point, your agent is sort of taking all kinds of actions, and so at any time, you want to sort of design your system such that you can pause it.

[14:55](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=895s) It might be going down the wrong route, about to make a mistake, or sort of consume a bunch of resources that you don't intend. And so you want a system that enables things like checkpointing, rollback, pauses, and resumes.

[15:09](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=909s) And then finally, cost-aware delegation. So every time an agent takes, um, an action, um, from the LLM's perspective, all actions are equal, all tool calls are equal, unless you do something about it.

[15:22](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=922s) And so you might h- in m- in the case of Blender, you might have it write some Python code that, let's say, I don't know, adds something to the scene.

[15:30](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=930s) But for any reason, let's say it tries to delete the entire operating system, you really don't want things like that to happen. And so you want a module that actively inspects, inspects the, and tries to estimate the cost of the action, and then knows when to delegate to the user.

[15:46](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=946s) And so I'm kinda getting to the end. Um, what are some of the key takeaways? The first one is know when to use a multi-agent approach. And so a multi-agent approach is not always the best thing to use.

[15:58](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=958s) And essentially, when you have multiple agents collaborating and you give them a bunch of autonomy, um, as part of that process, you also increase the surface for error. And so like any other tool, you should sort of inspect the problem space and sort of verify if a multi-agent system is actually the right tool for the job.

[16:19](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=979s) I always show this little graph, and so you have a big circle here, and if the big circle is the task that you're engineering or most engineering teams need to do, then this small circle here at the bottom is the task that truly benefits from a multi-agent system approach.

[16:35](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=995s) Um, it's really, it's really that small, and, uh, before you try to build an autonomous multi-agent system, uh, just think very, very carefully and ensure that you have good ROI on, on, on, on, on, on this specific approach.

[16:49](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1009s) The next, the next question I get is, "How do I know if my task might benefit from a multi-agent approach?" I typically offer a five-step framework. Um, the first is planning.

[17:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1020s) Does the task benefit from planning? Can you take a high-level input from a user and meaningfully break it down to the bunch of steps that lead you from an unsolved state to a solved state?

[17:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1030s) Next, can you take the task and sort of break it into multiple perspectives or personas? In this case, um, let's say you might have some, some, some persona that explores things like just planning of the task, um, maybe some personas that handle, let's say, code execution, all of that.

[17:26](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1046s) And so with the multi-agent approach, you can explore this sort of domain-driven kind of design. The third is, um, does the task require consuming or processing extensive context? And so here we are constantly snapshotting the state of the app, screenshots, all of that.

[17:43](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1063s) And it's kinda useful to sort of give individual agents each of these, uh, large pieces of context to process and then return to some other final coordination agent. And then finally, adaptive solutions.

[17:56](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1076s) As you take actions in the world or in the environment that your agent exists, the environment might change, and you might constantly need to sort of react to that.

[18:04](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1084s) And so you might need, like, an autonomous multi-agent approach here.

[18:08](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1088s) The second takeaway is eval-driven design. Uh, most people wanna start out with just building their agent. Um, it's typically a mistake. Instead, you wanna define your task, um, define your evaluation metrics, build the baseline that has nothing to do with agents, improve your agents iteratively.

[18:25](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1105s) In, in the case of this app, we had like a simple tight loop, then we improved it, we added a verification agent, and then we added, um, a planning agent.

[18:34](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1114s) And based on the interactive evaluation tool I built, I could see that all of these things actually had like ROI and improvement, and that's, that's why it makes sense to explore a multi-agent approach in this space.

[18:47](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1127s) And then the final thing is that academic benchmarks are great, but they're not your task, and so, um, you really should build evals that's sort of, uh, tuned to your task.

[18:58](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1138s) And then the second slide, um, s- the, the last set of, um, key takeaways are the design principles that we'll walk through today. I think this is the money slide here today, um, four hi- high-level things.

[19:12](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1152s) First, always ensure that your users can discover the ideal tasks that your multi-agent system is designed for. Um, provide user-facing observability traces. Um, ensure that your agents are interruptible, um, you can checkpoint and restart them.

[19:28](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1168s) And then ensure that, um, your agents can quantify the risk or cost of all actions and delegate to users as needed. And then finally, don't build the whole multi-agent system from scratch just to give a talk.

[19:40](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1180s) You probably know that. It's fun, it's a lot of work, um, and if you ever want to do something like this, consider using a framework to save you a couple of keystrokes here and there.

[19:50](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1190s) So at, at the last slide, I have a bunch of further reading, a couple of papers we've written on AutoGen Studio, Magentic-One, Magentic-UI, challenges in human-AI communication.

[20:00](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1200s) Um, these are all good references I recommend you take a look. And then I'm at the end of my slides. Thank you so much for listening. Um, I have a book I'm wr- I'm writing.

[20:10](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1210s) There's a lot more about this. Chapter three is really just about like design. Um, take a look, um, if it's helpful for you. And all the code for Blender LLM is also available.

[20:21](https://www.youtube.com/watch?v=fmZWvE7yDZo&t=1221s) Thank you. [outro music]
