UX Design Principles for (Semi) Autonomous Multi-Agent Systems

What should happen between dropping a data file into a web interface and receiving a useful visualization? The interface must help turn an open-ended request into concrete work. Victor Dibia approaches that problem through human–AI collaboration: he introduces himself as a principal research software engineer at Microsoft Research, with experience on GitHub Copilot, AutoGen, and AutoGen Studio. Earlier work at Cloudera and IBM Research combined machine learning with human-computer interaction.

In LIDA, a project Dibia dates to approximately August 2022, the user drops in a CSV or JSON file. The system summarizes the data, proposes questions about it, and generates visualization code for those questions. It then executes the code, performs post-processing and error recovery, and returns charts. This is already an agentic workflow: the useful result depends on a sequence of model decisions and executable operations, rather than a single text response.

The pipeline separates summarization, goal exploration, and visualization generation, with a code interpreter inside the visualization stage. Once charts exist, a diffusion model produces more varied visual representations of the data. The slide’s four modules—Summarizer, Goal Explorer, Viz Generator, and Infographer—make those responsibilities visible.

Dibia reports that LIDA’s error rate fell from about 20% with DaVinci Codex models to about 1.5% after tuning with GPT-3.5 Turbo. He describes the improvement as occurring roughly three months later, but does not specify the evaluation set or error definition for this comparison. The practical lesson is feasibility: a pipeline that initially required substantial recovery became more usable as its model component improved, and similar capabilities subsequently appeared in Microsoft and other products.

LIDA’s stages were deliberately assembled into a workflow. The next question was whether developers could instead define agents that exchange messages and organize their work around a problem. That became the premise of AutoGen, a framework for building multi-agent applications. Dibia presents AutoGen as having about 45,000 GitHub stars at the time of the talk.

AutoGen Studio exposes that composition through a web interface: create a team, drag agents into it, and configure the models and tools those agents use. The developer works with explicit building blocks instead of wiring every interaction by hand. Rather than spending the rest of the talk touring Studio, Dibia switches to a new application built from scratch, so that the implementation choices and their UX consequences can be examined together.

BlenderLM is a framework-free multi-agent prototype for translating natural-language requests into 3D artifacts. A user might ask for a scene containing a ball on a table; the application handles the operations needed to produce it in Blender.

The motivation comes from learning Blender itself. Dibia estimates that working through roughly four hours of the Donut Tutorial can take 40–50 hours. The learner must discover where controls live while also learning the underlying 3D concepts. Natural language offers a way to express the desired artifact without first mastering every operation. The polished example he displays is an aspiration, however: he explicitly says the prototype has not reached that quality.

The control architecture depends on how much of the solution is known in advance.

Fixed workflows can be reliable and valuable in production. Blender introduces a different challenge: taking an action changes the application state, and the system must react to what actually happened. In the exploratory approach, tools and observations become part of the control loop.

That choice creates three requirements. The system needs enough autonomy to address different tasks, must handle tool calls with side effects or unexpected results, and must break complex requests into steps that may run for an extended period. These are also the reasons a chat box alone is an incomplete interface: the user needs to understand and control work that continues after the initial request.

The BlenderLM web application connects to a running Blender instance over a WebSocket. It also exposes fixed tools that a person can invoke directly. Dibia begins by clearing the scene with a button; the socket connection streams the changed scene back into the web interface. This establishes a visible connection between an operation and its effect before any agent starts working.

He then selects an example requesting two balls with a shiny, glossy silver finish. Activity starts streaming immediately: the system announces task analysis, then displays a plan. The planner proposes adding a ground plane, creating spatially separated spheres, and assigning the silver material. The first visible change is the plane appearing in Blender. The plan describes intended work; the viewport shows executed work.

After tool execution, the application streams another update and runs a verification step. A verifier uses a scene snapshot, a log of the scene’s contents, and a visual representation to judge whether the system is progressing or stalled. That judgment influences the next action. The demonstration reaches a ball that Dibia can inspect and manipulate in Blender; he moves on before establishing completion of the entire requested two-ball scene.

Start with a goal and a working baseline, not an agent definition. Dibia’s construction order makes each new layer depend on something already testable:

1. Define the goal.
2. Build a baseline without AI.
3. Implement the tools a person would need to accomplish the task.
4. Define an evaluation test bed.
5. Add the agent.

For BlenderLM, the goal is to translate natural-language tasks into 3D artifacts. Before attempting that translation, the application needs to prove it can create an artifact at all.

The baseline is Blender’s equivalent of Hello World: run a script that adds a single cube. In Blender’s Python environment, the essential operation can be expressed directly:

import bpy

bpy.ops.mesh.primitive_cube_add(location=(0, 0, 0))
cube = bpy.context.object
cube.name = "HelloCube"

A Blender add-on and a client library supporting socket connections provide the surrounding integration. This separates a basic question—can the application send an operation and observe its effect?—from the later question of whether a model chooses the right operation. It also gives the developer a fast path for prototyping and testing.

Tools then determine the agent’s practical reach:

• Task-specific tools: Perform a bounded operation, such as creating a Blender object.
• General-purpose tools: Execute arbitrary code generated by the model, allowing a much broader range of Blender operations.

The broader tool expands capability, but it also expands what the system must inspect and control. Dibia recommends spending about 50% of development time on tools, because an agent’s capabilities are bounded by the tools it receives.

Evaluation infrastructure grows in stages. First, a Jupyter notebook lets the developer exercise the code and tools directly. Next, an interactive web UI exposes the evolving scene and the application’s behavior. A third proposed stage adds automated test suites, metrics, and an evaluation harness. The demonstrated interface belongs to the second stage; the automated suite is described as a next step.

Only then does the agent enter the picture. The base implementation is an LLM in a tight loop with function calls, continuing until it returns a final result. For a stateful task such as scene construction, Dibia finds that this alone is insufficient. He adds a verifier that examines scene contents and object lists after each step, asking whether progress has been made and whether the user’s task is complete. A planner then breaks the request into atomic steps for the execution loop to handle.

These roles make the application easier to reason about: execution changes the scene, verification evaluates the resulting state, and planning organizes the remaining work. They also expose the UX problem more clearly. The user is supervising an ongoing process with several responsibilities, not merely waiting for a response. Dibia offers four design principles for that situation as provisional guidance in an early field, rather than an exhaustive specification.

Capability discovery starts with the distinction between everything an agent might attempt and the smaller set of tasks it can perform reliably. Expose that dependable set through capability presets or example requests, as BlenderLM does. Suggestions can also depend on context: inspect the currently open scene and propose useful next operations. The interface helps users choose a productive task before execution begins.

Observability and provenance make the ongoing work intelligible. Stream activity logs so users can understand what the agent is doing, and provide debugging tools when behavior goes wrong. Token usage and time taken for individual activities help users explain a slow or expensive run. A changing viewport shows the artifact; the activity trace supplies the accompanying account of the work.

Interruptibility gives users a way to act on that information. An agent may take the wrong route, approach a mistake, or consume resources the user did not intend to spend. The system should support pausing, checkpointing, rollback, and resuming. Those capabilities let a person stop an unwanted trajectory and recover useful work instead of treating the whole run as an indivisible operation.

Cost-aware delegation addresses the consequences of the next action. Dibia argues that tool calls are effectively equal from the LLM’s perspective unless the system adds a distinction. Python that adds an object to a scene and code that attempts to delete the operating system cannot receive the same treatment. An inspection module should estimate an action’s cost or risk and determine when to involve the user. Here, cost includes destructive consequences, not just tokens or elapsed time.

More agents and more autonomy also create more opportunities for error. Before adopting them, inspect the problem and ask whether the benefit justifies the added complexity. Dibia illustrates this with a large circle representing engineering tasks and a much smaller circle representing tasks that truly benefit from a multi-agent approach. The drawing expresses selectivity, not a measured proportion.

Four suitability questions make that judgment more concrete:

• Planning: Can a high-level request be meaningfully decomposed into steps that move it from unsolved to solved?
• Different perspectives: Does the task benefit from distinct responsibilities or personas, such as planning and code execution?
• Extensive context: Would it help to distribute large inputs—such as application snapshots and screenshots—to separate agents that return results to a coordinator?
• Adaptation: Do actions change the environment enough that the system must repeatedly reconsider what to do next?

Blender scene construction supplies reasons for each: a desired artifact needs a plan, execution and verification have different responsibilities, scene observations accumulate, and every operation changes the state subsequent operations must use.

Task-specific evaluation should decide whether those extra agents stay. Define the task and its metrics, establish a non-agent baseline, and improve iteratively. BlenderLM followed that sequence from a simple execution loop to a verifier and then a planner. Dibia reports that his interactive evaluation tool showed improvements that justified the additions, although he supplies no quantitative comparison here. Academic benchmarks can inform a design, but they do not replace evaluations of the application’s actual task.

The closing design standard is a user who can discover suitable work, follow its progress, interrupt and restart it, and receive control when an action’s risk warrants intervention. Supporting that experience requires implementation work beyond the agent loop. Building BlenderLM from scratch made the pieces visible, but Dibia describes it as labor-intensive and recommends considering a framework to reduce that burden.

For continued study, he points to work on AutoGen Studio, Magentic-One, Magentic-UI, and Challenges in Human-Agent Communication. He closes with his then-in-progress book, Designing Multi-Agent Systems, whose third chapter focuses on design, and notes that the BlenderLM source code is available. The example leaves readers with both an implementation to examine and a concrete question to apply to their own systems: what does the user need to see or control while the agent is acting?