How We Build Effective Agents: Barry Zhang, Anthropic

AI applications can evolve from individual model calls for summarization, classification, and extraction into workflows that orchestrate multiple calls through predefined control flows. Those workflows trade additional cost and latency for better performance while preserving explicit control over execution. An agent introduces a different operating model: it determines its own trajectory and adjusts its behavior in response to feedback from its environment. Greater agency can increase capability and usefulness, but it also increases cost, latency, and the consequences of mistakes. 0:17

That distinction makes autonomy an architectural decision rather than a default upgrade. Agents are most appropriate for complex, valuable tasks where ambiguity makes the full decision tree difficult to specify in advance. When the possible branches can be mapped explicitly, a workflow allows developers to optimize individual steps, exercise more control, and generally operate more cost-effectively. 2:30

The economics matter because autonomous exploration consumes tokens. Zhang gives a high-volume customer-support example with a per-task budget around $0.10, which he describes as affording roughly 30,000 to 50,000 tokens. Under that constraint, a workflow covering common scenarios can capture most of the available value without paying for unrestricted exploration. 2:30

A promising use case still requires testing the critical capabilities on which successful execution depends. For a coding agent, those capabilities include writing useful code, debugging, and recovering from errors. A weak link does not necessarily make the entire application impossible, but it can multiply cost and latency across the agent’s trajectory. Zhang recommends reducing the scope, simplifying the task, and trying again when those bottlenecks appear. 3:29

Teams must also assess both the cost of an error and the difficulty of discovering it. High-stakes mistakes that are difficult to detect make it harder to trust an agent with independent action. Read-only access and additional human oversight can reduce exposure, but they also restrict the autonomy that makes an agent scalable in the first place. The resulting tradeoff is not simply safety versus capability; it is whether a particular environment supports enough trustworthy autonomy to justify an agent at all. 4:29

Coding illustrates how these requirements can align. Moving from a design document to a pull request is complex and ambiguous, useful code has meaningful value, and Zhang identifies Claude as already capable across multiple parts of the coding workflow. Most importantly, unit tests and CI provide concrete mechanisms for checking the output, making errors easier to discover than in domains where results are difficult to verify. 4:29

Zhang reduces the basic agent architecture to a model using tools in a loop. Three elements determine how that loop behaves: the environment in which the agent operates, the tools through which it acts and receives feedback, and the system prompt that defines its goals, constraints, and desired behavior. The model repeatedly interprets its available context, selects actions, observes results, and continues within that framework. 5:39

The practical reason for this minimal architecture is iteration speed. Upfront complexity slows experimentation before the team understands which behaviors actually matter, while improving the environment, tools, and prompt offers a stronger early return. Zhang notes that agent applications can differ substantially in their product experience, scope, and capabilities while sharing almost the same underlying backbone and even nearly identical code. 5:39

Once the environment is determined by the use case, the principal design choices become which tools to expose and how to instruct the model. Optimization follows after the basic behavior works: coding and computer-use agents may benefit from caching their trajectories to reduce cost, while search agents can parallelize multiple tool calls to reduce latency. Presenting agent progress clearly is another broadly useful improvement because it helps users develop trust in what the system is doing. 6:53

Agent behavior can appear sophisticated while still emerging from inference over a limited amount of information at each step. Zhang describes the model’s current understanding of the world as being represented within roughly 10,000 to 20,000 tokens of context. Developers should inspect that context directly and ask whether it contains enough coherent information to support the decisions they expect the model to make. 7:46

His computer-use example makes this constraint tangible. The agent receives a static screenshot, an inadequate task description, and a set of tools; only actions performed through those tools can change the environment. During inference and tool execution, it lacks continuous visibility into what is happening, and only the next screenshot reveals whether an attempted action succeeded or caused an unexpected change. This stop-and-observe cycle explains why actions that look obvious to a human observer can still be difficult for the agent. 9:01

Looking through that restricted perspective clarifies what additional context is actually useful. A computer-use agent may need the screen resolution to choose accurate click coordinates, along with recommended actions, limitations, and guardrails that reduce unnecessary exploration. Rather than adding information indiscriminately, the developer’s task is to identify the specific environmental details and behavioral boundaries that make the next decision more reliable. 10:05

Models can also assist with inspecting their own operating conditions. Zhang describes asking Claude whether a system prompt is ambiguous, whether a tool description is understandable, and whether a tool needs different parameters. His team also submits complete agent trajectories to examine why a particular decision occurred and what information might improve future decisions. This technique supplements, rather than replaces, the developer’s own understanding of the agent’s context. 10:05

One unresolved challenge is making agents more budget-aware. Unlike predefined workflows, autonomous systems do not offer the same straightforward control over cost and latency because their execution paths can expand as they explore. Zhang identifies the definition and enforcement of budgets for time, money, and tokens as an open production problem whose resolution could make additional agent applications practical. 11:10

A second direction is self-evolving tools. Teams already use models to improve tool descriptions, and Zhang suggests extending that practice into a meta-tool that allows agents to design and refine their own tool ergonomics. He presents this as a possible route toward more general-purpose agents that can adapt their interfaces to the needs of individual use cases, rather than as an established capability or demonstrated result. 12:11

Zhang also anticipates greater use of multi-agent collaboration, emphasizing parallelism, separation of concerns, and sub-agents that protect the main agent’s context window. The unresolved question is how such agents should communicate: many existing systems are organized around mostly synchronous user-assistant turns, while richer collaboration may require asynchronous communication and additional roles that let agents recognize and interact with one another. These possibilities remain forward-looking questions, reinforcing the talk’s immediate advice to use agents selectively, keep implementations simple, and iterate from the agent’s perspective. 12:11