AI Engineer World's Fair 2026

Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

Read the talk

Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4)

Priyanka Phatak and Gabe Cemaj explain how to version agent definitions, choose workspace boundaries, evaluate real behavior and keep a coordinator responsive while custom tools wait.

From a talk by Priyanka Phatak and Gabe Cemaj

At a glance

Ideas worth remembering

  • Keep one agent definition in YAML and synchronize deployed copies through CI/CD, including copies in separate customer workspaces.

  • Meaningful eval data matters more than a perfect score on an ill-defined task; historical sessions provide a practical starting dataset.

  • Use individual workspaces for the customer data segregation discussed here. Session metadata records customer identity without providing that separation.

  • For responsive conversation during custom tool waits, put the tool on a subagent and keep the main coordinator available for messages.

Check agent definitions into the codebase

An agent configuration needs a release path just as application code does. In the closing Q&A of Anthropic’s Claude Managed Agents workshop, Priyanka Phatak and Gabe Cemaj begin with a practical answer: keep long-term resource definitions in YAML in the codebase, then use CI/CD to apply changes through the ant CLI.

The CLI wraps API calls and accepts YAML, enabling what the answer calls “Terraform-like” operations. A checked-in definition becomes the input to an automated update whenever the resource changes. Anthropic uses this approach internally for long-term resources too. The comparison establishes a configuration-as-code workflow; it does not establish Terraform’s full planning, state or rollback semantics.

0:210:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:13 · section reference included

A custom tool wait changes which messages are accepted

The next question tests the promise that users can message an agent while it works. Two execution states behave differently:

  • Other work in progress: incoming messages can queue while the model continues working.
  • Waiting for a custom tool result: execution pauses, and the expected next input is that tool’s result. Sending an ordinary message at this point returns HTTP 400.

The stated reason is ordering: an outstanding custom tool call must receive its result before other calls proceed. This matters for human-in-the-loop tools, where the wait may be long enough for another user message to arrive. The Q&A returns to that architectural problem at the end rather than changing the API behavior.

A brief repository question follows. The other workshop materials came from Code with Claude, with corresponding conference recordings available on YouTube. Those recordings provide context for the neighboring repositories; this workshop serves as the overview for Managed Agents.

1:091:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:09 · section reference included

An eval harness must change as models change

New model capabilities can make old harness assumptions unnecessary. Anthropic evaluates early model releases against internal datasets so it can revise prompts, remove obsolete scaffolding and add what the new model needs. The scope includes general agent tasks, beyond coding.

Cloud execution makes large batches practical: the answer gives the example of spinning up 10,000 sessions in parallel, then evaluating their results. Parallelism supplies attempts to compare; interpreting those attempts is what determines whether a prompt or harness assumption should survive.

For builders, the recommendation starts with data quality. Claude Code can help construct an eval, but an ill-defined task can produce a perfect score with little connection to useful performance. Anthropic encountered initial evaluations that reached 100% and still left the essential question unanswered: what did that score mean?

Historical sessions offer a concrete starting dataset. Review the results those sessions produced, assign scores and use them to assess model behavior. This grounds evaluation in work the application actually encounters, while leaving the builder responsible for defining a meaningful problem and quality standard. The answer offers that method rather than an out-of-the-box evaluation solution.

2:472:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:47 · section reference included

Workspace membership determines resource sharing

The sharing question comes from a team already building shared skills and connectors in Claude Desktop. For Managed Agents, the resources under discussion—agents, memory stores and MCP channels—belong to a workspace. API keys within that workspace can access its resources, and teammates with access can modify what others build.

The organization’s platform administrator creates workspaces and decides who can enter each one. The practical design rule is to group people who should share access to the same resources. Workspace design therefore follows access requirements, rather than simply copying the organization chart.

Partial sharing exposes a limitation. The platform does not provide the fine-grained controls requested in the example: share a memory store with one group while sharing an MCP channel with another. A builder can implement selective access in the application integration for end users, but that application behavior is distinct from the workspace-level access available to platform users and keys.

4:444:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:44 · section reference included

More generated code creates a review bottleneck

Once agents generate code quickly, reviewing and merging it becomes the bottleneck. A pull-request integration can launch one or more Managed Agents sessions, with subagents inspecting changes and finding bugs. Anthropic reports that internal review agents reduce time spent per PR and help reduce bugs, but keeping up with the expanding volume of generated changes remains an open problem.

Review comments alone cannot establish that an application works. The audience proposes a review-and-revision loop followed by a deployable environment for testing. Another option is to run the application inside the sandbox and have Claude validate its own changes as a QA agent.

At the time of the recording, the sandbox lacked tunneling for a person to open the running UI directly. The suggested alternative is for Claude to interact with the application through a browser and validate it, potentially returning a video. That gives the builder a way to inspect behavior through an agent’s run, with a different review experience from opening the application themselves.

6:296:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:29 · section reference included

Reuse the definition while separating customer workspaces

An enterprise question first clears up a deployment distinction: on-premises operation was unsupported; serving multiple enterprise customers was possible. For a company supporting 100 customers, the available modeling choices include customer-specific agents, workspaces or organizations, with the integration maintaining the customer-to-agent mapping.

The follow-up makes the tension concrete: two customers need the same agent behavior, and the builder wants to define it once. There are two different ways to reuse that work:

  • Copy a shared definition into separate workspaces. Keep the system prompt, model name and tools in a single YAML definition, then synchronize copies through CI. Updating the source definition can update every customer’s copy.
  • Share an agent within one workspace. Maintain a customer-to-session mapping, using session metadata such as a customer ID to identify whose session it is. Separate agent copies are also possible within that workspace.

The deciding requirement is data segregation. Although the initial answer broadly associates customer-specific agents with segregated data, the follow-up explicitly directs builders to individual workspaces when segregation is required. Resources are allocated per workspace. A customer ID in session metadata identifies ownership; it does not supply the workspace separation recommended here.

When the audience clarifies that customers have a legal requirement to keep their data separate, the recommendation settles on one workspace per customer. The earlier CI/CD mechanism resolves the apparent conflict: one maintained agent definition can produce multiple deployed copies, so shared behavior does not require a shared customer workspace.

8:508:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:49 · section reference included

Use another sandbox provider for GPUs or more memory

Custom compute requirements belong to the execution environment. The provided sandboxes had a uniform resource configuration at the time of the workshop. An agent needing a GPU or a different memory allocation would use a self-hosted sandbox instead; Modal is the concrete GPU example.

The described integration connects Modal instances so the agent loop runs there instead of in the provided sandbox. Environment specifications and third-party integration cookbooks are available in the documentation, with Modal, E2B and Daytona named as examples. This moves the compute choice to the sandbox provider rather than exposing a larger preset in the managed sandbox.

12:1412:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:10 · section reference included

Map chat threads to sessions and make posting explicit

A multiplayer chat agent can map each conversation thread to a Managed Agents session. When someone posts, the integration injects a user.message and identifies the sender—for example, Gabe or Priyanka. Several people can then address the same agent while it retains the context of their shared thread.

Incoming messages and outgoing posts need different treatment. Give the agent a Slack posting tool so it explicitly chooses when to publish. Automatically mirroring every agent response into the channel would remove that decision and turn internal responses into public conversation by default.

Memory can be shared across sessions to carry learning between conversations. For channel-scoped behavior, the suggested design uses one agent per Slack channel, with separate memory and access limited to that channel’s messages. Thread-to-session mapping organizes conversations; channel-level agent and memory choices determine what context they share.

13:2213:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:22 · section reference included

Define completion separately from progress reporting

For tasks lasting more than six hours, the recommended mechanism is the outcome system demonstrated earlier in the workshop. Define an exit criterion and a rubric, and Claude continues iterating toward it—“mercilessly,” as the answer puts it. The presenters say no special module is needed merely to keep a session running beyond six hours; this is a description of supported operation, rather than a measured completion guarantee for a particular task.

Knowing whether the task is finished differs from knowing what it is doing now. The session event stream, available through an events API under the session, provides the observation path. A custom status-update tool can also let Claude send a human-readable update when it judges that it has made progress. That suggestion supplies agent-chosen progress reports, without specifying a fixed reporting schedule.

15:0615:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:34 · section reference included

Move the custom tool wait off the coordinator

The final question returns to the HTTP 400 problem: how can a conversational agent remain available while a human-in-the-loop tool waits? The proposed coordinator pattern puts custom tools on a subagent. The main agent remains free to receive messages while the subagent handles the blocking work.

In the concrete example, Agent A is the coordinator and can talk to Agents B and C. Agent B’s definition contains the custom tool. When B emits a tool-use request, the integration supplies its result through the main thread; the platform uses IDs to route it to the right subagent. B has a technically separate thread within the overall session, so the application does not have to treat it as an unrelated top-level conversation.

Which path stays available when the custom tool waits? The diagram separates delegation from tool-result delivery. The wait belongs to B’s work, while A remains the conversational entry point. This preserves the tool’s ordered request-and-result interaction while moving it away from the agent responsible for receiving new messages.

The final demonstration points to a gap where the main thread is idle while subagents continue working; they could also be waiting for tool results. That distinction matters: an idle coordinator does not mean the whole session is idle. The coordinator pattern keeps the agent “chatty” by separating availability for conversation from the state of delegated work.

A conversational coordinator with a waiting tool subagent

Application integrationSends user messages and supplies custom tool results through the main thread.
Agent A: coordinatorsession — Remains available for messages while delegated work proceeds.
Agent B: custom toolssession — Defines the custom tool; can wait for its result on a separate subagent thread.
Agent Csession — Another subagent the coordinator can address; no custom tool role is specified.
Delegation and tool-result routing share the main entry point, while the custom tool belongs to Agent B.
Read the diagram as text
  • Application integration. Sends user messages and supplies custom tool results through the main thread.
  • Agent A: coordinator. session — Remains available for messages while delegated work proceeds.
  • Agent B: custom tools. session — Defines the custom tool; can wait for its result on a separate subagent thread.
  • Agent C. session — Another subagent the coordinator can address; no custom tool role is specified.
  • Application integration → Agent A: coordinator: User messages.
  • Agent A: coordinator → Agent B: custom tools: Delegate custom tool work.
  • Agent A: coordinator → Agent C: Can communicate.
  • Agent B: custom tools → Application integration: Emit custom tool use through session.
  • Application integration → Agent B: custom tools: Tool result via main thread, routed by IDs.
16:3316:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:30 · section reference included

Resources

Transcript download
Read the transcript and explore slides

Loading transcript and slides…