Claude Managed Agents Q&A: Evals, CI/CD and Multi-Tenancy (Part 4 of 4) — Anthropic

All right. We have a question here.

Hey. Uh, how do you CI/CD the agent configs? Um...

Ah, that's a great question. Uh, so we have something called ant CLI, which is a CLI that basically wraps our API to make calls, um, and allows you to do sort of Terraform-like things where you pass in YAML. So you can put your, uh, long-term, uh, resources in YAML in your code base, and then build, um, CI/CD to basically update the ant CLI- with ant CLI using the, those YAMLs, uh, every time you make a change to your, to your resource.

Does that make sense?

Got it. Uh, so I assume you guys use the same internally too?

Yeah. That's how... If, if we need to build durable long-term resources, we just, we can check them in.

All right. Thanks.

Thanks.

Um, so I think you mentioned you can send messages whenever you want.

Mm-hmm.

Um, can you still send messages if the model's waiting for a, a custom tool result?

Uh, not when it's waiting for a custom tool result. We basically pause execution, uh, at that time. Um, but if the model is doing other work, uh, then we do allow you to queue up messages. Um, but when it's doing a custom tool result, we're expecting you to send that, um, so because of ordering purposes, we, we block the calls.

Got it. But what happens if you do send the message?

You'll get a 400.

Okay. That makes sense.

Uh, there was an ask to see the QR code slide one more time.

The, uh-

The, uh, the credits one? Yeah.

Any, any other questions? If you're, if you're shy, we-

Yeah

... we're also hanging out here

Yeah. We'll hang out after for any, any questions you don't feel.

Yeah.

Awesome.

Yeah. I'm looking through the repo. There's a lot of interesting workshop talks. Is there anything that's missing from that, like slides that would help us go through other-

Yeah, yeah

... sessions, like, that you can post, or is it pretty much self-explanatory through the, the README files? You just like have, you know... 'Cause you had like a skill.workshop, which-

Yeah, yeah. Uh, the other workshops came from our, uh, Code with Claude conference, um, most of which you can find online, uh, for the version we did in London about a month ago. So you can definitely-

Yeah. All, all, all those talks are on YouTube if you're interested in the other repos that are in there, but this one should give you like a pretty good overview and it's pretty similar to what we've done at other conferences.

Yeah.

Yeah. So just going back to evals, can you talk a little bit about, you know, how you guys are maintaining quality, best practices around the, the eval harness?

Uh, yeah. We run a lot of evals, especially as new models come up, as new capabilities come up. We have access to the models before you all do, obviously. Um, so one of the things that we as a managed offering wanna make sure we do is that we, uh, update our assumptions about the harness as early as possible so that we can remove things that are no longer needed. Uh, and we do all of that in the eval base. So we have internal tooling that basically looks like different datasets that we can then run our different sessions.

It's... Because it's all in the cloud, it's easy to say like, "Spin up 10,000 sessions to do this, uh, all in parallel." Um, and then we evaluate those results and tune prompts and tune assumptions, remove things, add things, uh, to make sure that for every model release, we're doing the best we can. Uh, and we're evaluating all sorts of things for generic purposes, not just coding.

So, so how does that extend into the agents that we would build? Is there a recommended path for, um, for evals, or is it kinda just bring your own solution?

Uh, I think the, the best advice we can give you is get high quality data uh, to evaluate over. I think Cl- Claude, Claude Code is pretty good at building an eval for you, but garbage in, garbage out. We've seen that some of our initial ideas for how to evaluate things were sort of ill-defined, and you get 100%, but what does that mean in performance? So iterating on your data and making sure that you're setting up the problem correctly. Um, I know it's like not a out of the box solution, but it... generally

the advice that we've seen works.

And you can probably look at like your historical sessions and like what results you got and then sort of try to score it and see how the model's doing. Um, that would be a good way of like building your dataset, looking at your historical sessions. Um-

I had a question o- oh, over here.

Hey, thanks for sharing. Uh, I have a question about how to share those agent and all the resources across all the o- the entire organization. The reason I'm asking is because we use Claude Desktop quite a lot. We're building shared skills. We're building shared connectors.

Mm.

But for the, uh, everything you shared today, for example, agent, memory store, or MCP channel, can we actually share them or can we partially share them across the organization to different departments? So what is boundary? What is the privilege about those resources? Are those resources bounded to my account or how can I control this? So what's the, what's the way to do that?

They're bound to your workspace.

Oh.

So any API key within your workspace can have access to the resources that-

So can my, can my teammates add it or modify what I'm building, or it's-

Yes

... completely controlled, right?

So I, I think your, um, Claude platform admin... So you'll have an org for your organization.

Yeah.

And then your admin will be the one setting up the workspaces and deciding who gets access to which workspace.

Yeah.

So I think you would basically design your workspaces depending on your access control.

Correct.

You'd probably wanna do like- W- whatever group of people you want access to resources X, put them in one workspace

So you can share whatever you want, even partially of the... You know, if you're the agent that can do something, but I only want to share part of that. For example, I share the memory to someone, I share the MCP channel to the others. Is that possible? Or I need to share the entire tech stack of what I'm building?

We don't have fine-grain controls of like-

Yeah

... this is shared, this is not. But you as a, a, as a builder and integration would be able to control that, uh, pretty granularly so that your end users can have access to-

Yeah

... partial sharing.

Right. Cool. Sweet. Thanks very much.

Thank you.

Yep.

Yeah. So basically, we can generate a lot of code with a lot of agents, and then we're getting to the bottleneck of, uh, reviewing that code, making sure you can actually merge it. Um, how do you guys think about that in terms of are you deploying? Like, I see you deploy every 30 days, which, uh, might be a different definition of deployment, but is it deploying something you can actually test to make sure it works well? Or how are you reviewing, uh, this many lines of code from all the agents?

So the, the original workshop that we wanted to do was actually answering that question, how do you, how do you build agents to review code? Um, what we found is you can build an integration with CMA, so that on PR it kicks off one, two, three, many sessions with many sub-agents to evaluate what has changed and find bugs. Internally, we, we have, uh, these sorts of agents do a lot of the uplift for us, and we've seen that that enables us to, uh, reduce the amount of

time we need to spend on-

Mm-hmm

... a per PR basis and reduce the number of bugs and keep the quality, uh, high. Um, but this is partially an open question of now we've removed the boundary of, uh, typing and exploded the number of PRs we can make per day, hour, second. Um, how do we keep up? And we're still exploring that, but-

Yeah

... CMA is part of the answer, I think.

Yeah. And you can iterate in a loop, you know, to like review, add comments, review until it's done, but then at some point you still need to run the thing to make sure it works well locally. So I think that's where the deployments can come in, where you can, um, you know, auto deploy something that you can test with a custom deploy agent, like you said. Um, it just depends, like if you have different environments for each branch, but...

You, you can run your application in the sandbox as well and have Claude self-validate the changes that it made-

Mm-hmm

... um, and make sure that like it acts as a QA agent as well.

Are these like long-running sessions that you can go in and actually play around on the UI?

We don't have, uh, uh, kind of tunneling today, but-

So you would have Claude go in there, play around-

Yeah

... with the browser.

That's right.

And then-- And validate. Yeah. And then maybe even give you back a video maybe.

Yep, exactly.

Cool. Thanks.

Any other-

Hello.

Hi.

Uh, for an enterprise company, if at all we want to maintain true tenant isolation that we want to support.

Mm-hmm.

I heard that you are not supporting yet the enterprise, uh, company, uh, use cases probably.

We're not supporting on-prem.

On-prem?

Yeah. But-

But what if I want to have, uh, let's say I have hundred customers that we support as an-

Mm-hmm

... enterprise company.

Yeah.

Can I have separate, uh, Claude managed agent instances-

Yeah.

Yeah

... per, uh, company of ours-

You could, yeah

... that we want to support?

Yeah.

Is that a viable option that you support? Is it only one instance that we get, or can we get one instance per company that we support as an enterprise?

It, it's however you choose to define it. You can define an agent per customer, and then that way the data will be segregated. You can use a workspace per customer if you want, like true isolation. You could do an org per customer. Uh, so it depends on how you want to set up, uh, your agents. And you would probably want to maintain that mapping of like which customer maps to which agent.

Okay.

Yeah.

Thank you.

I th-

Yeah

... I think I have a follow-up question. So, uh, just given the, uh, scenario that, uh, uh, we have two customers and we want to build, uh, the agent once, but the agent going to be shared by the two customers.

Mm-hmm.

Uh, how to achieve that?

There, there, there's many ways to do it. You could do it through automation. So you could have your agent definition live in YAML in your code, and then build tooling to update it to both of your workspaces and basically synchronize them. Or you can model everything in a single organization and then do the, the modeling that way. Um-

So you, you, you could create like a single agent across all your customers, and then you manage the mapping of like which customer is using which session.

Which session. Yeah.

Yeah. Then in, uh, in which workspace I ca- I create this agent?

Yeah, if, if you do it per workspace, then you need-- We give you tooling to synchronize-

Yeah

... a single definition in your code to multiple places, so that whenever you update it in your code through CI, you can update all of the definitions at once.

So you can either make a copy of your agent, which is essentially a template of system prompt, model name, tools, and you can like make a copy across every workspace. Let's say you go down like the one workspace per customer. Or you can just put all your customers in a single workspace, and then you maintain the mapping of which... E- either you do, again, have like copies of your agent or you use a single agent and then have the mapping of like which session maps to which customer. We have like a metadata field on the session where you can put in like, let's say,

customer ID, so you know which session w- mapped to which customer.

Okay. That, you know, like putting all the customer data in one workspace, how do we, uh, satisfy the data segre- segregation, uh-

Yeah

... requirement, you know?

Yeah. If you, if you care about data segregation, then you need to put them in individual workspaces.

Yeah.

And the resources are allocated per workspace. But there's basically tooling to enable-

Mm-hmm

... you to do, uh, each of, each of those things.

And when you say data segregation, like, is, is that like a legal requirement or is that something just you want to maintain?

Yes. Something legal, like customer doesn't want to share data-

Got it

... with other customers, you know. They want to dedicate.

Yeah. You would do it per workspace.

Okay. Okay. Thank you.

I saw a question here. You're gonna have to speak up or...

Um, is there a way to modify the compute environment? Meaning, um, in some cases, I may need to have higher memory for my agent to do the work, or I need a GPU for my agent-

Mm-hmm

... to actually do the work. Is it supported in this or...?

To, to the-- today, if you wanna do custom resource allocations for your sandbox, you would use self-hosted sandboxes. You would use something like Modal that has GPU offerings.

Yeah.

And we have cookbooks and kind of first party integrations so that you can hook up your Modal instances to run your agent loop, uh, inside of Modal instead of our provided sandboxes, which today are all the same. Uh, so if you care about different scale, you would use a different provider for that.

And do you have, like, a list of, uh, specifications-

Yeah

... of what the environment these agents have access to?

Yeah. You, you can find that in our docs, and you can find cookbooks to integrate with third-party providers as well in our docs.

Yeah, we have a bunch of cookbooks on, like, how to work with Modal or E2B or Daytona, wh- whatever provider that you wanna use for sandboxes.

Thank you.

Yeah.

Yeah.

Hi.

Hi. Thanks again. So out of curiosity, how to build a sim- similar version of CloudTag using CMA in a different chat channel, especially for the multiplayer mode?

Yeah. Uh, this definitely is something you could do-

Yeah

... uh, today, where you would on a per thread level map it to a session, and then any time a user messages that session, you inject that as a user.message, letting the agent know, "Hey, this is coming from Gabe," or, "This is coming from Priyanka." And so everybody can message, uh, that agent independently. You can have it share memory across sessions so that it learns, uh, from each other.

Mm-hmm.

You can give it tools to reply back, uh, using, like, the Slack post tool or something like that, so that the agent can decide when it wants to post. You don't necessarily wanna just mirror whatever the agent is responding. You want it to be explicit. Uh, but you could definitely model this, uh, using CMA and...

And if you want it to be like CloudTag, you'd probably do, like, one agent per Slack channel, and then every agent has its separate memory and only has access to messages within that Slack channel.

Yes.

Hey. Thank you for the presentation. Very useful. Quick question. In terms of, um, looping and doing long runs, um, in this cloud console, how do you see that? Um, is this something you have, like, some specific module that I, I'm not finding, or, um, are you, you planning to develop that, um, more in the future? Um, thank you.

D- do you mean how do we provide the observability for really long sessions or?

No, no, no. Um, so more focused on, um, using this platform to build those, uh, long-running agents, like hours and hours, like plus six hours, for example. Um, is this something that is already embedded in what you have built in the agents, or I'm c- more curious to know i- if you are developing something specifically for that, or how you see that trend.

The, the outcome system that I demoed is the best way to do really long sessions because you define an exit criteria, and Claude will just keep going until it meets that criteria, and it could be a really long task like, you know, rebuild blank. Uh, and it will just mercilessly iterate until it matches the, the rubric.

There, there, there's nothing special you would need to do to get the session running longer than-

Gotcha

... six hours. It should, it should just keep, keep going.

One quick follow-up. So if I run, uh, the agent for hours and hours-

Mm-hmm

... and I want to understand, like, checkpoints, status, the best way to do that would be through the session tab?

Uh, yeah.

It would be the session event stream. Yeah.

Okay.

There's a events API on-- under the session.

Okay.

You could have a custom tool called, like, update status for human-

Mm-hmm

... or something like that, and Claude would on intervals, whenever it feels like it's made progress, call that tool and give you a status update.

Thank you. Thank you very much.

All right. Any other final questions?

Yeah.

Yes.

Um, following up on that previous question about using this for CloudTag or just similar functionality, I've been, uh, analyzing a whole bunch of, uh, agent frameworks. So if you get a message while the, um, agent is waiting for a custom tool result, for example, human in the loop, um, you said it would four hundred.

Mm.

Do you have, uh, any suggestions on how you would deal with that?

Um...

Can you do custom tools on a sub-agent? I forget.

You, you could do custom tools-

Yeah

... on a sub-agent.

'Cause then-- 'cause you could basically have, like, a sub-agent that's handling your custom tool, and then that way your main kinda coordinator agent's always free for receiving messages.

In the API, can you directly return a custom tool result for the sub-agent's tool?

Mm.

Well, the sub-agent would emit-

Yeah

... the custom tool use, and then you would po-populate it back to that-

So essentially you-

You know, with-

... you define, like, your main coordinator agent, and then you can say-

Uh-huh

... Agent A can talk to Agent B and C, and then Agent B is the one, let's say, processing your custom tools, and your Agent B definition is where you'll define your custom tool.

Right.

And then that's the session thread that you would listen to to get the custom-

Is that a different session thread or the same session?

No, it's, it's part of, like, the same core session thread.

It's, it's a different thread technically, but you only interact with the main thread, and we-

Yeah

... correctly delegate it 'cause we know where-

Yeah

... it needs to go based on IDs. Um-

Okay

... but you-

And then the main agent will pass it-

Will pass it along

... onto-

Exactly

... the sub-agent.

Yeah.

So the session is still idle even though the other agent-

If, if, if a sub-agent is doing work, it's not idle. It's, it's-

Uh-huh

... still running.

But, like, if the main coordinator agent is idle, the sub-agents could still be waiting-

It's still doing work.

It's still doing work.

Yeah. You, you can see that here.

Okay.

Yeah.

That's really impressive.

This, uh, gap here, the main session is, is idle.

Oh.

The main, the main thread is idle while the sub-agents were doing work.

And they could be waiting for tool results.

Exactly.

Yeah, yeah. So if you're trying to do something where you want, like, the main agent to be always chatty, then I think the coordinator pattern works pretty well for that.

That's awesome.

Yeah.

I think we're, we're at time, but we'll, we'll be-

Yeah. We're-

... just outside if you wanna come talk to us. Uh, thank you for having us.

All right. Tha- thank you so much.