Scheduled Agents, Dreaming and Q&A: Claude Managed Agents (Part 3 of 4) — Anthropic

Okay, so yeah, let's, let's see what this looks like. Uh, analyze, uh, Google and Microsoft. Um, so ideally what would happen, and I hope this works live for y'all, um, is that, uh, this main agent is going to, um, evaluate if it has the information, and then it's gonna give us a custom tool where we're going to then spin up individual sessions for each of those financial artifacts. So you'll see this, this batch analysis thing. Um, it's gonna

call it for two tickers, and then it's gonna kick runs for each of those. Um, so this is how we can orchestrate our multi-session, um, harness. And each of those sessions has multi-agent, uh, that are doing collaborative work. And each of those, uh, also have outcome definitions so that there's self-supervision happening, using memory stores to learn from one another, um, et cetera, et cetera. Um, so let's move on to the finale, which is, um, what if you wanna do recurring work? What if you wanna do,

uh, you wanna have a list of companies you care about, and you want every day at nine AM or first of the month to do the analysis over all of these companies, uh, on a schedule? So that's what deployments are. Deployments is-- Everything is API-based, so we-- You can obviously interact with our platform via the console. Um, but the main way that we intend to you-- for you to interact with it is via the API and the SDK. So you can build, uh, custom UIs and custom workflows. Um, all of these are just building blocks, uh, that you can compose together

to build your application into something bigger. Um, so you can see here what Claude did, is it created a, uh, deployment, um, and that is right here. A deployment is-- takes in a agent definition. Uh, in this case, it's this, uh, research, uh, desk weekly memo using this specific agent. Um, it takes in a schedule, uh, of when it's gonna run, and it gives it an initial set of messages that it's gonna run with. You can also pass in, uh,

resources like, uh, uh, memory stores or GitHub repos or files that it's gonna use on each of those. Uh, and so what this means is that every, I think this is, uh, first of the month at one PM or something like that, it's gonna run, um, this, uh, specific session. I could run one now. So I could trigger it via the console. It's gonna do that. Um, but I could also build a UI for it, and that's what we've done here. Um, so I can... You know, I have all the CRUD. I

can list and, and search, uh, for my deployments, and then I can also kick one off, um, at any given point. And I don't know why they're failing, but I'll ask Claude about it later. Um, and it-- this has come up a few times, which is what about dreaming? What about self-improvement? Um, so before I get to your questions, which I'm sure there are many, uh, I'm gonna show you a preview that we've, uh, have out for, for dreaming. So what dreaming is, it lets you select, um... So, okay, before I show

you a previous one, it lets you select a memory store. In this case, it will, will use the memory store that we've been using. You can select a model. Um, you can give it custom instructions if you want to, but we'll use the default prompt. Um, and then we're gonna let it, uh, run, uh, the, the last thirty days of sessions through this dreaming analysis. So when I click Start Dreaming, what it's gonna do is it's gonna spin up a session using Managed Agents, actually. Um, spin up many sub-agents to analyze the

transcripts of each of the other sessions that we told it to evaluate at, all of them which use the memory store. Um, and it's gonna analyze, hey, did the memory store encode all the information that was required? Did a session, uh, run into any issue that, uh, that caused problems? Can we learn anything from it? Did this happen multiple times? Can we update the instructions in any way? And you can see here what it actually did is that it found that the skill that we created actually had a few gaps, um, on the way that it was written, and it updated the

memory store to tell it, even though there is this skill, you should extend this knowledge with the fact that we've learned from it. Uh, and you can imagine that we will continue to extend this to not only improve memory stores, but improve skills directly, uh, agent prompts, sub-agent prompts, et cetera, et cetera. Um, that's, that's it. That's the demo. I-- Sorry about the Wi-Fi. I hope you follow, uh, later on when, when you get home. Um, and let's, uh, keep going with some questions. I'll-

Yeah, we, we can all take questions.

Yeah. Frank, come on up.

Uh, Dave, we have a question over here.

Yep. You can go ahead. You can go.

Uh, thank you for the demo. Uh, we didn't see any, uh, telemetry, uh, instrumentation in the agent. Is that a default native feature?

What, what do you mean by telemetry? Because you have the full observability of all of the events, uh, that happened, right? If you go into the console, you can see it live, but you can also obviously get it from the API where-- This transcript view is like a nice view, but if you go to debug, it's every event that occurred in that session, so you can see exactly, for example, when did we start calling the model? Uh, when did we finish calling the model? So you can get latency metrics in that. Is there something else that you mean by telemetry?

Yeah. Uh, the thing is that, uh, usually in software development cycle, you kind of define what you want to observe. Uh, how did we kind of pick what did we want to observe?

Uh, right now you get all the events, and you just filter them on your end. If there are things that you think are missing, uh, hit us up, and we'll talk about adding them.

Okay. And can I bring my own observability stack in here? Um-

Uh, you mean like export to OTel?

Yeah.

Uh, you c-- you could-- Because it's an API, you could build that connector fairly easily. Uh, we just-- Everybody has different requirements, and we, we don't wanna be opinionated, but everything is just API calls.

Mm-hmm. Thank you.

We have a question.

Oh, I, I had a question. So how long can these agents run? I, I, I know you mentioned this dreaming thing, but h- can you run them indefinitely or do you see that persistent store keep increasing in size and at some point you need to re-compact it offline or something? That was-

We, we are running compaction for you-

Yeah

... so that you can run these for days. We've seen 10-hour sessions in production, uh, from some of our users.

So, so you could run these agents, like, for a year or indefinitely in theory?

Yeah, yeah. A- are you talking about running the agent for a year or are you talking about, like, one specific session that keeps going for a year?

No, basically the agent, if I had to-- If I had a task that I need to do every day, can I just keep it running for six months?

Yeah.

You could, yeah.

Will it, will it not blow its, uh, persistent store?

That's right.

Yeah.

A- and is there any orchestration piece, like, where I could have three agents, Agent A does something and delegates to B and C?

Uh-

That's some capability that you have or if you have something-

For, for now you would spend... you would do sub-agents and you would tell Claude what you want. We find that, uh, there's a desire as humans to constrain what Claude does and the way it does it. As Claude gets smarter, we have continuously removed those guardrails and let Claude-- just let it cook, uh, let it do what it thinks is best. So, uh, right now you think maybe three sub-agents is the best topology for it, but i- if we just let Claude figure that out, it actually ends up outperforming, uh-

But, but any topological or, like, graph-based workflow thing in the UI here, like, where I could have...

Not yet, not yet.

Not yet.

No, not yet.

Not yet, but, but maybe soon. Yeah. Uh, you, you can probably just, like-

Oh, yeah

... speak up.

So-

Yeah, yeah. Sor- sorry. We'll come back to you.

Sorry about that. Hey, uh, over here on d- on your right.

On my right.

On my right.

Yes.

Okay, yeah. Hi.

So question. Um, looks great. A lot of integration points. What I'm curious about, uh, for, you know, an organ- a complex organization that potentially manages multiple agents, um, you know, I, I, I think they want some sort of separation, RBAC, whatever else. Um, how is that-- What is, what is, I guess, the recommended approach for, uh, a complex organization that has those different agents, uh, in here to effectively manage?

We, we have workspaces. That's our primitive today, where you can put different customers in different workspaces and then, uh, segment them that way and give access to specific users into those workspaces. Uh, but we know that there's still a gap in what we offer and, um, that's, that's something, you know, on the roadmap as well.

Yeah. Uh, RBAC is on our roadmap, so we, we do wanna provide more, like, granular controls, where right now workspace is, like, a clean way of, like, separating your agents where you have, like, a group of people access a workspace A and they can access all agents within that. You can have workspace B with, like, a s- different set of users that can have different levels of access.

So hi.

Hi.

Excellent presentation.

Thank you.

Really excited. Uh, two questions. One is deployment. My understanding was that, uh, deployment is like at a snapshot, a moment in time of all the agents.

Mm-mm.

No?

Deployment is essentially a way for you to run sessions for a specific agent on a schedule. So you can say, like, "Oh, research desk agent, go research Nvidia once every quarter," and it'll just create a session every quarter to go research and produce you-

Okay

... that report.

And then memory, which is a state, is persisted across all sessions.

That's right.

You attach your memory store to the session, so you decide which, which session you wanna attach which memory store to. So the memory store is like a independent entity that you just attach whenever you need to.

Right. And the final question is harnesses.

Mm-hmm.

Right. One, what is harnesses for you? Because everyone have their own definition.

Yeah.

And second is the confident, right? So when I change something here, how do I know that it's not-

Mm. Like how you're-

... it's working as expected?

... confident.

Right, my own confident as the-

Yeah

... developer.

Um, for, uh, yeah, you're right. Everybody has a different definition of a harness. Uh, we-- When, when we talk about a harness in CMA, we mainly mean the tool loop execution, so managing the context and managing the session state and making sure that the agent can do the work that it wants to do. When most builders talk about harnesses, they, they really mean... Oh. They really mean, um, the ability to coordinate multiple agents and sessions together as a unit of work to achieve some sort of bigger

task in a topology of sorts. Um, we're giving you the abstraction of a single session as a unit of work for you to go out and build your harness, uh, with it as the session being one of the primitives. Um-

And then your second question about how do, how do I know how my agent is performing, uh, our best recommendation would be to run evals on your agent with, like, different types of tasks that you want the agent to perform and see how the agent does on those tasks. Um, Outcomes is a really good way where we steer the agent for you to get you the specific outcome where you can define an exact rubric for, like, "Hey, th- this is the exact outcome I want," and you can define max iterations, and the agent will keep going into it until it gets you that outcome.

Verifier. That last-

Yeah, the, the, the Verifier agent that, that Gabe talked about.

Hi. So-

Hi.

Oh, sorry. After you.

Cool. Hey, thanks for the demo. I enjoyed it. Um, my, my question at the core is a bit about how to choose between skills and agents, and I wanted to ask: How would you all say what's, like, the fundamental difference? Is there an advantage to the harnessing that surrounds an agent that makes it produce better results than a skill? Um, and one of the reasons I'm asking is 'cause, uh, I'm afraid of always investing in new kinda, like, I guess harnessing and tooling around models, and it's, uh, it can be kinda tiring always

trying to do a new thing. Um, so yeah, I just wanted to hear what you think is, like, is there a big difference and is it gonna change any time?

I, I, yes, no. Um, the que- uh, uh, thick harness, thin harness, thick skills, um, is, is a question that's constantly evolving. I think what we find is that as Claude gets smarter, the- ... boundaries and, and, and the help that we wanna give Claude start to melt away and disappear. And so what you need today might not be what you need tomorrow. The boring answer is just run evals and, and, uh, evaluate these things.

Mm-hmm.

That sounds exhausting, obviously. Like-

Yeah

... every time a new model comes out, I need to evaluate all these things. Um, what, what we find is Skills is a good abstraction today. Uh, it gives the model enough-

Mm

... information to do what it needs to do, but I presume that it will start to shift and melt and become thinner and thinner, where the model will just be able to do things on its own without any special requests and instructions, uh, down the line. So, uh, m- multi-agent seems to be more useful-

Mm-hmm

... uh, in my opinion, but your, your mileage may vary on your task.

Hi.

Hey. Hi. So my question ties up to last two questions, and especially on top of evals. One thing I'm excited about is dreaming, but how will dreaming actually affect the final outcome?

Mm-hmm.

Something that's where evals come into the picture. But you can have multiple iterations of dreaming, so that one, that can go into different directions. So is there a plan or do you already support some automated kind of dreaming with run with evals and see that, hey, how did the dirf- different direction go into? Which one should we pick based-

Mm

... on what's going on?

Yeah. So it, so it... Dreaming, what it does is it looks... Oh, you, you can choose the duration for which you want the agent to dream over. So you can say, "I wanna dream over this agent session over the last thirty days," and then look at all the transcripts and analyze what task you're trying to achieve and then come up with, like, a new memory store that's more geared towards, like, what you were trying to do. And we typically will run evals on our end to make sure that dreaming is providing you with better outcomes, and we've seen customers are able to get back-- get a

significant improvement on their evals by running dreaming and then using the new memory store. But agai- again, everything kind of ties back into evals, so, like, that, that's how you would know whether something is performing-

Where, where-

... as you expect it.

Mm. Yeah, I think it'd be good to have your own set of evals, um.

The, the other thing is-

Yeah

... dreaming is in research preview.

Yeah.

So these are early, early.

Mm.

It's like a big experimental thing.

Mm-hmm.

We've seen uplift, uh, on it generally, but this is... We, we hear you, right? Like, uh, how do I gain the confidence that this is actually-

Yeah, it's like eval's connected to dreaming, and then we can find out multiple pathways that ta- that has been taken and de- decide, "Oh, this makes more sense," because even though n- nine sessions may have been s-saying that, "Hey, this is wrong"-

Mm

... one says that, "Hey, this is good." So if you just rely on that session dreaming thing, I feel like we may not be grabbing all the insights without evals.

The... If I tell you just to trust me, is that...

Thank you.

Yeah.

Hi. Uh, thank you very much. Um, this is very impressive. Uh, those of us who have been working with Claude Code for a year or more, um, have been doing things that are similar to this, but this is nice to see this in a really mature form. Um, I, one question I have is, how much of this was just built on top of Claude Code? That is, if somebody was, you know, ambitious, could they, could they build something similar to this? And, um, what I wanted to ask you was, uh, encourage you to keep

things open. Uh, some of us are already harvesting what's in the, uh, JSON session files, uh, and starting to experiment with things like, uh, Heparag and that kind of thing for, for managing the, um, the memories and, um, and creating a, a memory pool. So, uh, congratulations. Thank you. Please keep it open. And, um, can we... Is there... Are you doing anything here that's not just built on site on top of Claude Code? Thank you.

We, we took the Claude Code harness as an inspiration. We found that it was very well attuned for coding tasks, um, and different... it's different fle- levels of flexibility for other tasks. We're building an API where we want you to bring your context, your information, your capabilities, and build on top of it, where Claude Code is trying to give you a, a product out of the box. So we have slightly different, uh, you know, value, uh, props. Um, and so it's not just, uh, Claude Code in a

box. Uh, though, though, you know, i-ideally, it performs as well, if not better, in all sorts of tasks. Um, so not, not exactly, but it's definitely inspired and motivated by, by some of the uplifts that we have seen from Claude Code.

And, and Claude Code is a terminal tool, and we obviously wanna be an API product, uh, and, and infrastructure, uh, that is gonna scale, that's gonna be reliable. So there, there, there were, like, challenges that I talked to you about, um, in the beginning of the presentation that we really worked at im- to make sure we're providing a really solid, um, product for you all to use.

Yeah.

Um, so for AWS Bedrock, is there a console, like Claude console, like you have here? Uh, that's just quite straight great for question was, yeah, I just wanna manage and see everything.

This is not available on Bedrock, but it is available via Claude Platform for AWS.

Right.

So there is a AWS version of console that you could go use this with.

Yep.

Yeah. Uh, first, uh, uh, is, is there any, uh, how to say, is there any restriction on the model to be used, like, uh, non-anthropic models?

Uh-

Just like, uh, Claude Code, I can run on non-anthropic models.

For, for now we're, we... You can-

Yeah, it's, it's Claude models only for now.

Oh, okay.

Yeah.

So, so the second question is, what about end user use? Uh, do they have to have a console account in order to use it or just a free, uh, u... Well, I can create a customized UI for this, and they can use it.

So Cl-Claude Console helps you with creating your API key- So that's how you get access to our APIs. So you-- Claude-- You're not paying separately for Claude Console. It's free to sign up. You create an API key, and then you're paying based on, like, token usage.

So for, for, for the case like, uh, SaaS, like, uh, maybe we have multiple tenant, right?

Mm-hmm.

So in order to provide this, uh, agent-

Yeah

... what kind of, uh, uh, how you say, model you have already, uh, depict on so that we can follow? For example, what, what kind of, uh, each company are supposed-- tenant are supposed to have a, uh, a different space or, or even there is a, a sub module for this, uh, each tenant. So are, are we going-- how do we-- are we actually going to organize that in term of this?

We, we, we have seen that the best model for multi-tenancy is not bring your own key because you lose observability 'cause it runs on your end user's API key rather than, than yours. So we encourage you to model things with workspaces as a tenancy primitive. Um, and we're improving that so that it's easier and easier to build, uh, on top of this for multi-tenant applications like-

Yeah. So I, I think currently you could either have-- put all your customers' data into like a single workspace, and then you maintain a mapping on your end on, like, which agent maps to which of your customers. Alternatively, like Gabe said, you could create a workspace per customer of yours, but that can get more complicated 'cause then you have to, like, recreate your agent-

So, so is any-

... across various workspaces

... you see the examples that, uh, people did the additional security layer or RBAC layer over the-- this workspace because to me, uh, it may be a little bit, uh, you know-

Yeah. Uh, that, that would probably be, like, a good, good way to do it right now 'cause we don't have any RBAC-

Mm

... uh, controls, uh, on our level other than, like, workspace. So everything within your workspace is accessible to everyone in the workspace. But we are working on making RBAC better and then making the experience better if you're working with Claude Managed Agents, but building on behalf of your customers and how do we make that easy something, uh, top of mind.

So, so is there any way you can show how to scale this, uh, i-if there's a multiple, uh, how to say, tenants working on that?

Yes.

Is there any scaling parameters like, uh, what is the, how to say, the, the minimal, uh, pod or whatever, these parameters that was... Yeah.

Uh, I mean, you're, you're defining on a... You can create as many sessions as you want.

Yeah.

Like, there's no limitation. I mean, we have some rate limits obviously, but, um, but there's no scaling parameter for you to say, "Now I need to run-

Oh, yeah

... ten thousand pods here."

So, so currently no, no such thing as like, uh, like HPA in the Kubernetes, like you can do these kind of setting, scaling, uh, parameters, right?

Well, you-

Like amount or whatever.

You, you don't need the, the, the-

You don't need to-

You don't need that-

... 'cause we're managing it all for you. You're just making API calls. That's all.

Okay.

Yeah.

Okay.

Yeah.

Sure. Uh, thanks for your, uh, answer. The last, uh... By the way, I, uh, the offering link does not work for me. I'm not sure-

Oh, no

... if it's the only one. But yeah. Thank you for the sharing.

Thank you. Thank you.

Yeah, go ahead.

Hi. First, thank you very much. I've been having a ball with everything that you've been releasing. And, um, I'm really interested in how you're using UI. Um, you're spinning up a server, you're distributing this, but how might we deploy that either internally, but I'm interested into sort of a customer base. If I want to make, for example, an astrologer-

Mm

... um, that person has to enter sensitive data, so it's actually really nice for me to be able to, like, have that in a sandbox.

You should have a layer. Yeah.

So there are some procedural things. You get like a star chart example.

Mm.

I'm not building this, but it, it's a good example. Like, I can have a sub-agent do an interpretation that could last for like ninety seconds. It could create some assets, et cetera. When the outcome is met, it could shut down. So this gives me a way to actually just make a really clean implementation where all I need is auth.

Yeah.

And then everything can spin up from there.

Mm-hmm. Yeah, exactly.

So there are a couple of things like MCP channels that kind of point to this with like fakechat, for example, a way for you to kind of just use Claude and have a channel to communicate with it through UI. But I'm seeing a lot where you're just spinning up a server inside a project that's easy to distribute internally, but how can you deploy that at scale?

Um, I mean, we've-- this is, uh, being deployed today at scale in production. For example, Notion external agents are built on top of this. So this is something that exists today that you can go and use and, uh, basically leverage Claude Managed Agents through Notion and Atlassian for Jira integrations, for example. Uh, there, there's definitely ways to do it. I think if you have specific questions, we could talk, uh, after-

Yeah, we, we can talk after this. But typically-

Mm-hmm

... you would probably build your product, your UI that you're distributing to your customers, and then calling into us for like your actual agent-

The phrase everything-

... creation management

... yeah, everything is through the API.

Everything's through the API.

Yeah, exactly.

That's, that's right.

Yeah.

Thank you.

So you'll manage all the API calls and build your product and UI around that.

Thank you very much.

Yeah.

Hi.

Hi. Hi. First, thanks for a great workshop and demo. And, uh, my question is not really to like the agent itself. It's just like I saw the-- and sorry, just released the Sonnet five.

Yes.

So I just wa-- was wondering if any- anything you guys could share with us. Yeah.

It's a great model.

Yeah. Try it.

It is supported in Claude Managed Agents.

Yeah.

Go, go give it a try.

You can use it today and see.

Yeah.

Hey there. Um-

Hi

... if we've got, uh, business partners that are using like Claude Cowork today or some of them more tech-savvy Claude Code to build some like spot solutions for specific business cases-

Mm

... what do you see as sort of the level of sophistication up level to start using man-- like, uh, managed agents instead? 'Cause this seems like more powerful.

Yeah.

But the, you know, Claude platform could be intimidating to-

Right

... folks that aren't engineers, let's say.

Yeah. I think, I think the console quick start is probably like the part of my demo where you're literally just like Telling Claude the agent that you're trying to build should make that pretty easy for even non-technical users to come in and build their first agent. But obviously this is an API product and is naturally gonna be a little bit more intimidating. But they're, they're comfortable with Claude Code. We do have a managed agent skill with Claude Code that basically just like in natural language will walk you through building your first agent, and that would probably be my advice, uh, to

try it out. I think the biggest difference is Cowork, you're buil- building personal agent. With this one, you're trying to build probably an agent that's like available to like your team, your internal organization, or you're building a product that you're shipping to your end customers. But if you're building personal agents, Cowork is still a really great tool to continue doing that.

Are there any skills to kind of like, take something that exists in Cowork and translate it into something that's in like a managed agent skill?

Mm.

I-- There's a skill to interact with our API so that Claude is probably good enough. So if you say, "Here's what I did in Cowork, and here's a skill to call the API," make the translation, I feel pretty confident-

Yeah

... on Five we'll, we'll do it for you.

Um, my question is about enterprise. Uh, there are cus- lot of enterprises who would like to run these things in-house because they have sensitive data sets-

Mm-hmm

... that they won't want going out.

Yeah.

Also, they may have already set up their own secure proxies to Anthropic's models. Um, do you guys... And also, the, the other question was about scaling. So within the enterprise now you can actually allocate compute to scale it our own way, right? Uh, so an on-prem kind of solution. Have you all thought about it?

Uh, definitely been a customer ask from a bunch of our enterprise customers and, and we're seriously thinking, considering it at the moment. But we, we don't already have something that we can, we can share with you. But we're, we're, we're seriously considering it.

Cool.

Yep.

Excuse me. Could you show the last slide that show the QR code of the workshop?

Oh, yeah. Yeah. Uh, are you looking for the repo or the credits?

Yeah.

The repo. Okay. Yeah.