How Anthropic Builds: Lessons from Labs — Mike Krieger, Anthropic

[music]

>> Joining us on stage is the co-founder of

Instagram [music]

and a member of technical staff at

Anthropic.

Mike Krieger.

>> How's everybody doing? I mean, good

morning.

Nice.

Um

>> Mike, thank you for releasing Fable just

in time for us.

>> Exactly for the conference. We timed it.

>> [laughter]

>> Um we're we're so glad to have you. Uh

you're uh one of the preeminent builders

and you're a leading labs at Anthropic.

Um how has your

model usage changed as as you've, you

know, seen models internally grow?

>> Yeah, I mean, for me it's been like both

the model shift and then my role shift.

So, I for like the first 2 years I was

at Anthropic, I was chief product

officer. And then I kept seeing people

build with the models and the FOMO just

kept increasing because I was you know,

use the models as much as possible. But

for example, on product strategy I would

write a strategy doc and then have

Claude critique it and maybe you can use

a workflow, but it's not quite the same

as like building in that pure way. And I

was like spending all my weekends trying

to build with it and I realized, "Okay,

I actually just need to shift. It's like

way too interesting a time." And it's

actually an interesting trend I've seen

now like several people that were CTOs

at other places are like now joining as

ICs at Anthropic and other places. But I

made a role shift and it was actually

right around the time where we started

getting sort of internal snapshots of

what became Mythos and Fable. And what

was really interesting watching that

sort of shift was um that

kind of change between I have an idea,

I'm going to like sort of break it down

in my head much more how I would do

engineering normally, and then kind of

iterate through these different steps to

moving to much more of the paradigm of

I'm going to describe the goal, like go

off and work on it, and then like we can

talk about what trade-offs you you know,

surface some questions along the way,

but then

figure out what where you landed and

where we can go from there. I find it's

hard. I don't know if people have this

experience where

people

people's only been re-enabled for a

couple of days. People's definitely way

way smarter than me. So, sometimes it'll

finish work and be like, here's the

trade-offs I made. I'm like, can you

explain it to me like I'm a little

dumber than you are because I need you

to like sort of break this down for me.

But, that's been one sort of big change

is sort of moving from that task

delegation to like express the end state

and then have it go and and cook on it.

>> Yeah,

we're all learning how to delegate

better. Tariq did us a huge favor

yesterday.

We

Did you want to read it in the

newspaper?

You know, that we have we have

write-ups of talks now in in like the

next day's newspaper.

>> He said, be unreasonable.

In what ways you know, you know, have

you been more ambitious?

>> Yeah, you're prompting.

>> I love that I I mean, I love that

framing. We actually just hit this

today. I'm one of the labs initiatives I

have is internal product, and somebody

was like, hey, it doesn't work the way I

want it to,

and can you make some changes? And I

realized, oh, I'm just going to go ask

Claude to do this. Like, why don't you

ask Claude? And this was a non-technical

person. So, I actually think as an

industry or even as a product team, we

have to teach people to be more

unreasonable in their usage, and it's

sort of hard to imagine. I think that

that

if I can digress for a second on product

design, I think right now the like kind

of first generation of AI products, we

put them too much in a box and constrain

their their sort of access to tools or

kind of degrees of freedom, which means

it was much harder to be unreasonable,

right? When you say, do this thing for

me, and then it would be like, well, I

can't. I can barely like I can write

code, but I can't really run it, or I

can kind of introspect my environment,

but not really.

Um and I think as you see our own like

product progression even with things

like co-work where like, you know, does

every single like knowledge worker need

a virtual machine that can write bash?

Like, on the face of it, no, but then

when you realize, oh, actually, that way

it can remediate an issue where, oh, I

tried to parse a PDF using our built-in

PDF parser. I hit this yesterday and it

was like, ah, I can't parse it this way.

Well, okay, well, I can probably write a

script that can do this as well. Um so,

I think that's it. My most unreasonable

thing though was uh one of our labs

projects I wrote in Python like near and

dear to my heart. All of Instagram was

in Python. But I think they're finally

converting it to PHP now that they have

um like models that can do it. I know.

>> [laughter]

>> Tokens. Um and uh for deployment I

realized that Cloud Code had like

figured out a better deployment story

with Bun. And I was like, okay, I need

to port this whole thing from Python to

TypeScript. Like, as a, you know, if I

put on my like 2010s engineering hat or

even my early 20 20s, like, that's a

dumb idea. Like, who would ever port

like, at that point, you know, a couple

hundred thousands of lines of code. Um

but I was like, I think this is doable

now and I basically created this dynamic

workflow setup and over the weekend had

it port the whole thing, like, verify

it, double-check it, then read both code

like this basically churn and churn and

churn and then came back Monday to a

completed workflow that was a ported

version of that thing. So, that probably

ranks on like the more unreasonable

things. Like, yeah, just port this

entire Python code base to TypeScript,

get it working, get it deployable in,

you know, a weekend.

>> Yeah, I mean, a lot of people are

talking about the the Bun Zig to Rust

version. I think a lot of people are

also like, well, it's a compiler, it's a

it's a runtime, it's got lots of tests,

easy to do. Can you port Instagram,

which you would know very well, to PHP

like that? Like a like a product.

>> Yeah, I mean, I think the product side

of it it's even I don't know if it's

easier or harder. One of the things we

did at Instagram, this is when Python 3

came out and we were able to add type

hints for the first time and it was

people had a lot of internal

conversations like, are we going to run

out of steam on Python? And my

perspective was always like, I think we

can take this way further than we think

we can, uh, but I think types are going

to help us not sort of be in our own

way. And we built this thing called

Monkey Type where we basically like

captured runtime type like basically the

types that were actually getting used in

production and then map those back to to

the types in the code base. And I think

because of that sort of pattern, I think

there's really interesting ways in which

if you're doing sort of conversion or

sort of cross compiling using LLMs, you

can also lean on production data a lot

more or run sort of like segmented

tests. I think that like there's a lot

of, uh, things you can do there. But

yeah, I think it's, I mean, the sky's

the limit there as well. I think the

hardest part is always finding the

boundary around where you can start

doing it incrementally without trying to

boil the whole ocean and like swap it

overnight.

>> Yeah, I mean, your users are your test

ultimately and, um,

you know, I we I also read another

article in the newspaper about how you

could just use rollouts and sometimes

you don't really know, uh, what you're

going to need it for, but when that

infrastructure exists for your

experiments and to roll things out, it's

enabled so much.

>> Yeah, I mean, I always found this was

advice we got. It's like we launched

Instagram and the happened to be the

first week everything melted cuz we

didn't really know what we were doing on

the back end side of things. And, uh,

coincidentally that week there was like

a lunch that one of our investors just

scheduled like not even for us. It was

just a

infrastructure lunch. And we ended up

spending we totally like monopolized

that conversation cuz everybody had

their own opinion about how we could fix

our scaling. Um, and like the two pieces

of advice I got there is like 2010 that

I like will forever retain is like, um,

like basically like

pre-measure everything that you think

you might even remotely need because the

worst thing is an outage where you're

like, well, is this like number normal

or is it high? And like, oh, I don't

know because I don't have data until I

just added this metric. And the other

one is being like really thoughtful

about knobs and feature flags. So even,

you know, early Instagram we had like a

very, uh, simple but really effective

like way in which you could do like ramp

outs and rollouts. And dynamic config

too where, you know, a lot of our

runtime configurations had to be

changed, you know, in a matter of

seconds so that we could handle load and

being able to like do that in a first

class way was was really important. I'm

seeing that definitely in in AI as well

where, you know, we're making all sorts

of different trade-offs and having that

kind of runtime configuration is super

key.

>> Yeah.

Uh my my favorite scaling story of

Instagram by the way, I think it's like

your launch day when you you DDoS

yourself with the email.

>> Yes.

>> Which I people should look up that story

if uh if you haven't seen it. Um I

wanted to go into tags. Uh

very very major shift. Uh it's it's how

60 something percent of your code is

written today.

>> Yeah.

>> Um how do you square that with

everything you just said where it's like

very dynamic? Like you don't actually

ship one app, you ship one app with

3,000 flags.

>> Yeah.

>> And like, well, what are you working on

today? I don't know. Like it's it's for

this segment of the population.

>> Yeah. Yeah, I mean, I think there's a

bunch of things. So, like with I was

really excited. I was talking to Swix

earlier like, I'm really excited that we

have tag out there because it is uh how

we've been working for a while and I

would get up on stages and people like,

"How do you work at Anthropic?" And I'd

be like, "Oh, yeah, we use these things

like that are not quite Claude code, but

you know, uh but it's hard to describe

it, but I mean, if you like got to poke

into Anthropic, like you would see uh of

course Claude code usage for things that

are like more interactive or if you're

kind of iterating on a particular uh

sort of

sort of specific thing where you want a

lot of like a high sort of bandwidth

back and forth, but most usage is

actually much more delegating uh via

tagging and via tag. And you can say

like, "Here's the And the reason it's

really interesting is how multiplayer it

is." And it reminds me sort of of like

um actually like Midjourney, like the

fact that everyone was on Discord seeing

how other people were using it. I think

it actually to your earlier question

really helps with that unreasonableness

or ambition where the first time you see

somebody tag Claude and be like, "Hey,

you know, don't just fix this bug, but

like now you are responsible for this

part of the code base and I want you to

monitor this feedback channel and

proactively take on tasks and then fix

them and then also take like, you know,

if this API changes, do that." Like I

saw somebody do that. I was like, "Oh,

wait, I've I've totally underutilizing

this thing. I've just been using it as

like a glorified Claude code and slack.

Like that's definitely a totally like

sort of new version of it, right? And

then more advanced version is really

trying to start thinking of it as a

teammate that is actually sort of holds

context, has memory, and can be

proactive. And that's just really

changed how we operate internally. It's

much more like this multiplayer async

proactive way than it is a you know,

most people often their own CLIs.

>> Are you bottlenecked by code review and

get? Obviously, there is code review,

but someone usually still looks at it.

Is there a world in which you just merge

it in?

>> Yeah, we're it's a really good question.

We are definitely still bottlenecked on

reviews, especially for things that are

like touching some architecture pieces.

And it's actually more subtle than just

being bottlenecked on review, cuz that's

you know, okay, we can carve out time

differently. It's like bottlenecked on

human ability to even

like fully conceptualize what we're

doing. So, one of the reasons we built

Claude code artifacts that we shipped a

couple weeks ago was partially for that,

which is

you would send somebody a PR, and then

they'd be like, I don't know, man. This

is like 2,000 lines of code. Like, it

looks like code to me. And what we

started doing instead is sharing much

more like, here's a Claude code

artifact. Like, here's the explanation.

Here's the intention of the the change.

Here's the trade-offs that were made.

And like, I think that's going to be

much more be the trend by which we

communicate, which is the code is

ultimately, you know, verifiable using

some things, but actually like

discussing intent and trade-offs, and

then measuring in production I think

that at least the direction of travel

we've we've gone. And I don't review

when I get a pull request, I wish I

could say I reviewed every line of code.

I definitely do not. I like actually

talk to Claude about the the code and

say, all right, like, these are the

questions that I would have. Can you go

investigate it? So, it is kind of

Claude-powered code review, but still

human-driven. And and for the really

important ones. And for the ones that

are like cosmetic visual changes, it's

much more like look like we'll fix

forward if we need to fix forward, you

know.

>> Yeah, totally. I think a lot of people

are here are trying to figure that out,

too.

I wanted to talk also a little bit about

Anthropic Labs in general.

Nilay Patel,

who you've probably met before, loves to

ask ask the question like draw the org

chart.

Like how like people, you know, you ship

your org chart. Like I think it's

important like everyone knows cloud

code. Now you've got tags.

Um How are you structuring the labs?

>> Yeah, it's a good question. Because what

we were trying to wrestle with was you

want sort of people to be supported

like, you know, I think the death of the

engineering manager discipline has been

greatly exaggerated. Like I think

there's still a lot of coaching and

interpersonal pieces and personal

development that I think is still

really, really important. But especially

in a labs type group where like our

whole cadence is two-week reviews where

every project goes up for we call it

persevere or pivot. So basically every

project is up for review and either it's

time to, you know, keep going,

persevering, or you know, it's time to

pivot it or even shut down. And, you

know, we've shut down projects basically

every single one of those cycles and

it's like the more you do it, the less

it's just like, "Oh no, my project is

shut down. I failed." It's like, "No,

that is definitely the intention of the

labs team is to prototype quickly, try

to ship internally, maybe get it to

early access, and if it doesn't work,

wind it down." But because of that kind

of like rapid iteration, it means that

if you align the org chart too much to

the individual projects, you're going to

end up like re-orging every two weeks,

which would be a total nightmare. And so

we've actually ended up with this

interesting setup where like the the pod

or the team that is working on a given

we call them bets within labs,

definitely just draws upon like all

right, somebody from product, somebody

from the eng team,

you know, I'll jump in when it's a

product I'm particularly interested in.

I'll come in and work together with the

team on it. And that's the unit for that

time. And there is the concept of a bet

lead or a directly responsible

individual. But the interesting thing is

that they don't manage usually any of

the other people, which kind of breaks

the that kind of previous way in which a

lot of these things were done. But I

think it leaves it leaves us to be

really flexible when you say, "Okay,

actually this project is not going to

work out. Let's disband and keep going

and it's not a big deal." And the engine

manager is much more playing the like

make sure every individual is assigned

to the thing that they're most excited

about and that they're working in the

best way possible. Now, what we do sort

of solidify is when there's a product

that has like legs. Like Cloud Design

for example, started in this sort of ad

hoc sort of grouped way and then now

that like we've shipped it, it's gotten

traction, we've done like a big second

release in June. Like it's becoming like

we've hired people for that specific

team and it has more of a of a

structure. So, it's like loose until it

gets solidified down the line.

>> What's the future of Cloud Design? I

think a lot of people are very

interested in It's one of your biggest

launches this year.

Where does this go?

>> I think for me, I mean

the things that are holding back Cloud

Design from being even better is better

interaction with our other surfaces. So,

you know, I was designing something or I

was talking to to Cloud Code the other

day. I'm like, I want a really much more

seamless like what I'm talking about the

design for it, you know, and then after

design back to that. I think in general

it's I mean this goes back again to kind

of unconstraining Cloud. Like the fact

that our surfaces don't talk to each

other as well as they could. I think

really holds back a lot of interesting

ideas around what we could do. So, I

think that's one like kind of major area

that we're looking at.

And then the other one is people like

the lines between a Cloud Design and an

app get blurry and blurrier over time.

Like I've seen people Of course there's

no like persistence but build like fully

functional like even games which is

definitely not what we designed Cloud

Design for but you can do it. It's just

HTML and JavaScript.

So, blurring those lines even further

and thinking through like what is the

path from a like fully featured design

that looks really well to really good to

something that is maybe more like a

artifact where you're actually able to

go and you know, persist data and share

it with others and build from there. So,

I think that those lines get really

interesting over time, too.

>> Yeah.

A big part of design is having taste. I

actually asked Fable what Fable wants to

ask you

and this this this is what Fable came up

with. You deleted almost all of Bourbon

to get to Instagram which is like you

had a whole you know solo mode whatever

thing and you went to Instagram. What

would you delete in AI or more spicy

what would you delete in Claude?

>> Oh, I like the spice.

I think I mean we have it's interesting

we have a one of our slack channels is

like project unship which is like what

is in the product right now. It's

I mean this is hard at Instagram. The

Instagram we what

things that had like four to five

percent usage you're like oh that's

really not very many but then you have

like 20 features that each have four to

five percent usage is like the classic

Microsoft Word problem of like everybody

uses some disjoint subset of the of the

functionality. So that that's always the

challenge. Now I think we're younger

product so hopefully we have less of

those things. Like we unship styles I

think recently where it was like used by

a small percentage of people and was not

really AGI filled in a lot of ways it

was like very sort of prescriptive in

the way that it worked and skills very

much better applications and then like

that. So I think you have to be willing

to take the primitives of like one

generation of AI and like unship them or

at least like supplement them or

supplant them with the next one as well.

I think the biggest thing is I look at

it and I've been spending some time like

outside labs on some of this is like man

like we're asking people to make like

code versus co-work versus like chat

distinctions and like one they don't

interoperate well and they can't

delegate to each other and two I think

the average person off the street could

not explain to you why those are all

different. So I think deleting some of

the product complexity within our our

code or our product I think is a a thing

that would would serve well. Also

because then Claude can do what it needs

to do and and do well. Like there's

nothing more frustrating than having a

co-work session where you're like great

I've mapped out exactly what I want you

to build and then be like can you please

like create a paragraph that I can paste

into Claude code? Like that is some 2020

you know kind of workflow there that

really shouldn't exist anymore.

>> Yeah. Um

I think

drawing lines on what you don't want to

do and also sort of leaving room for

others is interesting. Um a lot of

people today is like the startups day

for AI E or obviously very

sympathetically aligned to startups. Uh

but there's some anxiety in the room

because tomorrow's Anthropic could wake

up and publish

some markdown files that destroy my

industry. Um so

>> [laughter]

>> uh why should we not all just give up

and join Anthropic? Like why bother

starting any other company?

>> Um I I mean actually joined one of the

main reasons I joined Anthropic was

because I saw how much this was like,

you know, the models weren't that good

at coding up but they were getting

there. Like how much it would unlock

like whole like next generation of

startups. Not because it was going to

solve their ideation or their taste, but

because like it would make

experimentation way simpler and and

would get you to move faster. And I

still like really believe that. And I

mean it's the reality of, you know,

uh

And we saw this with like Instagram.

Like we would get questions from

investors like, well, what happens when

Google launches a photos product? It's

like Google's going to launch a very

googly photos product and it's going to

have to be bound by the integrations

that they already have and it's going to

be like it's going to play to their

strengths. And I think that is going to

be true. And I'm not like giving advice

on how to compete with Anthropic, I

guess in a way, but like it's actually

not because we're also a platform which

is like there's so much I think room to

be like laser obsessed with your

particular vertical or your industry or

group of people that you know really

well in a way that like none of the labs

are ever going to get to that level of

uh of understanding and like therefore

get that kind of adoption and user love

and and build that out. Now, it's

definitely harder in the age where like

the models can just do a lot and so

there's, you know, some of these things

can be like skillified and like maybe

don't need their own dedicated product.

But I think it's like the hard stuff is

still hard. It's like understanding the

needs of people, like figuring out how

you're going to reach them, uh listening

to them and iterating on them really

quickly. Like

it is still the case that like a group

of four or five people obsessed with a

problem is going to move faster than

those same people at any other kind of

organization that are like, you know,

subject just to the complexity. I just

mentioned the like the fact that we

have, you know, a lot of different

products that kind of interoperate. Like

that's a interesting constraint that we

have to work through. It's an advantage

in other ways, right? So, yeah, I'm

still like very long and bullish on

startups and um it's just

it tapers over the fact that like

writing code was never the like the

limiting part. You know, maybe it was on

the timeline perspective, but it was

never like the thing that was going to

like make or break a startup. It's

really that space and user

understanding.

>> Yeah. Uh domain knowledge.

>> Yeah.

>> Uh today is also our day for vertical

AI. Uh one of our

uh returning speakers and top speakers,

Chris Lovejoy, uh was always talking

about vertical AI. He was in from

interior in the healthcare space. And

then recently I was I invited him back

and turned out he you guys just hired

him for your uh healthcare efforts. Um

we also our next big one is also

finance. You know, we have a yeah, a

finance track. You guys just had a huge

finance event in New York City. Um and

where our next uh AI is is sort of

finance focused. What are you seeing

there? Any you know, any potential uh

for Claude? Obviously a lot of Excel

Excel spreadsheets.

>> Yeah. No, I think that there's there's a

lot in there, too. And that's like an

area where uh you could see the model

get clearly better at it like sort of

generation to generation. And there's,

you know, there's some good sort of

vertical specific uh uh finance startups

that have like done their own um

evals, which has also been interesting

to to track. And it's not like we're

like sort of playing to the eval, but it

is a useful sort of barometer around

like is this actually getting better um

at these finance use cases. I think the

interesting blend that's going to happen

um is this mix of, again, the model

having the flexibility to like dive in

and create just-in-time analyses or

dashboards or workflows with like some

sense of like what is the not immutable,

but at least like verified sort of set

of data. And so like uh

set having all of that be totally free

form, I think is a recipe for confusion

and is like not what most companies in

the financial services space want. So,

finding that right uh sort of cut line

where you have verifiability and audit

logging and and sort of data provenance

here, but not in a way that constrains

the kinds of applications that you can

build on top, I think is a lot of the

art that we're seeing in that space as

well. Um and I think, you know, if you

solve it well, you can you can get the

best of both worlds. The hard part is a

lot of the systems that were built to do

the verifiability audibility like are

kind of almost by design not super

flexible in terms of agentic workloads

on top. So, I think there's opportunity

at both sides of the stack there.

>> Yeah. Um I think I I also agree we'll be

exploring that in in New York. Um the

last thing I want to end on is on mental

health, which we don't talk about enough

in technical conference conferences. Um

you've seen a lot of hyper growth.

People are just always refreshing their

timelines and it's exhausting.

Um

how do you advise people who are working

996 to avoid burnout?

>> Yeah. I mean, I think this is a hard

one. I mean, and

it is, I'm sure you are experiencing

this cuz you're all working in this

industry like

it is

you know, multiples more intense and

things move much more quickly at an

Instagram like our the two things that

we were thinking about was like, what is

Apple going to announce at WWDC and is

it going to like totally mess us up or

boost us, right? So, that's like once a

year. Um or, you know, maybe a

competitor launches every three or four

months, right? And uh it is definitely

not that. It's

a topic we we do a when we do do our

weekly all hands. Usually on Wednesdays

and we have a slide that's like the week

in AI at Pinterest is and it's only

Wednesday and like and inevitably like

some competitor has shipped a new model

and like there's been a like new product

and maybe there's some interesting thing

happening

um uh on the regulation side. Like if

things are moving really, really

quickly. I think the way I try to stay

at least relatively sane, um one is like

actually carving time off and I think

the topic co-founders do a good job of

like saying like, look, like burnout if

you you out, like you're kind of done.

I've seen it happen unfortunately to

people I'm really close to and then it

takes a long time to recover from that.

Um so actually encouraging people like

there's no job that is so important that

you can't be offline for a couple of

days. Um so I think that's like a big

key like piece in there. So like let's

>> [applause]

>> strongly believe.

Um

And if it is you're probably doing

something wrong and you talk to somebody

who could be a mentor to figure out how

you can unblock that. Um and then I

think the other one as well is like I

love sports and like uh

this is the notion of like you're never

as good as like your best game and

you're never as bad as your worst game.

I think that's also really true. Like I

know like in AI there's like the you

know it's so over we're so back thing.

Like that like if you internalize that

that cycle is always going to be at play

in some way, you realize like it's never

that bad. Like Ben Horowitz's book is

the hard thing about hard things has

this chapter on like we're effed it's

over and like that feeling as a startup

that probably many of you have had at

startups where you're like oh I can't

believe this thing happened like we're

never going to like recover from this. I

definitely we definitely had an

Instagram a couple of times. And then

you get through it and like it's that

like def like defines the company when

you can actually go through that. I try

to remind myself and the team here even

with an Entropic which is like like this

is a is a fast-moving but is also a long

game. And it's like we're never it's

never just about today's model launch

and reaction or this product launch and

everything else. Like you're playing and

you're building and you just have to

trust that you're building like the team

and culture that is going to get through

those things and have that sense of

perspective even if perspective is

saying like look 3 months ago we were in

a similar position. Maybe it's not a

year it's just a matter of months but

it's still like zooming out and not

taking things not letting your internal

sort of like sense of self and success

be so driven by the day-to-day.

>> Yeah. It has anyone any coach or mentor

said something to you that you repeat to

yourself that gets you through the hard

tough times?

>> Um I think the biggest one was

this like sense of like if you're

feeling something it's really often the

case that other people on the team are

feeling it too. So this is like advice I

got from my my coach around just being

like like just verbalizing emotions like

even saying like hey I'm feeling really

stressed out about this or yeah I'm

really sad that we are shutting down

this labs initiative. I literally had

this meeting a couple months ago where I

was working really hard on something and

I kicked off the meeting like

I'll kick it off like I'm really sad

like I'm frustrated like I wish this

thing had worked out and I think that

holds the space for other people to be

like yeah I'm pissed off too or like I'm

sad too and like I think giving that

advice around like not

I think if you can get yourself to be

open and vulnerable it often like lets

other people verbalize that and then you

can from there you can be like great

what are we going to do about it like

you know it's much easier to start from

that place.

>> Yeah we actually kicked off AIE with a

session from Carol Robbins who runs

touchy feely at Stanford

and I can't think of a better way to end

than encouraging people to talk about

their feelings manage their mental

health and keep shipping.

>> Yeah. Thanks so much Mike.

>> Thanks for having me.