No, That's Not a Software Factory — Ryan Cooke, WorkOS

Well, thank you for joining

the final day of the conference here. I

know it's been a long week.

But I'm excited to talk about a topic

here that is very near and dear to me.

Software factories.

Um

You this is probably not a new concept

to any of you. I'm guessing if you're

all here, you're very familiar with

software factories. You know, Ramp

blogged about their inspects

several months ago. I think maybe

December last year. I forget exactly

when. It's been a little while. And that

really kind of like excited the entire

industry of thinking like, oh, we can

actually build software very differently

now that AI is extremely capable of

writing code. Um a lot of other

companies have followed suit. Um and

there's kind of like a standard set up

here. Uh you have a sandbox that you put

your code into. Um you put a an AI agent

in there. Maybe a Claude code section or

an open code.

Um and with a prompt, it generates a PR

and you merge it.

Um

You might be building this already

[clears throat] yourself.

We have taken a slightly different

approach at WorkOS.

Um and the motivation for what we are

building is that we see a lot of the

kind of success metrics um really

focusing on output, right? Like you'll

see posts on X and blogs

talking about the percentage of PRs or

the number of pull requests or how much

code uh is being generated by AI

moving into production. Um and this is

just really output metrics that

um

you know, this can actually disguise how

well these systems are working.

Percentage of PRs or number of PRs

may be lying in overall increase in PRs.

It's very hard to distinguish

whether the output is actually driving

outcomes.

And so that's kind of where we focus our

efforts when we're thinking about a

software factory. It's like if we're

going to go put engineering time into

building this level of automations and

there's a

bit of work that needs to go into this.

Full-time teams dedicated to this.

How do we want to think about the

success of this and whether this is

actually driving value for our

organization, our engineering

organization?

And so we measure instead of just code

output, we're looking at outcome

metrics.

And that is really about like are we

accelerating our ability to deliver

features?

We think that the dream of the software

factory is that it gives each of our

engineers a small engineering team for

themselves. And so the consequence of

that really needs to be that we're

building more and we're shipping more

and that as we embark on complex

features, those can get built a lot more

a lot more quickly.

So what does this actually look like?

Well,

we kind of started with the same thing

you've seen in other other

factories.

We start with a sandbox we built on top

of Cloudflare. We put an open code model

router in there. We're feeding it

prompts.

Um,

and we kind of very quickly kind of ran

into this situation where this wasn't

driving more outcomes. This was not an

incremental increase or an exponential

increase over engineers just driving

cloud code on their laptops. It was

actually pretty indistinguishable for

us.

And so we really kind of started to

think about, well,

how can we embed our engineering

processes into the factory itself? It's

not sufficient that the factory is

producing code. We actually want it to

take a lot of the other work that our

engineers do to produce products and

automate that as well.

So, we kind of split this into two

parts.

One is we have the system we call TARS.

This is a way for users to interact with

the coding agent. And it is embedded

into the tools that we use. So, not just

Slack, but also Linear and GitHub.

We subscribe to webhooks through TARS so

that TARS can actually track the

progress of projects

in addition to generating the code

outputs.

And then we built a separate system

called Horizon.

And this is the infrastructure

orchestration layer. This is a lot more

similar to

Inspects and Minions and and some of the

other systems that other companies have

built.

But, it sits in front of an MCP gateway.

And I'm going to talk a little bit more

about why that MCP gateway has been

really transformational for us.

So, by

feeding our webhooks, activities that

are happening in other uh source code

systems and project tracking systems,

we're starting to get to that level of

autonomy where

our

factory is performing product

engineering work, not just code work.

So, what's an example of this?

In Linear, we can define tickets that

have dependencies. So, one ticket blocks

another ticket. Pretty common for, you

know, breaking down uh large units of

work into smaller ones.

Because

uh TARS is getting webhooks on ticket

completion, it can automatically pick up

the next ticket in a cycle. And so, we

can do through our planning process

construct like a map of how this plan we

think this plan is going to get executed

and TARS can start to execute on that

autonomously. The other thing that we

have is in between those steps when a

ticket is completed, we can ask TARS

"Can you reevaluate the linear project

and let me know if it's missing now

tickets?" Because as you do work, you're

learning about where the gaps in your

plan. And we want to continuously keep

our our plan fresh by using the agent

itself. Because the agent is determining

that there may be missing pieces to what

we initially planned for the project and

the goal that we have in mind.

So, we run this product engineering

culture at WorkOS. This is where

engineers are responsible for a lot of

the product functions. We don't have

product managers on teams today.

Um and one of the rituals as part of

this product engineering process is we

create a hilltop document. It's a PRD

and it is intended to

both define what is the purpose of this

project, but also incorporate

um what are customers talking about?

Like where are we seeing the need for

this unit of work? Uh it looks at

competitive analysis. Like are there

similar products out in the market today

that we can draw inspiration from? We

start to bring design early design

screens into this.

Um and we outline like what are the

major milestones. And so, by kind of

consolidating and canonicalizing this

information into a document, this is

something our product engineers have

been doing uh throughout the entire uh

company.

Um

now we can ask give this resource to an

agent and an agent can break this into

units of work.

Uh and so, this has been a really uh

powerful way for us to take existing

processes

and encode them into the factory

themselves. Like I mentioned that it's

listening for web hooks. I'll show an

example of what this looks like, but it

it can see that the hilltop document

through linear tickets has been reviewed

and approved and by virtue of it being

approved can start work on this project

automatically. So we don't need a human

to be shepherding this agent through

every single step of the life cycle.

So we created an agent specifically for

this part of the process. We call it the

PM.

And it is doing the first draft of the

hilltop based on the brief specification

that we give it.

It's adding context to that in addition

to reading the human reviews and then it

picks up the implementation and breaking

that into tickets.

And there's an opportunity for humans to

stay in the loop for each part of this

process. I mean it's very common for a

human to intervene or comment on a

ticket that's generated by AI, give it

further guidance or refinements.

Um but it's really about that like cold

start problem where we can kind of get

over the hump of creating all these

different resources. Okay, so what does

this actually look like?

So here's a project I kicked off last

month two months ago.

Um we're adding a new API to products I

lead called Vaults.

And with a command here in

um

in Slack

I can kick off this project with a

simple description, usually a few

sentences of what I'm trying to

accomplish.

Tarsin goes in and creates all these

resources for me. I don't need to go and

create the project in linear. I don't

need to create a draft of the notion

document.

Um

all of the decision logs and the open

questions.

Um Tars is creating a first pass at

that. And so this then gives me an easy

framework for me to step into as the

lead product engineer and start giving

more specification, rounding out areas

that aren't well defined with knowledge

that I had know about what we're trying

to accomplish with the project, and then

hand it back to Tars uh for execution.

So, here's an example of uh the project.

Um and it's like I said, it sets up

these first

milestones around our product

engineering process. So, we haven't done

the hilltop review yet here. When that's

done, we'll uh

we'll we'll mark this ticket as complete

and Tars will pick that up and move on

to the next stage of project

implementation.

So,

what we found is like a lot of

teams are now doing more project brief

and hilltop work because our agent can

seed all that information. It's that

blank page problem with writing. If you

can come into a document that already

has information, and you know, these

agents aren't perfect. There are times

where it grossly overestimates what

we're trying to accomplish with this

project, and we have to cut out a lot of

the scope that it comes up with. But,

that's fine. That's a lot simpler for an

engineer to add input into into rather

than them spending time to set up all

these primitives themselves.

And then we can continue to drive this

work, you know, through Slack, through

Linear,

um and ultimately uh results in PRs in

GitHub.

The other thing that this lets us do is

we don't necessarily always need to use

our coding agent to implement uh pieces

of work. Uh Devin is very popular. We

have a lot of folks that really like

using Devin. They can point Devin to

these tickets and the same documentation

to give Devin the context it needs to go

and implement different parts of the

work. Uh same with the local cloud code

if you're just using Opus in a local

harness.

Um it can grab through MCP all of these

uh

all of these pieces of documentation and

use that as context for its work.

And then, of course, we have people that

collaborate in these project channels as

well. You'll notice our security team

will come in and look take a look at new

projects, weigh in on the security

implications, and so forth.

So, I alluded to we we kind of early on

in the the development of our factory is

created our own MCP gateway.

Um, we call it our context engine.

Um, this connects into all of our

internal systems, but it also builds

system prompts and context around what

how to navigate these tools and when to

use these tools.

So, it is connected into Snowflake,

which is our main data lake. We have

semantic tables that we've built in

Snowflake that describe product

utilization or customer conversations.

And then, we can provide an agent

through our MCP server in the tool

description

list of here are the tables, this is the

content they

contain. If you're trying to answer

questions about this type of content,

write a query for these tables. And so,

it gives us a little bit of orientation

to an agent to understand how to use how

we use our tools, how we've organized

linear in Snowflake, and give some

guidance on where to find information.

What has been really

surprising about this is we actually now

use this MCP server across a lot of

other different internal tools.

We open it up for people to query

directly from Slack

to do data or customer analysis. And so,

while we initially kind of envisioned

that this would just be a way to connect

our agent or coding agent to all of our

systems. Uh this has been an incredible

piece of leverage for a lot of our

internal teams, and we're building a lot

of other tools on top of this MCP

gateway. So, if you're just getting

started with thinking about a software

factory, it's well worth your time to

invest in an internal MCP gateway server

that both connects all your tools,

has descriptions that can tell the agent

how to use those tools and how you

specifically organize the information in

them, and I think you'll find that that

ends up being useful in a lot of other

use cases.

Like I mentioned, we try to automate

other parts of our software development.

Um we have like bug

bug requests that come in through Slack.

Uh so, TARS through webhooks can listen

to those and take a first pass at

triaging and implementing opening PRs to

fix bugs.

Uh we have a lot of our customers uh in

shared channels in Slack, as well.

Uh we found TARS to be pretty useful to

uh triage

support requests that we're getting

through them, because it can look at the

code, it can often understand like where

the customer might be running into

problems with our product uh from the

code perspective.

Um I showed you an example of how we

kind of organize major features around

this.

Uh and then the last place I think is

really exciting, we're all trying to get

to the self-improving software, or the

self-driving software. So, we're using

uh TARS to actually build out our own uh

sandbox infrastructure.

So, we want to move off of some of the

sandboxes as a service and own that

infrastructure layer ourselves. Um

Some of the reasons we have for that is

we want really deep control over the

session information, and

uh be able to move workloads around

different parts of our infrastructure.

Uh so what we're building out next is

our memory layer and we want this to be

an evergreen context uh about what every

person in the company is working on,

what team do they sit in, what products

are they responsible for, and then at an

organization level, what is the

semantics about Work OS and how we do

our work. And this we want to be able to

plug into both our software factory, but

lift that context out of our factory and

into our other AI tools as well.

So, I think this is kind of our our

major takeaway of where we've seen a lot

of folks invest effort just around

outputs and where we think we can

actually get that human exponential

value out of our factory.

Can we be actually delivering values to

our customers? We think a lot about

shipping and customer impact and so we

want to encode that into their our

software factory itself. Um certainly

worried about, you know, introducing

instability into our products. So, all

right, can we measure

uh defect rate and other

um

you know, time to recovery uh metrics?

Um and then a lot of anecdotal, like we

want to see our engineers using TARS as

a sign that this is giving them value

and accelerating their work. We want to

see that they're moving off electing to

move out of their local harness and into

a cloud sandbox.

And then the real goal is we can use

this information because we own the

infrastructure, we can see what's

happening in the infrastructure. We use

this information to self-improve the

factory.

We want it to learn, we want it to get

better about how it writes code.

We want to see where um our engineers

can improve their skills. There's so

much moving quickly in AI. It's kind of

like a constant race to keep up with the

latest

um

you know, tips and techniques.

We can actually point agents at sessions

and see where are the gaps and how

people are using

our factory. Where is there a skill that

we should be building because the agent

made a mistake. What skills are no

longer relevant? Maybe the code has

changed to a degree where a skill that

we wrote six months ago is now obsolete.

So, that continues verification.

Semi-online

is where we kind of see a big advantage

to actually owning all the

infrastructure ourselves.

So,

say it for the third time, it's worth

repeating.

Sandboxes are great.

It's cool to run an agent and have it

open a PR.

We really think about our software

engineering practices and processes and

we want to encode those in automation.

I'm Ryan. I'm one of the engineers here

at WorkOS. We have a booth right

opposite the speaking area. If you are

building software factories, I would

love to talk to you. I would really love

to hear how you're handling

authorization. This is something I don't

talk about because we haven't figured it

out yet for ourselves, but maybe you all

you all have some insights. So, please

come by the booth, talk to me, would

love to chat. Thank you.

>> [music]