AI-Native Organisations Run on Skills: How to Structure and Scale Them — Imad Touil, QuantumBlack

[music]

Thank you for joining me today. My name

is Imatel. I'm a distinguished engineer

at Quantum Black. And today talk I want

to really cover the uh how AI native

organizations runs on skills. But before

I really get started, um I just want to

do like a quick exercise to show of

hands. Can you raise your hand if you

have already created and using skills?

Amazing. Now can you keep your hand up

if you are using it and sharing it

within your teams?

Great. Now keep your hands up if you

have governed maintained skills across

your organization.

Amazing. I see a few hands. But that is

what this talk is about. Today I'm

really looking to break down why this is

really critical, how it's important and

how actually you can adopt it across

your organization. But before I get

started, what I really want to cover is

actually the agentic software stack. So

the agentic software stack has two

loops. So the first loop is all you

know, right? It's the code and agents or

the code agents harness right

at this you have some core components.

You will have your context manager, the

tool and MCPS, memories and states and

skills loader, right? But then there's

an outer loop which is your workflows,

right? Those has skills, sub aents, MCP

servers that you use and some hooks.

Sometimes you need them to have this

running properly. You will need some

enablement components at the bottom. So

what you would have is an environment

sandbox. You would have your MCP gateway

to manage and simplify all of the MCP

tools across your organization. a model

gateway to again manage and optimize for

all of your LMS either like you know

open source like running locally or or

or frontier models and also a graph

knowledge graph that abstracts your IT

core systems your codebase your skills

registry and at the end workflow

marketplace and then you have your

context layer right the context layer

will bring all of what is needed to get

the task done so this is the project

instruction think about it as the cloud

code MD file the agency file your tools

and MCP schema is actually to understand

which tool to use and when your memory

right conversation history with the end

user the human is in the loop and

finally the retrieved contents that's

you can pull from either like files your

codebase etc now what I want to really

focus on today is the workflow right and

I think this is where we kind of like

think um it's kind of a pretty simple um

uh workflow in day-to-day when you're

trying to actually create an end to-end

product software delivery life cycle in

the organization. In reality, we all

have seen the four steps. specify to

define what you want to build the design

or plan to plan you know what what are

you looking to build then you go to the

tasks you break it down into tasks and

then finally you start implementing

right I think this looks familiar I

think this is how most of our coding

agents actually are kind of shaped today

in reality that is not how it is

composed and at scale when you look at

the organization complexity

this is just one step in the journey

this is like building a product

increment

When you really look at the overall end

to end life cycle of something that a

business want to build to capture the

value out of it all the way to ship it

to the client, you will start first of

all by defining your product strategy.

What to build and how to build it,

right? You set the define the success

metrics. You identify and you break down

your plan road map for your products.

And to do this, you may need a lot of

insights, right? So then you do like

your market research. You do a

competitive analysis. You bring this as

an input with some customer interviews.

Then you go to the discovery side,

right? So then you start discovering.

Okay, now I understand what to build.

I'm going to break this down into like

some problem statements, find the

solution, validate the solution, and

then probably like experiment and then

create user stories. And before we start

building in reality actually need to

prepare your data, right? In some cases,

you will need actually to clean up your

data catalog that will support the build

of your products or maybe adjust some of

the endpoint connection and integration

to your core systems that will help you

actually build your products.

And this is where the data product

delivery comes. So you build your data

pipeline, you validate your data quality

and you put your catalog your

[clears throat] catalog data assets

ready for development. Then we kind of

like go back to the product increment.

That is where we will start. But then

building a product in every organization

that I have been uh serving for the past

18 years in my career. I can see that in

one organization you will find like

different DLC iss kind of like uh you

know scatterers across organization.

Some of it is actually for a mobile

application other is a different

department or different platform. Some

of it is internal platform that is for

your um employees. Other is actually

customerf facing. So it is not like a

one workflow that can actually build

anything you want for your organization.

When you figure out like what to build

and how to build it, you need to run it.

Then we come to the platform engineering

ops, right? That is again when you have

your provisioned infrastructure,

thinking about how you you build your

your infrastructure as code modules etc.

And then you launch your products and

the moment you launch it then you start

kind of like the journey of optimizing

the performance of your products trying

to look forward like any incidents to

resolve and then you start the loop

again. Right? So at scale when you look

at really building a digital platform

not like a very simple products that you

can you know solo build and deploy the

landscape is way complex that is

expected and what you're looking at here

is literally like probably 10 20% of

what is it and is really different from

organization to organization now going

back to the uh the stack right when you

look at the workflows there's like four

core components the first one is hooks

MCP servers and sub aents those are kind

of

given but it doesn't really bring the

right kind of like structured value to

your workflows right that's why scales

is one of the critical components in

your workflows hooks basically what it

does it just kind of like pre kind of

like trigon events to to do something

along your workflow the MCP service we

all know that you know you may need an

MCP tool but tell me like who actually

build a lot of MCPS we just use MCPS

tools that is actually provided by the

tool that we used to use before right so

we don't really own

The sub agents is just to minimize the

context window. We just delegate to sub

agent to execute a specific task what is

needed. So at the end of the day you

will find all of your knowhow is

actually at the skills level. And if you

don't have the right structure of your

skills then you're not really having a

deterministic workflow. And one thing to

mention is workflows think about them as

harness blueprints that actually shape

the behavior of your coding harness for

example in the runtime.

Now looking at the rise of skills

adoption right so just eight months ago

entropic published the first article

about skills right two month later we

had this standard an open standard that

is adopted and starts a lot of agent

harnesses starts adopting this new

standard [snorts] around February um

this year we have seen most of the

actually agents adopted this even though

you don't see them right if you pay

attention when the the uh the agent is

thinking And you can see that he's

pulling skills um as he's going and

doing the task. The other implication

here that you need to pay attention to

is actually the number of skills

created. Right? This is just like

literally a snapshot that I did across

some public GitHub repost, some public

skills registries, right? There's way

more than this publicly and within your

organizations, right? So the de the

creation of skills and the demand is

rising. But we need to understand why

in the latest skills bench comparing the

latest models against like you know

running the same task against auto

engineering and cyber security without

skills it did well right because that's

what expected and it's going to continue

to be improving day after day right but

then when we applied skills that's our

more deterministics the outcome was

clearly higher than than That's it.

Now if you think about now like how like

the anatomy of skills and how you

actually design and implement skills in

your organization. This is not like a

new problem that we are trying to solve

here, right? We have solved this with

the microser kind of movements, right?

So your microservices need to have all

of these design principles, right? This

is like a software kind of like you know

problem that we use to solve is is

similar. So skills need to be reusable,

right? Need to be modular, need to be

disco like you can discover your skills.

So if you are sitting in one team and

you need the skills you actually can

automatically discover and capture the

skills it's portable that you can

actually use skills across workflows but

also you can use skills across harnesses

again everyone adopted the same

standard. So if I'm having a skill on

cloud code and I want to move it to

cursor, it's going to just work.

Specialized skills that is where the

value you should not build like a one

skill like a monom or again it should be

specialized to define one tasks

specifically. Composible uh skills need

to be designed in a way that's actually

can compose. So you don't have

duplication across your skills when

you're trying to run them conflicts each

other. Consistence that is actually what

one of the key items of skills is

actually consistency and deterministic.

And finally cost efficience and for cost

I can go for another hour talk but it's

basically the skills comes to solve a

key problem around the context window

right is actually putting with the

disclosure pro progressive disclosure

pattern the right skills the right

amount of skills in the right time to

solve the right problem and that's

reduce the token usage. This define a

new unit right that makes your knowhow

in your organization executable portable

and cheap.

On the right hand side, you just a very

simple example of data retention policy.

Right? When it came to regulation, you

need to understand and make sure to

instruct your agents while you

manipulating. For example, um your

customer data, you should make sure that

this data manipulates according to the

regulation, right? And that bring me to

the next example.

On the left hand side, you can see that

there's like a think about like a a

catalog of skills. And in the right hand

side is your harness and the output is

on the right hand side. Right? So the

composible skills at the regulation

level. You have the skill that I just

showed earlier which is the retention

policy. But you will need disclosure

standards. You will need the GDPR rules

that need to be respected. You will need

the fill in templates. Right? So all of

this kind of think about it is like

defining how any data any feature that

is built across your web, mobile um like

you know different applications across

your organizations is really respecting

these rules and this gets pulled

automatically by on the runtime by the

regulatory disclosure review workflow

and the outcome is expected is

deterministic. You have an audit uh

audit reports that you can actually

store. you have um specific uh

identification of if there's anything to

improve and that this kind of like loop

back to improve your your codebase.

However, if we don't govern skills, we

will start creating a new class of

technical depths, right? First of all,

you will find out that you are having a

lot of duplication in your organization.

So if teams are not collaborating and

everyone is think about it using the

same technology stack the same

infrastructure you're for sure building

the same skills over and over again

without sharing them quality. If you

don't test and make sure that you're

maintaining and you're validating your

skills not against your task but also

against the latest models that comes

right then the quality starts

degragating over time.

You should be able to discover your

skills in in reality without a

governance you cannot really discover

it. Think about it as the the backstage

the IDP right it's come to solve a

problem where okay I need to understand

who owns this microser back in the days

right I don't need to talk to anyone I

just need to tap into the service

catalog and immediately find who

actually own it that is brings ownership

part if you don't have an owner then no

one will be able to maintain scale those

skills

composibility is not something that

comes by default you need to have a

governance way to to align what to build

and how to design it is think about the

domain driven approach that we have been

taken also for for many years, right? Is

similar to how you shape your skills

catalog.

Security again some of uh all of us like

we experiment with the public skills,

right? But when you think about it, some

skills may have some prompts injection

and skills actually does have scripts

because that is the deterministic part

of it because it can run a specific

script for a specific task. So if you

don't have again a pipeline that check

your security, you may be pulling

something that is insecure

and permissions. Not every skills is

actually something that anyone in the

organization should access. Some skills

may have some business logic that is

very sensitive, right? So the access

control is is also crucial at this

stage.

Now how to bring this to your

organization? First of all, you need to

allow at the individual level to create,

test, improve and use those skills.

Again, it shouldn't be random. It should

be structured way. There's a different

tools out there. You just need to decide

which tool actually you want to agree on

and you use that mechanism at the

individual level. The moment you create

a skills, you need to be sharing it with

your team, right? Your team starts

collab collaborating to improve the

skills and think about it. You build the

same technology stack, building the same

products. So, it's going to evolve

really quickly. But then you move on to

a very critical point which is the

centralized platform. That is where all

of what I've been covering so far come

to play. You need a centralized platform

that have a catalog with metadata in it

that's actually can discover skills and

could be searchable. You can have an MCP

that's actually plugged to this cat

catalog, search for the skill and a CLI

to pull the skills back to your either

your IDE if you're locally or to your um

sandbox in your factory. Then you have

the dependencies. So you need to

understand the dependencies between the

skills as well. You have the versioning

uh and and life cycle. So you understand

which version of the skills is actually

the latest. Um and a very good example

when I'm using for example building a

functionality I can the agents

automatically up capture that there is a

latest version of the skill and pull it

right. So this versioning help also to

pull the the the right latest changes

from the skills registry access control

again as I said if you don't know who is

accessing what that is a huge uh uh gap

and finally evaluation observ

observability and then all of this is

actually play around a governance and

this is where technology stop solving

the problem right so you figure out all

of this all good now who's going to

govern this and that is where dep you

know it really depend how your

organization is structured today that is

where you should have your architects,

your engineer leads, infra leads etc and

cyber leads actually sitting down owning

part of those domains and making sure

that the skills we need to get updated

is actually according to the policies

you want to adhere within your

organization and drive this change.

And finally when you get this right what

you will have you would have at the

organization level all of your teams

pulling from one centralized place high

quality skills executing them and

pulling them back um to to the

centralized platform if it is improved.

Now what I want to bring this because

it's a little bit of a inclear view. So

what I created I created a simulation

right. So think about this. This is your

organization today, right? And what I

have here, I have um just a random

teams. I have 15 teams created um five

to 12 like per team. Uh you have skills

per engineers contribution. You have the

average skills utilization kind of like

on average like how much time uh skills

are being pulled per day, the

duplication across the team kind of as a

ratio and the skills quality and

security ratio. Now if I run this across

six months what's really happening and

think about it this is already happening

within your organization is teams are

creating and using the skills right but

we don't have visibility and skills

again they're tightly coupled to your

productivity uplift if for example the

example that I shared earlier on the

regulation if we don't have a skill

about the regulation that is someone is

vip coding back and forth and trying to

figure out exactly how to steer the

agent to implement it properly right

that is burning more tokens from one

side costwise but also the productivity

is spending more time rather than giving

in one shot the right answer and the

quality and security is similar if you

don't have clear you know skills defined

and maintained you will have a low

quality in your implementation because

then it's up to the human to decide this

and different the m in the maturity from

team to team you can see the difference

and that is why for example if I look uh

just randomly at this this is like you

can see the productivity of this team is

a kind of a medium right If I look at

this one is a is a bit of like I don't

know it's low medium productivity

quality and security also medium but

when it came to the cost is really high.

Okay, now let's actually say okay how

this looks like if I governed all of the

my skills in my in my organizations.

What's going to happen is of course some

of them will split right and this is

reality isn't going to be perfect um as

we expect but at least what you will see

you will see actually some common ground

across all of your teams. The moment you

govern, you publish one skill, the next

engineer trying to build a new skill,

the coding agent harness will identify

this skill that is already available and

pull it. Right? So you you almost solve

all of the issues that that I covered

about the governance.

And one last point is when it came to

skills, um it's just one component of

your workflows as I said, right? So that

doesn't mean uh if you figure out skills

that says you're good. No, you need to

apply the same kind of like approach and

solution for your whole workflows. And

you may think to apply this again if you

think about it like if you have a a

centralized platform that have all of

your workflows, right? From one side

you're centralizing the workflows which

is also having the skills but also if

the next engineer came and want to I

don't know like provision infrastructure

they can tap into a workflow and build

that workflow with the required skills

and run it and test it again. And if

it's something you need to be improved

in the workflow, you can easily push it

back um to the centralized platform for

your organization to use.

[snorts] Now before I wrap up, um what I

want to leave you with is this is just

the start the beginning. You see like

this just we're talking about six to

eight months. What's coming next and I

would invite you to already explore is

skills registry, right? You should have

one if not already. And the good news is

all of the players that's been solving

the IDP problem like um internal

developer portal they already start

centralizing this capability right so if

you don't have it today maybe in a

couple of months you will see it coming

but also there's a lot of tools that's

actually solving this specific problem

second is skills eval evaluation there's

still kind of like a discussion on what

is the right approach to to to you know

to to evaluate skills the easy thing

that I found so far very valuable is

actually test like you you l static test

your or evaluate your skills against the

entropic best practices. Right? If the

skill is not invoked properly, if the

skill is not structured properly,

there's a high chance that it's not

going to be high quality. And finally is

auto evolving. And again, this is what

everyone kind of like is is the next

hype right now. Like yeah, I can create

like a an a closed loop that can evolve

automatically my skills. So what right?

If you automatically start this machine,

the impact will be way more than it is

today because what I shared earlier is

going to be just maintaining auto

evolving skills without that governance

in place that actually put the

guardrails for your organization.

And at this point I would leave you

here. Thank you so much for your

listening and looking forward if you

have any question I will be in the

leadership lounge. Feel free to grab me.

Thank you so much. [applause]

>> [music]