From 36% to 100%: How Self-Improving Agents Write Their Own Skills — Rafal Wilinski, Runlayer

Hi, everyone. I'm Rafal. Today, I wanna talk about self-improving agents, but not in a single player mode where just one agent gets smarter. I wanna talk about how we can apply that concept and mix it with network effects to get a self-improving company or maybe even, uh, an enterprise that self-improves while you sleep.

Um, imagine the best engineer you ever worked with, or a salesperson, or a lawyer. Um, the one that every time they cracked a genuinely hard problem, they wrote down exactly how they did that and handed over the playbook to the team. So whenever they face a similar problem, they don't have to figure it out from the scratch, right? And imagine that happening for every hard problem across your whole company, and that happening automatically, uh, each time. That's a dream, right?

But it almost never works that way. Um, even when you crack a really hard problem, um, oftentimes the knowledge, um, all that debugging, um, all the, all the reasoning behind that is just gone. Oh, sorry. Um, all that, uh, all that reasoning is just gone when you move on to the next, the next task or close the session. But I wanted to convince you that this is actually possible, and we already have all the primitives in place to make it happen.

If it sounds interesting, let me intro- introduce myself once again. Hi, I'm Rafal. I'm a founding engineer at Runlayer. We are making the golden path for AI at your company. Um, before that, I was leading Zapier agents and different AI teams, but generally speaking, my previous work was about making agents not just flashy demos, but reliable things in production.

Um, before we go into interesting parts, I wanna break down my topic into three separate pieces and start by just improving agents. There are many ways to do that. We can use better models. We can use better prompting tools, harnesses and whatnot. But I wanna focus today only just on skills because they are super accessible, right? At least in theory. Um, to make sure that we are on the same page, a skill to me is a playbook that essentially agent can read, um, when it thinks that the knowledge is

relevant. From the technical perspective, the, the term is called progressive disclosure, and it means that at the very beginning, the harness or a client sees just a summary of the skill or a short description of the skill. But when it thinks that the knowledge inside the skill can be useful, it can read the whole of the skill, inject it to the main context, and based on that knowledge, it can change the trajectory of the agent. Why is this so important? Well, if you haven't been living under a rock for the past

six or twelve months, your workload probably has changed a lot, uh, and that's because the agents got so much better. Actually, the models got so much better. And, and it's not just vibes. It's measurable. Um, here you can see a chart that probably you have seen too many times already. For those who don't know, it's METER benchmark. It measures for how long agents can work independently while still having a decent chance of completing a task. And the word chance is essential here. Um, we are

going to go back to it in a moment. But the trend is clear. The frontier capabilities, the frontier intelligence is getting better and better. They are better at multi-step deep work. There's only one caveat if that frontier intelligence is not being taken from you, and I'm as fabled probably just like you.

But there is a catch. With great power or long task horizon comes great responsibility. If an agent can reason for much longer, it also means that it can dig a really, really, uh, deep rabbit hole. Um, it can spend hundreds of tool calls trying to defend a wrong thesis, reason for hours. The next agent can do exactly the same thing or maybe even do something wrong, um, worse. Um, so the initial trajectory matters a lot, and that's why skills are so

important because they guide the agent towards the right target. Otherwise, you'll waste time, you'll waste tokens, and most importantly, you'll also waste money.

Um, valuable skills are the rituals that nobody wants rediscovered. That knowledge is often not glamorous. It's probably deeply embedded within your company. And if agents would try to rediscover that knowledge from scratch, like for instance, the sequence of scripts and feature flags in order to make a nice rollback, or what's the exact wording and language to use when dealing with that pesky es- uh, enterprise customer that's about to churn. Um, yeah, if agents would try to figure out that from

scratch, they are likely going to fail, and oftentimes that failure can be really costly. That's why I believe so many AIPs or POCs are failing because you're not giving agents the relevant context and the playbooks how to act within your company. So skills are great, but I feel like they are not problem-free. Um, there are three most important problems, and the first one is that they are local first and dev-centric. What I mean by that is that, um, when you

look at the skills ecosystem, you're going to find, um, Markdown files, JSON files, CLI commands, Git repositories, and if you're a developer, that's great. But if you're, um, ado- uh, VP of AI or AI adoption lead, you need to think more empathetically about how you can roll out the AI strategy across your whole company, which includes non-technical people, right? And the challenge here is how non-technical people can actually create and share, uh, the

skills. Uh, I love this meme. Um, this is the junior developer or a non-technical person discovering command ex- npx skills, which is the most important, but the most popular command to install them. That's the only command I know that installs just, uh, not just the skill, but also free prompt injections, um, a wild script that requires root permissions, and someone else's API key all in one run. Trust me, at Runlayer, we are scanning thousands of skills every day, and

there is really weird stuff inside. Okay. Second problem is that I think we all like-- we all love the situation where we are facing a hard problem, and there's already playbook on how to deal with that situation, right? But oftentimes there's, there's no playbook, right? Because it time- it takes time, and effort, and intention, and energy to reflect on a problem and leave something useful for posterity behind. And the third problem is that not every

LLM client supports skills, right? If you're in cloud ecosystem, it's probably great. If you're a developer, you're probably going to manage, even if the skills support is weird, depending on which client you're using. Um, and once again, if you think about marketing, HR, legal, finance, those departments are probably still using ChatGPT or some other weird LLM client, and skills might be not supported there.

And some time ago, there was this big heated debate when the skills were introduced. Um, some people proclaimed that, "Thank God MCP is finally dead because I'm not interested in dealing with OAuth. I can now hard code a bunch of CLI commands, um, hard code my API key inside Markdown file and call it a progress," right? Um, but I think everyone missed the point. Skills can be delivering, um, the knowledge how to do something, and tools gives us capabilities. Tools from MCPs give us capability to do those things. But

MCP can be also used as a protocol to deliver the knowledge, to deliver the skills. We can use MCP as the distribution layer. So instead of installing skills locally, you can create a remote MCP server, which can become the source of truth across your whole company. Clients can connect to that. They can ser- they can search for the right skills. Uh, server can enforce the policy and make sure that everyone gets the best version of the skill possible, depending on the query and, for instance, the role that they

have assigned. We've been doing that internally at Runlayer for the past six months, and it's been working quite well. Um, but it's no longer just an internal thing. We see that the broader MCP ecosystem is also going that direction. Um, there's already a working group called Skills over MCP, and while the exact API is not final, direction is clear. MCP can be used to deliver the skills, to deliver the knowledge across your whole company.

Um, yeah, and while the official spec is leaning towards MCP resources, the support for resources is also mixed, so we are going with tools. Um, and yeah, if you remember the problem, the first problem I mentioned and the third problem I mentioned, which is, um, the problem that they are dev-centric and the lack of, uh, support for clients, MCP as a distribution layer fixes that because it's unified and on-- and works in almost every single client, not just developer

clients. So the problem of devel- of distributing skills to non-technical people is essentially solved, right? Plus, you have now a centralized API, so as a VP of AI or, like, AI adoption lead, you have one place to govern and manage what's the knowledge that's getting shared and distributed across your company.

Okay, so now we know how we can improve agents because we have, um, skills. We know that we can use MCP to distribute those skills across all the clients and agents. But let's talk about something more interesting, which is how agents can improve themselves. And whenever I, uh, see a new paper or a framework speaking about this topic, I always go back to, um, Voyager paper. Unfortunately, you cannot see the video, but it's Minecraft gameplay in, in background. So in, in twenty

twenty-three, right after GPT-4 was released, so ages ago, the question was: For how long multiple agents or harnesses can explore Minecraft space, right? Um, they dropped the agent into Minecraft with no instructions. They set its own goals. Agent just kept getting better without even touching the weights or, or model and the prompts. And in twenty twenty-three, I think that was pretty groundbreaking. How did they pull it off? Um, the key thing to

notice here is the skill library. So Voyager did not just explore the Minecraft space care- carefully and throw the trace away. When the agent figured out something, um, it saved the recipe as a code that, uh, that consisted of the tool calls that lead it-- that lead the agent to the discovery. So later on, when the agent decided that, hey, I need to create a, a crafting table or a diamond pickaxe, it didn't have to figure it out this whole knowledge from scratch. It could

simply go to the skill, uh, library and retrieve it as a code, as a recipe, um, that can be reused. And this is really important. This is the core idea that I wanna carry forward, is that, um, exploration should be turned into reusable procedural knowledge.

And we started doing something very similar at Runlayer. Um, we let the agents do the real work, and when the run succeeds and when there is something interesting in this run, there's multiple tool calls, um, we just take the trace, and we ask Frontier LLM to simply distill a skill, right? And while it's super simple to distill a skill from a single run, um, I think we humans and agents learn the most from mistakes, right? The real

sharpening happens when the agent fails. Um- So a run that would go sideways is actually the richest signal that you can get because it exposes the missing guardrail, the missing library, some kind of edge case that you haven't thought about upfront. Um, and every run or, or pass or fail just feeds our distiller mechanism, constantly making our skill better. Um, the successes are making the skill more trustworthy because if the agent achieved the goal, it

means that, hey, this skill works, and if it doesn't, it actually makes skill the more robust because now we know what's missing in here, and it all happens completely autonomously, asynchronously without human in loop. And the initial results were great. Uh, we had, uh, this task of trying to run some Chromium fork inside AWS Lambda, and we were trying to do that using Sonnet four point six, so not the most intelligent model, but sometimes it succeeded. Um, but

it was just, you know, it's not production ready to have thirty-six percent success rate. Um, so what we did is we flipped the flag for self-improving agents, um, and once in one run, the... one of the agent invocations cracked the code, our skill distiller create a reasonable skill. It was, uh, then included in all the future runs, and the success ratio went to one hundred percent. So yeah, that's great. It's like a developer with better attitude towards documentation. Um,

the success rate is the easiest number to show, but good skill is narrowing the search space. It makes the agent... Uh, it narrows the search space, and agents starts doing, um, stops doing tool calls, and it's exploring less branches. It's wa- it's wasting less tokens and, and less money.

But putting savings and accuracy aside, I think there's also a deeper reason why you should be distilling skills. Um, the problem is that frontier intelligence is rented or borrowed. It can disappear. I think, yeah, we all miss Fable. It was a great model, right? It can deprecate it, or it can get dumber. Like, we complain every single day on Twitter, on Slack that Opus is really dumb today, right? So while we have the access to the best version of that intelligence, we should be using that

to discover and persist those use- useful procedures. We should be building a catalog of how the works gets done inside our company as a collective subconscious.

And how we do it? Well, I believe we already have all the pieces of the puzzle here. Um, we have skills as a way to retain procedures and improve agents and guide models. We have MCP as a unified distribution layer for those skills, and you also have a central repository, the knowledge base for all of those skills, which is constantly being updated and fed with new data. Um, yeah, we also have the continuous learning flywheel for one

agent, but what's interesting is what I promised you at the very beginning, the network effect. Um, so what happens when you multiply that by every agent and every single client in the entire company? At Runlayer, we call it self-improving organizational flywheel. Um, we group a bunch of similar runs and try to distill a skill. It passes through our various series of checks, like whether it's not containing some kind of prompt injection or maybe PII, um, and bunch of other

stuff. Once it's good, it's landing in our skill library, and thanks to our MCP gateway, it's being served to each and every client and agent within the company. What we end up having is a constantly evolving knowledge base, which is always up to date, continuously updated with new alpha, fed with new edge, uh, edge cases. So once one agent figure out a really hard problem, it's getting dist- it, it's being distilled into a skill persistent in all organ- organizational library, library.

So whenever someone else, um, has the same problem in the future, they don't have to figure out the solution from scratch. They don't have to know, uh, what's the exact sequence of this magical commands that are going to restart the database or roll back the production. They can simply, um, reach out to the playbook they have. And why is this so important? Because I believe, like, fifty percent of us probably already had an existential crisis thanks to the models, and

the thinking goes along the lines of, uh, okay, so models are becoming commodity. Everyone here has access to them. Um, cost of software is going to zero. Everyone is... can be rebuilding my product, um, at least on the surface. In fact, so many people are working today across the expo, and they are thinking, "Hey, everyone's building the same thing, right? So what is my moat?" Um, and I believe the moat can be in building this flywheel

and distilling the knowledge to this flywheel because it's like your own hard-won lessons distilled, governed of how your company, uh, does things, you know. And it's captured, compounding, raising the floor for everyone across the company, giving them access to the best knowledge available. Um, it is also a way to be resistant against model intelligence deprecation, and it's a great way to lower your costs without ever changing a

model. So yeah, that's probably one thing that competitor can't copy, your internal culture and your identity.

And yeah, that's pretty much it. Uh, thank you for coming. If it resonated with you, uh, I'm Rafal. I'm working at Runlayer. We are figuring out the AI golden path for you. I'm open to talk about skills, MCP agents, whatnot, and this is exactly what we are building at Runlayer. Thank you.