Skills are the New SDKs

Talk transcript

1. 0:03

   We'll talk about skills today. The current challenges about context, and then lastly, about the eco-ecosystem that's being built around skills lately, especially with OpenClou and all of those stuff getting a lot of attention in the industry.

2. 0:22

   So let's get into it. Every AI or agentic app typically has three layers. The one that user sees, which is typically the UI, the user interface. The one that model sees, which is the system prompt, as well as the tool descriptions.

3. 0:38

   And the one that data sees, which is the schema of data or the input and output of the tool calls. And these little bugs typically live on these second and third one, which is slightly hidden from the user.

4. 0:53

   And so let's talk about these problems. The first one is that beautiful lie that we've been told that the latest frontier models have infinite context windows. They typically promise that each one has one million, five million.

5. 1:09

   Even latest models, they're promised as infinite context window, which incorrectly shapes our thinking perspectives about RAG and MCP in a sense. Because if we consider that there's infinite context window, we just think that we can dump the whole list of documents to the context and expect it to magically work.

6. 1:32

   This applies to MCP as well. Let's say we can write hundred tools and expect the LLM to use or pick the right tool at the right time, which is typically not true.

7. 1:42

   Because in reality, the longer context typically doesn't mean better performance. Because each time put more context into LLM, that's one more place where the LLM might be misled or its context might be poisoned.

8. 1:58

   In fact, there was a paper called context rot, and it proves that after 25% usage of the context window, so for example, for one million token, if you've used two 56K of it, the performance starts to degrade.

9. 2:17

   So I'm pretty sure it starts really well as day in the life, but as time goes and as the conversation grows, let's say there is hundred prompts input and output responses in your one chat, and add it to hundred chats, and then each one of them kind of calls MCPs or tools each time out of them in

10. 2:38

    the middle fails. And at the end of the day, the context is almost eating itself in a sense that the latest fronted model can perform

11. 2:47

    like a really poor model. I'm pretty sure we all have seen this, "You are absolutely right," from Claude Code, even though feels assuring that you're absolutely right. At one point it makes you feel bad that Claude Code is making the same mistake it made five minutes ago.

12. 3:05

    The second problem is mostly about documentation websites. This is an interesting statistics that just from last year to this year, the traffic to documentation websites increased from 10% to 50%.

13. 3:21

    That's coming from coding agents. The challenge here is the fact that docs are written for humans with models and expected in a different format. Because docs typically require some kind of intuition, asking for a follow-up, or if you don't understand them well, you can just Google it quickly and then find the answer for it.

14. 3:41

    But models typically doesn't have those capabilities until you actually build a context engineering around it. I wanna add one more strategic point about the future of enterprise AI and why we think that skills might be a huge contributor to where the enterprise agents or enterprise AI is heading towards.

15. 4:01

    Let's talk a little bit about modes these days. Now, if you wanna understand modes today or tomorrow, we can go back in time to the history of software to see how has it been in the past.

16. 4:14

    Typically, the things that have been the hardest to replicate have been the modes for the software companies or even hardware ones. So for example, whoever controls the hardware, the data, or especially the integrations in the SaaS era, would be the ones that would control the market most of the time.

17. 4:32

    So the real mode here would be friction, where you would make it genuinely hard to you-- to leave-- to, to make someone leave your platform. So the core idea here is the switching cost was too much that it, it was just hard to switch.

18. 4:49

    Now, these days we are hearing news where Claude Code would rewrite hundred thousand or a million lines of code from Python to Rust. So basically we're getting really, really good at switching codes and reducing it.

19. 5:03

    Now we can have better tools, better APIs, just better languages, and then switching it in few days.

20. 5:11

    Now, skills are creating an entirely different ecosystem where instead of creating a friction mode, they're actually creating fluency mode. So the core idea is switching the delay a little bit so that each time your skill makes the experience better for whoever is the one that, that's using your platform.

21. 5:32

    And that fluency compounds by experience. The, the real result that the users of your platform is getting from here is the experience. If you would look at the definition of experience, we can realize that it's satisfaction or minimum amount of friction between intent and an outcome.

22. 5:52

    So basically whatever the user is trying to achieve through your platform, achieving it in maybe minimum amount of time or trying to optimize the whole satisfaction layer or whole experience layer of it.

23. 6:04

    If you look at the equation, you will see that overall friction modes and fluency modes are the opposite. Friction modes are typically defensive, so you're trying to make your experience in a sense that someone wouldn't switch.

24. 6:18

    Now, fluency modes are offensive in a sense that you're making the whole experience really nice, in a sense that you're making it genuinely hard for someone to switch anyways because they like the platform.

25. 6:33

    So we think that the feel of the thing or the reliability of the outcome from the start, which is your intent, to the result is the whole experience layer, and the skills are one of the best tools that can actually quote unquote commoditize that experience layers for agents.

26. 6:51

    So the real mode in SaaS era might be the core idea of skills, where you would commoditize the feeling of your platform, or in other words, how trustful or reliable you are in terms of intent and outcome equation.

27. 7:09

    When someone evaluates the platform, they typically will have a checklist of things that they take a look, right? They typically look at how secure your platform is. Are you compliant with all sort of new standards that's co- that's coming up every week?

28. 7:23

    How is my data governance layer? Where is my data stored? How are your SLA guarantees? Can I see the logs of my agent? How about tracing? H- how is the integration layer?

29. 7:34

    Are you able to integrate to my auth layer that's connected to my internal on-prem systems? So now we think that checklist really has a new item, a new guy in the room, which is teachability.

30. 7:46

    Again, coming back to the experience layer for agents. So here the core idea is if someone is new or if my agent harness is new to a platform, how easy is it for that agent harness to pick up that operational knowledge that you've encoded to the skill and apply it directly and then get into the result in

31. 8:06

    less than few seconds? So the core idea of teachability is to make that experience easy as possible whenever someone wants to use your platform for the first time or for subsequent times.

32. 8:18

    So if you wrap it up, we think that the skills might play a huge role in the enterprise AI or enterprise agent world in a sense that it might define what the mode is.

33. 8:29

    If the mode these days is about fluency, skills will help it in a sense that the value that you are giving will get compounded by each skill you add to your platform.

34. 8:42

    When I think about standards, I feel like I just need to skip this slide for this audience because standards are good, right? For example, React came about recently, but if you go back in time to twenty years ago, it, it almost feels like LLM and agent ecosystem right now.

35. 8:58

    So twenty years ago, we didn't have React. It introduces a philosophy on reactivity and modularity to the, to the way we build apps. And same for LLMs and agent ecosystem in a sense that we don't yet have that React moment.

36. 9:13

    It's still up on air in terms of how to systematically develop really long-running agent systems.

37. 9:21

    And as I said, context engineering in a sense is kind of a one attempt to, to solve that problem in an organized way. So if you would summarize what context engineering is, it's almost an idea of minimizing the amount of information that flows to the LLM at the right time, so that we can filter, so that we

38. 9:41

    can fill the context with the right information at the right flow, at the right moment.

39. 9:48

    So let's look at this example on the right. So if you consider a typical agent, whether it's Claude Code or an, or an agent that we built for a customer, it typically has system instructions, cloud.md, couple built-in tools, maybe also connection to MCP, right?

40. 10:09

    And on top of that, a user or a customer is kind of prompting the LLM. Typically, this forty percent is the smart zone where the LLM can potentially perform well.

41. 10:24

    But if you, you are getting past this forty percent context used, it typically goes into a [chuckles] zone called dumb zone. The core idea here is the fact that even though you have hundred percent context available for you, but as you go past forty percent, um, then you are actually getting really dumb responses.

42. 10:45

    So the core idea here is, is to make sure that you're not passing forty percent even without starting to work with your agent. So imagine you have all sort of different instructions, a lot of MCPs here, and then you add one more block here for another MCP.

43. 11:02

    One more block here for another MCP that's connected to internet. One more for web search and things like that. And then even you haven't ever chatted to your agent yet, now you've filled your forty percent context window.

44. 11:16

    And then when you just start to chat for the first time, you start from dumb zone from the first call.

45. 11:23

    So typically, creating a great context has pieces to it, right? Let's cover each one of them briefly. The first one is instructions that kind of comes by default. You put it to the system prompt.

46. 11:37

    The second one is external data. Maybe the user will upload some PDFs or any kind of RAG. And the execute history. Obviously, LLM needs to have access to what happened on that session.

47. 11:50

    What's the past tool call results and where are we in terms of state? And the latest agentic systems has memory in a sense that they can remember things across conversations.

48. 12:01

    And then the output schema, the, the correct way to output all of these stuff to the user. So as you see, each one kind of adds one more level of complexity to the things that agent needs to handle, and it almost creates n times n problem in a sense that one agent or one LLM needs to be

49. 12:20

    connected to all of these stuff at once, at the same time, at any time of the conversation. Which is pretty hard task to do actually, if we would emphasize LLMs.

50. 12:32

    The core idea is instead of hard coding everything into either one of these items basically, and then loading it to the LLM, we would progressively load whenever those instructions is needed to the LLM, and then LLM would have access to a piece of it at runtime, and then it can load it to its memory whenever it's needed.

51. 13:00

    The problem that skills to solve this, it enables a rich context access. In a traditional context, everything goes into the context which would let to context rot problem that we covered.

52. 13:16

    But with skills, those capabilities, whether it's about, uh, web search or whether it's about some capability that doesn't exist in the main agent, that lives externally, and then LLM or agent can have access to it.

53. 13:31

    So if it would give an example and compare with MCP, typically if an agent is connected to an MCP server, it... Even this number is really small for 15 MCP servers.

54. 13:44

    If it's connected to 15 MCP server, I'm pretty sure it's consuming over 100,000 tokens per session just in tool definitions itself, without starting the conversation yet. But same operational headache becomes really small, probably at least 10 times small with skills, thanks to progressive disclosure.

55. 14:06

    So this one is only expose metadata of skills to the agent or to the LLM at runtime so that it can act like an index in a database, just like how indexes work in a, in a DB so that you can quickly search for stuff.

56. 14:22

    Same for the metadata for skills. Even if skills might have thousands of tokens in it,

57. 14:29

    LLM has access to the metadata of it on which most of the time lives at the top of the file, which is being loaded to the, to the context on demand.

58. 14:44

    This creates mental model shift on developers as well as the industry in a sense, because the skills are, are kind of becoming almost like a framework across platforms like CloudCode, Codex, and then all of them supports this perspective.

59. 15:03

    Why you would ask long-running agents, even coding agents are acting like a general purpose agents, right? So when a customer asks us to build a supply chain agent or an agent that's in a manufacturing space, we don't really have to build a separate agent for that specific use case.

60. 15:23

    It's the fact that we can still take the main general purpose agent as the, as the engine, and then put skills on top of it. So you see how skills creates a shift in terms of perspective about how to build solutions in a sense that right now you don't have to actually build the agent itself, the engine

61. 15:43

    itself, but actually build skills for it, and then that's way to switch agents between industries, between different domains.

62. 15:56

    All right. Moving on to part two, where we'll cover briefly on skills versus MCP. So they kind of solve slightly different problems in a sense. So I just want to clarify this.

63. 16:07

    Both of them are from Anthropic, both MCP and skills, so they're not competing for a thing. So in a sense that MCP kind of definitions for what actions the agent should take in a sense that it doesn't change the agent definition, and then what the agent is, what capabilities it has.

64. 16:27

    And for skills, it's different in a sense that it helps to think about problem and then change. You can basically customize the way agent acts through skills, how this think about this problem.

65. 16:40

    And then skills introduces you a way to self-modify in a sense that agent can modify or update its own skills based on the experience that it had with that skill.

66. 16:54

    On the other hand, for MCP, it's sort of a server that sits in a separate environment where the agent has access to use that tool, but it doesn't have access to the code of that MCP tool in a sense.

67. 17:08

    Anything to add here, Carson and others?

68. 17:12

    I, I think this is a good opportunity to answer to Joji's question. A thing a skill can do is expose an MCP server in its skills MD that says, "Here's your MCP server."

69. 17:22

    It could also just actually, you can dump the tools that were in your MCP server into your skill directly, and the agent can then execute them. So it can actually service and create its own MCP server as part of a skill.

70. 17:34

    This is why we get so excited about the robustness of skills versus MCP. It's not just self-modification, which you could replicate with MCP, right? MCP with memory and caching and user customization and using tool calls that are actually agents themselves.

71. 17:48

    Those agents may even have their own skills that you could specify as part of the tool as load these skills and execute this thing, right? Like there's this kind of incestuous relationship, but still skills live on their own as a distinct thing that can operate, templatize code, execute code, execute MCP servers, build and create and run, and

72. 18:06

    then therefore add to their root context window new MCPs, just all via the skill. So it's like an entire complete package Of what an agent can become. Essentially, Copilot could be a skill [laughs]

73. 18:20

    So the core idea is use skills when the hard part, hard part is just about reasoning or thinking in the right ways. But MCP solves other problem for us in terms of authentication access or the horsepower.

74. 18:36

    If you want to actually isolate the process for, for some of the execution that really requires a lot of resources, then that's easy way to let that have its own server

75. 18:47

    That can interoperate, right? Skills run with your, the agent's machine. So there's a little bit of a loop-de-doop, but they're still sufficiently different that MCP, you know, gets to live in its own world and be a production piece of code, and skills get to be kind of viable.

76. 19:04

    I think MCP is, until it adapts, which it is actively working to adapt, right? It's the older standard, and the skills, we didn't know quite how powerful we're, we were until we had 85,000 of them out there.

77. 19:15

    You still don't... You want that hosting capability, right? I don't have access to a GPU on my computer that is sufficient in order to do significant operations or to do semantic search across 400 terabytes of documents.

78. 19:29

    In those cases, I still need a robust hosted MCP server. The way I get my agent to use that MCP server is you can use skills to do progressive disclosure for that MCP server that then has that 500 terabyte disk or the access to 75 GPUs or whatever it is that that tool needs.

79. 19:46

    So they're still interconnected in a way that they both have their uses and are kind of inescapable. Is that MCP server going to access data in a restricted environment, in your medical rec-records or, or whatever that your personal computer can't access?

80. 20:00

    Does it need resources that are unachievable on the agent that you're executing on? It, it's essentially a way to execute remotely and, and that still always has a benefit.

81. 20:09

    And one last argument against MCP, which might turn out to be interesting, is the fact that it turns out the current agents doesn't need a lot of tools. If you go into the way Codex or Cloud Code is developed, you will see that there's only, like, a handful of tools that's being run.

82. 20:26

    It's not like there's hundreds of tools that's being executed in a, in a separate MCP server, which also proves the fact that a, a typical skills folders or capabilities folders with different options can replace MCP as long as you have a good base reasoning model.

83. 20:46

    All right.

84. 20:46

    Skills has two different readers. The first one is the front matter, the top of the skill, which is here. If you see an example of our, one of our skills, you see that this is almost part of that MD file, but this acts like an index for that skill.

85. 21:04

    So at runtime, this is the only thing that's loaded to the agent's context, agent system prompt. But this one is loaded to the agent's context, which is the body of it.

86. 21:15

    So when I say there's two different readers, the first reader is the agent's runtime, which is the system prompt. So the reader for this one isn't the context itself.

87. 21:26

    It doesn't really burn a lot of context because it's just a small XML that has probably less than 100 tokens typically, which agent do have access to prior to the execution.

88. 21:37

    And the second one is the markdown body, which is read by LLM if activated. If the LLM thinks that we need that skill, then it will be loaded. So this is the first one, less than 100 tokens.

89. 21:50

    The second one, on activation, which is typically less than 5K tokens. The level three is what third one is, which is scripts. The scripts can be in two ways.

90. 22:02

    For Cloud Code, for example, they're executable in a sense that the Cloud SDK or Cloud Code can actually execute this get_deployment_features.py, and then only put the output of that execution back to the context.

91. 22:16

    But a lot of other examples, a lot of other platforms supports this one, which is also code, but not executable in a sense. Just serves as a purpose of example for the agent.

92. 22:30

    So three levels of managing a simple skill. First one is the front matter, second one is the body, and third one is extra scripts or context. Part four, talking a little bit about ecosystem and what's going on around there.

93. 22:43

    So there is 26 plus platforms that support this right now, like Cloud Code, Codex, Copilots, Gemini CLI, and others. There is probably at this point 100,000 of skills out there, and then there is marketplaces where people sell skills for money.

94. 23:00

    Devin Jensen talked about Claude Hub, which is skills registry for them. That being said, we are also part of ecosystem, and we also published our open source agent skills repository.

95. 23:11

    This is a really good demonstration of both the power and the risks of agent skills. OpenClaude right now has become the repository that has the most stars, beating Linux and React, and I feel like there is a lot of marvelous ways they've done the context engineering in a sense that works across different apps and chat applications.

96. 23:33

    But I feel like probably a couple details regarding their sub-agents, which kind of serves the purpose of keeping the context clean, as well as self-evolving skills, made it really impressive for whoever looked at the code.

97. 23:48

    The, the fact that it can kind of self-heal based on the experience of it with skills as well as write its own skill to extend its capabilities, in a sense, scary.

98. 23:59

    But what made a good impression, the fact that this is just a coding agent that has access to about 10 tools, which can do anything for you, as long as you have skills, uh, on top of it.

99. 24:12

    So that kind of changes the way we think about customers or- enterprise solutions in a sense that we just need to develop one good agent, which is base for everything, and then you can just build skills on top, which solves your domain-specific problems or whatever your customers are asking for.

100. 24:34

     So if Open Cloud can write skills, then let it write it, right? But this touches the fact that typically we need humans to write skills. There is a lot of research on this that's recently published, but they published that actually the LLM-generated skills hurts the performance of LLM, in a sense that it uses more tokens.

101. 24:55

     It's, it spends more time to go on and actually reason out about the problem rather than helping it to be faster or to use less context, in a sense.

102. 25:05

     There's a lot of risks to consider here. If, if the LLM is writing and executing its own skills, then

103. 25:12

     obviously there's a ton of risks in terms of prompt injection because everyone is writing skills these days and then publishing it out there. Pretty sure you have seen on the news there is bad stuff that, that happened with Open Cloud.

104. 25:24

     And there is no isolation. One of the advantages of MCP in the sense that you isolate the process for your MCP and your agent, but everything happens on your laptop, on your agent, on your environment.

105. 25:37

     And last one here is about the fact that these marketplaces still lack the verification control. Just like how NPM was scary 10 years ago, in a sense that you would still check the number of stars or who developed this before you downloaded the package, same for skills.

106. 25:56

     Closing thoughts. Context is a budget. Context is almost like a limited resource that we need to carefully filter information. Definitely longer context doesn't mean better. Every irrelevant file, every irrelevant web search or error message really drags agent's attention, and then you lose a lot of reasoning power.

107. 26:14

     Skills are complementing MCPs, at least right now. A skill is only as good as the human who wrote it based on our experience. And lastly, skills are software, which can take weeks to build, so that we should actually start versioning them, evaluating and testing them, and actually writing good skills.

108. 26:33

     There's a couple resources that we've used. Thank you.