Skills at Scale

Talk transcript

1. 0:00

   [upbeat music] This workshop is gonna be Skills at Scale.

2. 0:16

   We're super excited to be here with you today. There's gonna be an interactive component, and we also want you to feel free to interrupt us and ask questions as we go.

3. 0:23

   Yep. We'll show that slide again with the s- with the, um, uh, QR code and the, the instructions to clone the repo. That repo has the skill, uh, the skills that we're working on plus, uh, the slides that we're presenting, so you'll have all of that as reference material.

4. 0:37

   Um, I am Nick Nisi. I am a developer experience engineer at WorkOS.

5. 0:42

   I'm Zack Proser. I'm, uh, also a developer experience engineer at WorkOS, and we're on the Applied AI team.

6. 0:47

   And this is, like, working with agents, that's just, like, what we do now. Uh, Zack, when is the last time you wrote a line of code by yourself?

7. 0:54

   Uh, I think I did a CD, uh, in a directory- [laughs] ... uh, recently. [laughs] Otherwise, it's been, like, probably six or eight months now.

8. 1:02

   Yeah.

9. 1:02

   Maybe longer. Yeah.

10. 1:03

    Same. Same. We've been early on from the, the Opus 3.5 days, uh-

11. 1:08

    From copying and pasting back and forth through GUIs-

12. 1:10

    3.7

13. 1:10

    ... to now. Yeah.

14. 1:12

    Yeah. Uh, it's gotten a lot better, and I don't know, there's a, a mythos out there that it's gonna get even better. [laughs]

15. 1:19

    Uh, we are at WorkOS, and, uh, if you are, uh, interested in, like, securing MCP, for example, or just setting up auth, uh, for this new agentic startup that you're, you're working on, uh, reach out to us.

16. 1:31

    There's a number of us here with these shirts on, and, uh, we'd be happy to talk. We're also hiring.

17. 1:35

    Yep.

18. 1:35

    So, yeah.

19. 1:35

    Thanks to the AI installer that Nick built as well, you no longer even have to configure install AuthKit yourself. It can just do it for you, so pretty easy to get started.

20. 1:43

    For sure. All right. So as you know, when you're working with these, uh, with these systems, every single conversation that you have starts completely from zero. Uh, you're always just, like, passing in new information to it.

21. 1:57

    You've got to reiterate how you do things, uh, and Claude never... Claude, for example, never remembers that it ever talked to you. It just continues on a conversation, and so we have to provide that information fresh each time.

22. 2:12

    Yep. Um, so for example, let's say you have a skill or let's say that you're talking to, uh, just in disparate terminal tabs. You're looking at different code bases over the course of a week.

23. 2:22

    Every single time you start talking to it, you need to reload all of that context first and say, "This is what I care about. This is how we do things here.

24. 2:28

    This is what we're particularly concerned with," right? And it ends up eating a ton of time and slowing you down.

25. 2:32

    Yep. And of course, there's things like AGENTS.md or CLAUDE.md, uh, that you can put in information about the repo or about how you like to work, like in a, a global directory, uh, so that it can remember that and understand it each time.

26. 2:44

    You're giving that instruction each time, kind of like appending it, uh, so that it will remember, oh, in this project, we actually use Vitest and we use pnpm, so you should use those each time.

27. 2:54

    Uh, so it can... That, that's like the way to give it some memory for it to understand how to go.

28. 3:01

    But it can still get it wrong. It can still, uh, decide not to follow things that you have. I've definitely had cases where I'm like, "Do this, this, and then this," and it skipped the step in the middle, and I say, "Why'd you skip that?"

29. 3:11

    And it was like, "Oh, yeah, you told me to do it. I, I didn't feel like it." And, uh, that's, that's how you know it's a real engineer. [laughs] [laughs]

30. 3:20

    Yeah. So one of the nice things about Skills is that you can think of it as like a, a discrete unit of work where you can encode everything that's super important to you, everything you don't want it to miss, everything you don't wanna repeat yourself.

31. 3:29

    It's almost like carrying, if you will, the dry pattern into the agentic era in a way, um, and not re- repeating yourself. So and as we'll see, that becomes incredibly powerful regardless of if you're a solo developer working on your own startup with 12 agents or if you're on a traditional dev team with 12 team members.

32. 3:45

    Yeah. It doesn't know what you know, so you have to be very specific and be thorough with what you want it to know because it's not always going to figure it out.

33. 3:52

    Sometimes it feels like magic 'cause it just does, but a lot of times you have to put in the work to do that, and that's what, what, like, those memory files are, like CLAUDE.md, uh, and other memory files.

34. 4:01

    It even has... Like, Claude, for example, has its own built-in memory where it kind of keeps track of things that it thinks are pertinent to the, the way you work or the project that you're working on, and it will save that off.

35. 4:12

    Yep. And, and you know, of course, this works too in just a single, uh, project context with some of those files that Nick mentioned, but then again, the problem is that you're still tied to that repo.

36. 4:20

    Uh, you need your team members to remember to pull updates to that specific project skill if, if they want the context. Um, there's no necessarily built-in script execution, so how do you get...

37. 4:29

    How do you interleave a deterministic result when you're having a non-deterministic conversation with an LLM? And e- eventually, that starts to get pretty gross.

38. 4:38

    Yep.

39. 4:38

    So...

40. 4:39

    And the... There's downsides to this, uh, specifically these memory files where if... Th- they're tied to the repos that you're working on, or you have to put them globally so it affects everything.

41. 4:48

    Uh, and you can't do things like, uh, give it, like, more smarts, like execute this script. You can, but it's kind of not built in, so to say. Um, but that's kind of like where...

42. 5:00

    Uh, uh, th- the things that you put in there are not always, like, transferable or portable to other projects that you might be working on, uh, and so we need a better way to do that,

43. 5:10

    and Skills are that next step.

44. 5:11

    Yep. Indeed. [laughs]

45. 5:13

    Uh, it's a way to make things more portable, uh, and you can use scripts to inject real data, uh, and you can make them composable, uh, so that they can be very small and very focused on exactly what you want them to do.

46. 5:24

    Uh, and that way they're, they're very small, like a very small footprint in your, your context window, but also, uh, you can build them in such a way that they are only going to be applicable when you actually want to do whatever that skill is set up to do, so you're not just bloating the context with everything

47. 5:39

    from the start every time like you are with a CLAUDE.md.

48. 5:42

    Yep. And so if you haven't seen or heard of a skill before, just to level set really quick, it, it could be as simple as a, a single static markdown file, uh, without any scripts at all.

49. 5:52

    Um, but let's look at the difference at what might happen. Let's imagine that we're roasting a repo. We're onboarding a new team member. We wanna make sure that, you know, they're kind of up to snuff on how things work here.

50. 6:00

    Without a skill, if you're just talking to any generic agent with no specific context injected, you're gonna get, "Okay, looks pretty good." It's gonna be generic advice. It might find some low-hanging fruit.

51. 6:09

    If you say instead as little as 30 lines of Markdown specific to your use case and your conventions, your constraints, uh, you can start to get back very, very, like, hyper-specific feedback about this is how we handle, you know, uh, routing in this project.

52. 6:23

    These are, um... You know, we, we follow semantic and commits or whatever, and we've got README drift here, and that's unacceptable, right? So it, it can take as little as 30 lines of Markdown or less, and so that's s- the ...

53. 6:35

    one of the first things that makes skills incredibly powerful is that it's a very minimal investment on your part, and it could be as simple as a small Markdown file, and it becomes a composable unit of work that you can share across code bases and your team.

54. 6:48

    Yep. And you're, you're codifying exactly what you want it to do, and you have freedom to express yourself in the exact way that you want it to do that.

55. 6:55

    And there's a number of different ways and techniques that we'll talk about throughout this workshop, uh, but it's much better having those skills and knowing that they exist and, and knowing not just, like, that you can use them, but also setting them up in a way that the LLM can decide to use them when it makes sense.

56. 7:11

    Uh, and it's going to give you a single repeatable way of doing that thing in the way that you expect it. If I just tell it some generic thing like, "Ah, look at this and tell me how, how good of a repo it is," depending on the re- the model that you're running, uh, maybe the, the num-

57. 7:25

    the amount of thinking that you have turned on, uh, et cetera, it might give you more or less information, but it's never gonna be the same thing each time.

58. 7:32

    If you want it in a very specific format, you want this report in this exact way, that's what a skill is. You're teaching the, the LLM how to do something in the way that you expect it to be done, and then it will follow that much more closely.

59. 7:46

    All right. Let's take a look at, uh, how a typical skill might break down. So again, it could be a skill.md, so simple as a Markdown file. At the top, you'll notice i- a front matter, so think about, um, anywhere else that you've used, like, a YAML-based system, almost like headers, right, in, in other languages or formats.

60. 8:02

    But you've got a name and description, and this description is, is incredibly powerful and loaded. This is what the LLM is gonna use at runtime to essentially do routing and determine if this skill is relevant to the task that you've assigned it.

61. 8:15

    Um, so that's kind of how the AI finds and routes to your skill, and then additionally, you can provide, uh, additional context and, and then even scripts so that y- Again, think of it as your option or your on-ramp for interleaving determinism with the non-deterministic LLM conversation.

62. 8:32

    Yep. Uh, so the, the most important things here are exactly what Zack said, the, the name and the description. Uh, it's a misnomer that skills are only a, a single Markdown file.

63. 8:41

    They, of course, can be, but they're more like a folder with a skill.md file in them, and then they can have anything else in there as well, and we'll kinda talk about that.

64. 8:50

    But they can have references to other, other, uh, things that, that they might wanna know. They can have scripts that it should run, uh, and they can have images.

65. 8:58

    They can have all sorts of different things, uh, and then use that in different ways. But the most important piece of it, uh, from the start is the description, which we'll talk about.

66. 9:08

    Let's, uh, also just, uh, talking about constraints, one of the things that's kind of, um, an- not intuitive is that it can be more powerful just to, to provide a few constraints a- as opposed to overly, uh, being overly prescriptive in exactly how you want the task done.

67. 9:22

    So if you pr- provide just three constraints and say never be vague or, um, when you cite code it always has to have a specific line and a Git commit reference with it, um, then you'll get better performance than if you end up, you know, bloviating in the middle of a Markdown file so it's like a novel.

68. 9:37

    Um, this is actually a common failure mode when designing skills.

69. 9:39

    Yeah.

70. 9:40

    Yep.

71. 9:43

    So today, we're going to, uh, put all of this together into a skill that we're gonna build here in the, the workshop, and it's just called Repo Roast. We tried to think of, like, a, a fun, generic skill that would be appa- applicable to anyone who is working in, you know, JavaScript or, uh, different languages, uh, but

72. 10:00

    also i- like, like, if you, if you're not really. Like, i- if you have an idea of what a Git repo is, this is applicable to you. So it's kind of transcends all of that, uh, and is general- generally, uh, something that's useful for, for everyone, but also kind of fun.

73. 10:14

    We can kind of be more or less serious with it, uh, as we're, we're putting that in, so it's going to allow for a lot of creativity as we go.

74. 10:21

    Yep. And feel free to also, uh, kinda use this as a, as a place to inject the actual constraints or the requirements that you have at work that you're kind of, uh, struggling with or testing.

75. 10:30

    Um, so we'll kinda get the baseline together, and then you can start customizing from there, and we'll have some time to share and discuss them later too.

76. 10:36

    Yep. So, uh, this is that slide from the beginning. Um, if you haven't yet, uh, please download this, uh, clone this repo, uh, and work in there. We've got kind of the basics of the skill.

77. 10:46

    Um, and what we're gonna do is just kinda get it set up, and you can make it your own. We've got some general guidelines and some tips to do, uh, but the fun is going to be that we have a room full of people, and we can have a room full of different ways of, uh, analyzing this.

78. 11:00

    And we'll also share that. Uh, in that repo, there is a share.sh, uh, that you can run, and it will just ask you for your name, and then it will, uh, put that into a, a KB store.

79. 11:10

    And then I can pull it down, uh, quickly on my machine and then run it against some repos, like, on the screen, so we can share these, uh, at the end of the day.

80. 11:16

    Yeah. It'll be kind of a fun way to experiment with different, uh, approaches to the same skill.

81. 11:20

    Yep. [clears throat] So speaking of loading skills, we should talk about how skills load. Um, we, we are generally kind of talking ... Like, you'll hear Zack and I kind of always just, like, when we're talking, we're saying Claude, 'cause we tend to use Claude.

82. 11:35

    How many here use Claude as kind of their daily-

83. 11:37

    Daily driver? Whoa, okay

84. 11:38

    ... that's, uh, pretty much-

85. 11:39

    More than I expected

86. 11:40

    ... it's like 91%, uh, market, and then everyone else, like, there's Cursor, and I have been dabbling with Pi. Pi is amazing. Uh, but also Anthropic [laughs] won't let me use my ... [laughs]

87. 11:50

    I, it's unclear. Can I use my subscription with it? I don't know. Uh, maybe I'll find out today or this week. [laughs]

88. 11:56

    You can pay more for it. You can-

89. 11:57

    Yeah

90. 11:58

    ... pay more in credits for it. That's fine.

91. 11:59

    For sure. [laughs]Um, but when you're using these, so y- like, the, the main thing, and the reason that we're so excited about skills too is that they're generally applicable to all of the major models.

92. 12:09

    So Codex supports them, Claude supports them, Cursor supports them, uh, the, uh, desktop apps-

93. 12:14

    Yep

94. 12:15

    ... like, like, uh, Claude Desktop supports it, so even, like, if you're non-technical, you can be working on skills and sharing skills and, and using skills.

95. 12:22

    What, what was the skill that you did last week with the recruiting team in desktop?

96. 12:26

    Yeah. Uh, I was working with our recruiting team, kind of, uh, they're at an on-site, and I was zoomed in with them helping them build a skill that could, like, take, like, candidate information and format it in specific ways and understand, um, you know, what they're looking for in different things and kind of build reports automatically.

97. 12:42

    Uh, so it's things that they could do pretty simply, but they, um, because of, like, the, the beauty of Claude Desktop and all of the connectors that it has, like, it could just reach into Slack and pull in information from there.

98. 12:53

    It could reach into Notion and grab that information, and then mix that in with, like, the recruiting software that they use, and put it all together into a, a single report that then they can share to build from there.

99. 13:04

    So it wasn't like, this is the final report that we use for everything. This is a building block that then they can use to do different things in different places.

100. 13:11

     Yeah.

101. 13:11

     And so it was really powerful for that.

102. 13:13

     And as soon as you gave them that skill, then everyone on the team is running it in a uniform way.

103. 13:16

     Yes.

104. 13:16

     Which is the, the power of it, too. Yeah.

105. 13:17

     For sure. And so where do those skills go? Uh, well, the, the most basic place is if you have a repo, there's ... You can just put a .claude directory, and then, uh, have a skills directory, and then a folder, which is the skill name, and then a skill.md, all caps, uh, in there, just like this, and

106. 13:35

     that will be a skill that lives with that repo. And so anyone who is using that repo, it'll just automatically load that and understand how to use it. You can also have that same .claude directory in your home directory and put those skills there, and then they're generally applicable to everywhere, uh, that you would be using Claude.

107. 13:51

     Same thing, uh, there, there's kind of more standardization for everyone else on .agents. I wish, [laughs] uh, there was, like, agents.md and instead of claude.md, and, uh, .agents instead of .claude.

108. 14:03

     Uh, but maybe we'll get there one day. So, uh, you can put them in there, and, uh, if you've ever used, like, the MPX Skills, uh, tool from Vercel, that is just kind of symlinking them all into all of these different directories.

109. 14:13

     And so the skills are generally applicable everywhere. That's just a, an easy way to load and install them, which is why it's so popular.

110. 14:20

     Yep.

111. 14:23

     But the, the main dev loop with it is, like, you edit the skill, save it, invoke it, see what output it is, and then do that process all over again, uh, and test it.

112. 14:31

     If you're using Claude as well, Claude ships with a fantastic, uh, skill builder skill or skill creator skill.

113. 14:37

     Skill, yeah.

114. 14:38

     And, uh, that is really good for critiquing your skill, setting it up in a way that Claude would expect it to be, uh, and even evaluating it, which we'll talk about.

115. 14:45

     Yep.

116. 14:46

     All right. So we're gonna start by, uh, letting you go ahead and work and build the foundation. So you should have that repo, and, uh, we just wanna get started with it.

117. 14:54

     So, um, the main things that you wanna do is you wanna set up a, a proper description for it. Now remember, this description is not for humans. The description is really more for the LLM so that it knows when it should use the skill automatically.

118. 15:10

     And so you wanna set that up in, in some way. Uh, we recommend in, in some way where, like, it, it describes, like, oh, we're gonna roast this repo and s- like, like, the, the user wants to roast this repo and get a, an analysis, a fun analysis of it, uh, or, or something.

119. 15:25

     Be creative and, and fun with it. But then you should just be able to, like, open up Claude and say, "Roast this repo. Roast my repo," and it goes and does it.

120. 15:31

     Yep, and then remember that in general it's recommended that instead of being overly prescriptive in how to do something, provide your constraints instead. So say, "We're using this format in this repo," or, "We follow these coding conventions," or, "We never do X or Y," and then allow the LLM to make the right determination at runtime.

121. 15:48

     Yep. Yeah, definitely, like, closing it off like that. Don't s- uh, prescribe what it should do. Kind of give it advice on what it shouldn't do, and let it be more creative on things.

122. 15:57

     But you can also, like, change that as well and be more assertive on things that you know you want in a specific way.

123. 16:04

     So let's work on that. Um, a couple of things that, uh, like, tips that we wanna talk about in this first section is, uh, and this I think might be pretty applicable, only applicable to Claude right now.

124. 16:14

     I did ask Pi if it could do it, and Pi just, like, made an extension that made it work. So, uh, that's, that's awesome. Uh, but if you use the bang and then backtick, uh, backtick's for like a script call, uh, Claude will do, like, an interpolation of that, just like how JavaScript has, like, the dollar sign,

125. 16:31

     open curly brace, and closed curly brace. It'll just, like, instead of having whatever was in there, uh, like this, um, where it's saying stale to-dos, and then it gives you a command to run, it will just replace this with a list of the stale to-dos because it will actually execute this grip command, uh, and then do all

126. 16:46

     of these pipes to all sorts of different things. That's totally not slurping up keys or anything.

127. 16:50

     Yeah.

128. 16:51

     Um, and then-

129. 16:52

     But you, you can imagine how this is really powerful if you're, like, say, you're doing your morning report, your, your, kind of like your Git status report. Any of the pieces that you want to be output in a deterministic way, that's an ideal, uh, use case for this kind of script interpolation.

130. 17:05

     Yeah. This is really great because you're not, you're not saying, "Go grab the latest commits or the latest 10 commits and give me some information on it." You're saying, "Here are the latest 10 commits in the exact format that I expect you to understand them in.

131. 17:18

     Go and do something with that information." So it's not guessing. It's not gonna be non-deterministic each time. It's going to start from this deterministic base and then go from there.

132. 17:26

     It's also very token saving. If you've ever said, "Go and figure out the 10 commits," and you've run that-

133. 17:31

     Yeah

134. 17:31

     ... more than once or on three different terminal tabs, you know the first two might get it perfectly right, and the third is, like, spinning and r- reading Git docs and, you know, before it finally gets there.

135. 17:39

     So this is a way to say, once you've formalized a piece of your workflow, you can just codify it and say, "Run this exact script."

136. 17:46

     Yep. Yeah, like we said, without scripts, the AI, AI is just speculating on what you mean when you say, "Go get the latest commits."

137. 17:55

     Um, yeah, and just remember that descriptions are routing rules, right? They're, they're less for us, and they're more for the AI to determine when to use it. So for ...

138. 18:01

     A good example is you might have a couple different image generation skills, and they're all kind of littered in the projects and maybe in your global skills. Maybe one is more applicable to your personal blog and you say, "On my personal blog, I always ship pixel art."

139. 18:12

     So if we're writing on this domain, this is the skill to use, right? If we're going to work, it has to be formalized and we use a completely separate image generation system, or we only fetch images from S3.

140. 18:23

     That's where you can kind of codify that in your description.

141. 18:28

     And if you're not sure, by the way, you can always ask Claude. That's the other, like, kind of secret hack of this era that everyone forgets, is that a lot of times the models are capable enough that you can ask them, "Have I done this right?"

142. 18:38

     Or, "When would this apply?" Uh, so you can say as a test run, when would you load this currently? If I only want it to run in these conditions, is this the best description for me or not?

143. 18:47

     Yep. And a great example of this is when we were building this, I asked Claude. I was like, "Hey, I know I can do this, but do you actually support like skills calling skills?"

144. 18:55

     And it was like, "Oh, let me go check." And it loaded like a Claude code analyzer skill to get that information, and then do that research and come back and say, "Maybe." [laughing]

145. 19:04

     Yeah. It was like kind of, but you probably don't wanna do that, so.

146. 19:07

     Yeah. Uh, so your turn. We- we're gonna take some time to go do that.

147. 19:11

     Yep.

148. 19:11

     Uh, to, to let you go do that. Um, and when we do these breaks too, this is a great time if you have any questions, uh, or have, uh, discussion topics that you'd like us to talk about.

149. 19:22

     Uh, we can do that. We're trying to like fill the dead air of like you working on these with, um, general topics. So if there's something that you want us-

150. 19:28

     You're supposed to say that part out loud.

151. 19:29

     Oh, that's okay.

152. 19:29

     Yeah.

153. 19:30

     Uh, if, if there's something that you want us to talk about, uh, we can definitely do that. Otherwise, uh, we've got some discussion topics that we thought we could talk about.

154. 19:37

     Yep.

155. 19:38

     Uh, but if you ... Also, if you have any questions or, um, any of that, we can definitely-

156. 19:42

     I'll run over to bring you the mic, and, uh, feel free to shout out any questions.

157. 19:46

     Yeah.

158. 19:47

     Um, but yeah, if o- otherwise, then feel free to, uh, just start on this.

159. 19:53

     And if there's any questions, let us know.

160. 20:01

     Yeah, question. You wanna run the-

161. 20:03

     Sorry. Where is the question? Yeah.

162. 20:08

     Future us. Very good. Nice.

163. 20:11

     Yeah.

164. 20:11

     Um, you talked a bit about this in the beginning. Um, but I always wonder where to draw the line between, um, encoding instructions, um, in like rules, Claude.md and so on, and creating a skill for something.

165. 20:24

     Um, so I'm curious if you have like h- what's your mental model to making that decision? Like, have you landed anywhere? Like, do you always start with the rule and then you make a skill if you can make it specific enough?

166. 20:34

     Or do you always start with a skill? Like, how do you go about it?

167. 20:37

     Yeah, great question.

168. 20:38

     Great question.

169. 20:38

     Uh, I usually... Like, like the, the one, the number one rule that you have to remember is that the skills... Or, or sorry, the Claude.md or the Agents.md, that is going to be loaded every time.

170. 20:49

     When you kick off Claude, that's going to fill your context window, and if it's filling it with a bunch of nonsense that isn't actually applicable to what you're specifically doing, then you probably don't want it in there.

171. 20:59

     Um, I can show an example of like my, uh,

172. 21:03

     what is it? .claude, uh, and then I think claude.md, if I can spell.

173. 21:12

     This is my claude.md. It's extremely small. Uh, it just tells it that I want things to be a little bit more terse. Don't bloviate, I just want to know exactly what you're saying.

174. 21:22

     Uh, be extremely concise. And then I also like, I have this plugin that I'm working on, it's a, a, a skill actually called Ideation, and I, in here I put like some configuration for that so that all of the projects, I, I basically want them all to put like the ideation, the artifacts that it's generating, into, uh,

175. 21:39

     my Obsidian vault. So it puts it all in there so I can more easily like find the connections between things. Um, but otherwise it's like extremely tiny. And so that, that's one thing that goes into it.

176. 21:51

     If it's only relevant to the repo... Like, like specifically, you know, I'm tired of it using NPM when I want it to use PNPM, for example. I'll put that in there, like a single line that just says, "We use PNPM here."

177. 22:02

     Um, and then anything else, like if it's, you know, more specific about testing or anything like that, I kinda leave that to skills so that it's only going to be loaded when I'm actually like writing tests.

178. 22:13

     To, to the second part of your question as well, the... And we'll talk about this a little bit later, but the other thing that's really fun to do is basically wait a week while working on it, and then go back and ask Claude, "Analyze my week's worth of work, and then what are the skills I should split

179. 22:27

     out of that based on this?"

180. 22:29

     Yeah.

181. 22:29

     Um, so again, ask the system to kinda help you do that. Another question back here. Yes, sir.

182. 22:34

     Okay. You can hear me, right? Yeah.

183. 22:35

     Yep.

184. 22:35

     Yeah.

185. 22:36

     Um, so stop me if you're gonna talk, if you're planning to talking about this later. I was wondering about, uh, global skills, uh, which we will share amongst colleagues.

186. 22:46

     So we're all, at the moment we've, we've got I think 60 engineers. People are writing their own skills. We're chatting on Slack, "Oh, I've got this great skill, it's really good."

187. 22:54

     So then obviously, uh, engineering managers are like, "Well, we should be sharing these. Where is it? Where do we keep these? Where do we keep them in repo? Um, what's our artifacts library?"

188. 23:04

     And then others have said, "No, we don't want that, because if I put my skill up and then someone's like, 'Oh, I'm gonna change that,' then we're gonna have MR requests, and then we're gonna have to review changes to skills."

189. 23:13

     Yep.

190. 23:13

     So then we'll get someone else saying, "Well, I kinda like your skill, but I'm now gonna push my version of that skill with a very similar description to the shared repo, which everyone's gonna get."

191. 23:22

     And then suddenly we've got 10 frontend.ux skills-

192. 23:26

     Yep

193. 23:27

     ... and they're all, the agents then, which of these do I actually use?

194. 23:30

     Yeah.

195. 23:30

     And we're wondering if you guys have got to that stage of how to maintain. And then the next one is, three months later a new model comes out, and these skills are actually a little bit too verbose.

196. 23:41

     Mm-hmm.

197. 23:41

     So who's evaluating the skills and checking them and saying, "Okay, let's cut these from the global because now..." You get what I'm saying?

198. 23:49

     Totally. Totally.

199. 23:49

     This is where we're at with-

200. 23:51

     Yep

201. 23:51

     ... how, um, so a lot of engineers are just like, "No. No skills. Everyone does it on their own."

202. 23:57

     I know, I know-

203. 23:58

     "We are not sharing anything."

204. 23:59

     Yeah.

205. 23:59

     So sorry, that was a bit of a rant.

206. 24:00

     No, no, no. That's-

207. 24:01

     You get where I'm coming from, yeah

208. 24:02

     ... fan- fantastic question. Uh, I'll take a first stab. Interested to hear what Nick says. We have, um, published, uh, maybe you wanna pull it up, like GitHub WorkOS skills.

209. 24:10

     That was one of the first places that we started publishing generic skills, and that's been incredibly useful because, for example, I was building generic RAG pipelines, and then we found that agentic tool calling is higher performance.

210. 24:20

     So I can side load those skills that Nick put in there that are specific to certain documents. To the problem of i- individual engineers like saying, "I want a slightly modified version of this," I, I, um, I would almost say like in that case, cool, you've got a forked skill you keep locally.

211. 24:35

     Um, and then to your question about, you know, re-evaluating the skill, I think asking Claude, like, with your current model, look at the skill using the skill builder. I- is it right for truncation or is there like, you know, additional extensions that we need now?

212. 24:49

     Um, but I'll also share that we are feeling that same pain as I'm sure everyone else is, and I think the management layer is just shifting to that kind of-

213. 24:56

     But they think even if you ask-

214. 24:58

     Yeah.

215. 24:59

     E- Sorry.

216. 24:59

     No, you're good.

217. 25:00

     Even if you then ask Claude, let's say, a week later, a month later, "Hey, review our skills. There's 30 skills to review," and it comes up with lots of suggestions, you then got to open a merge request for possibly one human or two humans, or maybe you...

218. 25:12

     we can automatically say the, the person who wrote this skill originally has to be one of the reviewers. They have-- Have you got down to that yet where-

219. 25:21

     We, we-- I don't think we've gotten to that level with ours because ours started kind of formalizing documentation into buckets that were then-

220. 25:28

     Mm-hmm

221. 25:28

     ... easily side loadable in different systems. Um, that does sound painful. I'm curious to think... What, what do you think about that, Nick?

222. 25:34

     We haven't even got there. We're just f- people have just like foreseen that this is gonna happen.

223. 25:38

     Yeah.

224. 25:38

     So they're actually blocking us using shared skills at work-

225. 25:42

     Right

226. 25:42

     ... because they think this is gonna be the problem. So [laughs]

227. 25:45

     Yeah.

228. 25:45

     Like literally we're overthinking it massively.

229. 25:48

     Mm-hmm.

230. 25:48

     We should just do it and try, but still-

231. 25:50

     Right

232. 25:50

     ... interested to hear what you guys-

233. 25:51

     Definitely. And I, I also think it's gonna evolve rapidly too, right? As we're seeing, like there's still... We haven't quite hit the LLM training wall, right? So there's gonna be kind of additional capabilities coming online and, and yeah, what does it look like in six months?

234. 26:03

     Could we pair the skills down even further and get the same or better performance?

235. 26:07

     Yeah.

236. 26:08

     But yeah. I'll say that that's, um... Yeah, that sounds like a typical human problem of, uh, my skill, your skill, right?

237. 26:13

     Yep. We have a number of... Like, to, to build on that, uh, we have a number of ways that we solve that. Like Zack said, this, um, this skills repo, this is like our public skills, uh, that you can just install with like NPX, uh, skills add, uh, and, and those are all available.

238. 26:29

     But then we also have, uh, some like internal skills that are more, uh, generally applicable to like engineers at WorkOS. And so it's like there's an auth specialist, there's a DX specialist, there's uh, a ghostwriter, d- different ones like that.

239. 26:42

     Um, and then I have my own plugin marketplace as well, where I put a number of skills that are applicable to me. Uh, and so I just load from all of these-

240. 26:51

     Yeah

241. 26:51

     ... uh, in different ways. We also have like a, a big monorepo that like most of the engineers work in, and you can... A, a lot of skills just end up in there if they're monorepo specific.

242. 27:00

     Yeah.

243. 27:00

     Uh, that's a much easier place. But yeah, it's the same problem. Like you gotta get a review on it or it's gotta be... It, it feels kinda dirty 'cause you're just like appending that to the work that you're also doing, so it's like an, an add-on, which doesn't feel super great on the PR.

244. 27:12

     If I reverse engineer to some degree the plugin system, I think that's what they're trying to address kind of.

245. 27:16

     Okay.

246. 27:16

     'Cause you can also install like a version of a plugin the same way you can an NPM package, right? So maybe that's kind of like the interface on top of the repo, and then the tooling that I'm seeing everyone keep building repeatedly is like the tool that reads from a repo and installs skills into various places, like

247. 27:34

     two ease and stuff that make that kind of like nicer. But that might be a solution to some degree where it's like, cool, there is this mas- you know, master skill of this, but I'm running this version because I need this fork.

248. 27:44

     Um, and then, but it's not as gross as it sounds because there's an actual standardized API with the plugin interface, right?

249. 27:50

     And it's all versioned.

250. 27:51

     Yeah.

251. 27:51

     When you NPX... Sorry, guys.

252. 27:53

     No, no. That's, these are great.

253. 27:54

     When you NPX, do you then have flags?

254. 27:57

     If you were to go into the public one or your internal workflow at all, I'm front end, I don't want every skills, even if the a- agent knows not to use the back end.

255. 28:09

     Do you then flag way like, um, MPX public flag, just front end or just UX or just product? It's just come to my mind. I've never thought. Have you done it?

256. 28:17

     Is it, is it-

257. 28:18

     I haven't. No. Um, I haven't used NPX for that. I just used like the... Like, like I said, we're mostly Claude. I, I've used the, the Claude marketplace, like the slash plugin marketplace ad, and as long as your Claude instance can, uh, access like an internal Git repo, it can just pull from there.

258. 28:34

     Uh, and so that's what it does.

259. 28:35

     It will, but it would pull all the skills, even the ones that you don't need, 'cause you're front end, you don't want the back end skills.

260. 28:41

     Oh, yeah.

261. 28:42

     That's it.

262. 28:42

     Yeah. Then that, that almost sounds like a packaging thing to me. But I-

263. 28:46

     Yeah

264. 28:46

     ... I think, I think that you're kind of like in good company in a sense that it seems like, you know, we, we're kind of got three marketplaces that are super relevant separate from or in addition to the project specific like skills, right?

265. 28:57

     And then it just kinda becomes a matter of taste of each individual engineer saying like, "Oh, I'm gonna run this version of that skill." But then something like the plugin, like interface is the way that you have a uniform way to approach it, which you could actually write docs against for onboarding and say, plugin add these three

266. 29:13

     marketplaces when you come on board, and then if you're on a front end team, like plugin install from the front end marketplace or whatever the case may be. But that's like still at the end of the day on the back end, that's like repo management and it's, right, it's similar to how it works with code.

267. 29:27

     Yeah.

268. 29:27

     Yeah. Great questions. Um, yes, sir.

269. 29:30

     Sorry.

270. 29:31

     I forgot who was gonna ask. Um, yeah, um, so actually two questions. The first one is, do you do any like formal skill evaluations, like a skilled benchmark so that as new model drops, which skills are relevant?

271. 29:41

     Yes. Um, in the, the public skills, specifically on the, the ones that I use internally, I am a little less formal about it. Uh, but the ones that we actually ship, uh, we do ship, uh, in the...

272. 29:55

     Where is it? We have like a whole evals framework, uh, that we wrote to make sure that it, it lives up to the standards that we have. And we're gonna, we're gonna talk about this a little bit, but like it's mostly, uh, like doing several runs where it will r- load Claude without the skill and ask it

273. 30:12

     to do a task, and then load it with a skill, and then it kind of has like a rubric on confidence or, or like a, a grade that it gives it, and it's, it, it'll fail if that grade is less with the skill than it was without.

274. 30:25

     Uh, it'll also fail like it, you know, it, it tries to be, I think, eighty percent above or higher. So like eighty percent of the time, uh, or maybe it's ninety percent, uh, it's going to get this right with the skill, and sometimes it gets it right without the skill.

275. 30:37

     So the skill is maybe only adding one or two percent to it, but that's something that we track and keep on top of as new models drop.

276. 30:43

     Yeah.

277. 30:43

     Okay. Yeah.

278. 30:43

     It's, it's sort of fuzzy math, but it's almost like by having this, this established baseline, you can at least test that way.

279. 30:49

     Yeah.

280. 30:49

     Sorry.

281. 30:50

     Uh, it makes sense. And then the second question was, uh, about, um...

282. 30:56

     Wait. Sorry, one second. Right. Uh, skill pickup. Uh, so if you get lots of skills, uh, the models might ignore a skill or decide, I don't need a skill, I'll just, I'll just do it.

283. 31:10

     What's your kind of experience with this to, A, like, test it, find it, and then maybe improve it? Yeah. Um, great question. We... That, that is a problem, and the more skills that you get, like, you can have conflicting skills.

284. 31:21

     Uh, and so, like, which one is it gonna pick? Um, the solution to that, like, like, for the WorkOS one specifically, like, we try and keep it... Like, for these public ones, we try and keep it, like, very generic.

285. 31:31

     Like, mention all of the, you know, the, the acronyms and things that we would want, we would expect to cover, uh, from that. So that'll trigger it to load.

286. 31:39

     Uh, and it usually does a pretty good job. You can also, like, if you're in a skill, uh, or, sorry, in Claude, uh, you can just do, like, WorkOS, for example.

287. 31:48

     Like, uh, the slash command, uh, if you know that you wanna do it. And so, like, a lot of times we'll just, like, suggest... You know, if, if that's what you want, I'll say like, "Just run /WorkOS," and it'll-

288. 31:57

     Yeah

289. 31:57

     ... it'll load it.

290. 31:58

     I'll, I'll, I'll call a skill by name if I want a specific, like, image gen or something, and I'll say... Or I'll say, like, use the Superpowers brainstorm skill in order to determine a better plan.

291. 32:06

     Yeah.

292. 32:07

     Yeah.

293. 32:07

     Okay. Thank you.

294. 32:08

     Yep. But also, if you got... If it really wasn't behaving, that's why you use the bang, and then put a command.

295. 32:15

     Right.

296. 32:15

     Yeah. [coughs]

297. 32:18

     Um, I had a question on, um, how do you decide when to create a sub-agent versus a skill, and can you reuse a skill into a sub-agent? And there's just some times that, um, I'm gonna create a skill, and then I'm gonna, like...

298. 32:35

     Uh, maybe I should've written, um, a CLI, 'cause, uh, why did I even made a skill in the first place? And I, I struggle between these three things.

299. 32:47

     That's a great question. Uh, on the can skills... Can sub-agents use skills? I actually [laughs] am, like, blanking on that, so I'm asking Claude. Uh, and you can see that it loaded the Claude Code Guide pl- uh, skill to go check that.

300. 33:00

     Uh, so this is a great example of doing that, and we'll get the answer here in a moment. Uh, but that's a great question. Uh, sub-agents is something that we don't cover a ton in this workshop, uh, but it is something that's super valuable.

301. 33:11

     And the s- the number one thing that I think i- uh, think of, like, when I think of w- when to run a, a sub-agent versus a skill, is do I want it to have its own standalone context?

302. 33:21

     Uh, so that it can go do, like, a bunch of work on, on something, and then that's not eating the context window of, like, the main task job that, that we're doing.

303. 33:31

     Uh, and then that way it can just, like, do a check-in on that. And so, um, for example, uh, I have this ideation plugin. It's kind of like a, a planner or, uh, uh, a Superpowers, uh, type thing that I like doing.

304. 33:43

     And as part of that, like, I'm really, like, focused in on feedback loops to itself so that it doesn't have to bother me all the time about, "Hey, does..."

305. 33:52

     You know, "Does this look correct?" Or, like, tell me, "Oh, it's done," and it's totally not done. Like, I want it to prove to me, without me having to go look at the code, that it's, it did the work that I expected it to do.

306. 34:01

     And oftentimes that's feeding the information that I would look for back into it and making it just go in a loop over and over. I hear there's a, a Ralph Loops pl- uh, workshop after this, so you should check that out.

307. 34:12

     Um, but it, uh, it, it, like, in that case, like, when it's doing those reviews, those can, like, muddy it up, and so, like, I kick off a sub-agent to go do those reviews, and then it just reports back like, "Ah, there, I found these problems."

308. 34:25

     And then it just has a list of those problems, and then it can feed back to itself to do it again. So I'm not eating that full context window every time.

309. 34:31

     Yeah. Now also further confused by agent teams, which are different [laughs] than sub-agents too, right? Uh, yeah. Another question. Thank you.

310. 34:40

     And, yes.

311. 34:42

     Well, we got another... One other question here.

312. 34:44

     Yes. Um, I have a question about the, the overrides in a skill. So for instance, you, you put a default and you say, "Or whatever the user decides." But I find it's very random, or at least I cannot really reproduce that, and sometimes the overrides doesn't work.

313. 34:59

     Yes.

314. 35:00

     Do you have any idea? Or like, uh, I, I just want to find out what's going on.

315. 35:06

     Uh, my, my best suggestion for that is just ask Claude. [laughs] Like, why did you pick that over the other thing? Uh, and how can I improve that in some other way?

316. 35:15

     Like, like, you consistently, or like s- you consistently enough pick the wrong choice, or you don't respect my override. Why is that? What can I do to improve it?

317. 35:25

     Um, I wish I had a, a more clever answer, but usually it's just, like, I ask the machine.

318. 35:30

     Yeah. Just, just ask Claude.

319. 35:31

     Yeah.

320. 35:31

     He, he is good enough. Thank you. [laughs]

321. 35:34

     Great question. Question?

322. 35:42

     Uh, you called out, uh, Superpowers was a, a, a skills library that you referred to. Is there other, uh, skills libraries beyond you guys that you commonly use?

323. 35:53

     Yes. Uh, definitely. So Superpowers is one that I actually didn't use until yesterday when Zack showed me it. Uh, and I s- I installed it. It has a number of different skills in it that are, are pretty helpful.

324. 36:05

     Um, these slides are actually written in Slidev, uh, and you might notice... Well, you won't. Uh, I don't know if I committed it. Um, let me go to the full repo here.

325. 36:17

     But in here, there's an agents directory and a skills and a Slidev skill, and s- Claude might have had a hand in writing these slides, uh, which is really cool.

326. 36:25

     We'll kinda talk about that, but, like, some of the real superpowers, I think, are, like, when you assign it to do non-coding things, uh, because you really feel this magic.

327. 36:34

     Uh, and we'll, we'll kinda show a demo of the Remotion skill.

328. 36:37

     Yeah.

329. 36:37

     That one blew my mind. Uh, it's, it created a video based off of a prompt, and-

330. 36:43

     I, I now use that as my... So every F- Friday when we have the all-hands, and it's like, what quick demo of what you got done this week, right?

331. 36:49

     It just reads my Git history for the week And then builds a movie

332. 36:54

     about it, which everyone is, was tired of on week three, and they're not gonna- [laughs]

333. 37:00

     They're not gonna stop. They're just gonna get more, like I'm gonna introduce characters and s- it's gonna get awful. [laughs] But, uh, yeah, the Remotion's incredible. It'll even pop up a, like, Chrome-based web editor where you can go and be like, oh, trim and cut, and like let me add some fades, right?

334. 37:14

     Um, so that's insane. And then my favorite one that I'd probably get the most leverage of, uh, since installing it was just a... I, I built it with Claude, just a simple Python wrapper around NanoBanana.

335. 37:23

     Mm-hmm.

336. 37:24

     Um, the image gen model from Google, which continues to improve. So I just say, "Hey, now it's on V3. Go update it." And we'll show a little something later, but, um, essentially with that...

337. 37:34

     So most images now I generate with that, it takes like sub seven seconds in a single prompt. But using that same model, I was able to, say, uh, take a single string from the user that's a prompt, say, like a child running through a field.

338. 37:46

     First it makes that image, then it uses their video API, Veo, hands that static image to it, and says, "Animate this static image in the most obvious way possible."

339. 37:56

     So one user prompt of child running through field, nothing exists, and then 30 seconds later you have a video of it running through. And I was able to use that same method to do all of the interstitial scenes that I needed in a 32-minute film, and I am not a video person.

340. 38:11

     Like, I mean, I like using like DaVinci Resolve and editing stuff, but I'm not an animator, and I was able to get all of that done in like may- maybe an hour.

341. 38:20

     Um, so those are, those are pretty trippy too. Um, if you wanna get really, really down there, like I've got s- I have like Claude reading my biometrics and stuff, and like pushing back on me and telling me to like take it easy this afternoon 'cause you didn't get any sleep.

342. 38:32

     So, um- [laughs] But there's, there's not like necessarily a skill for that yet. I, I think the, the ones that are really powerful are when Claude, uh, the other day blew my mind by saying, it was also superpowers, "This is easier for me to show you the variants if I just mock them up in a web browser.

343. 38:45

     Would you prefer that?" And I said, "Yes, please." And then it showed me all those, and I was like, "A, go." And then we just built from there, like saving countless tokens on just text-

344. 38:55

     Right

345. 38:56

     ... iteration.

346. 38:57

     And I'm using that NanoBanana skill right now, but uh, I just ran /plugin and I'm looking at the marketplaces I have installed, and some of the most important ones to me I think are the, the Claude Clo- Claude Plugins Official one.

347. 39:09

     Uh, I think that's where it has a nice, um, skill reviewer skill or skill creator skill, uh, which is really good. Uh, Obsidian is something that I use all the time, uh, and so having the Obsidian skills and it knowing just how to use that.

348. 39:22

     Uh, so it's based on what I want, but then also one that, uh, is actually very good... Oh, where did it go? Plug-in. Uh, is the, um, Codex skills.

349. 39:34

     Marketplaces. I don't know why it's not showing it.

350. 39:40

     There. The OpenAI Codex ones. Um, it's not scrolling down. But anyway, that is like, uh, Co- Claude does all this work. Codex is pretty good at reviewing it. So this is a skill from OpenAI that just like pipes that to Codex and says, "Review this," and it goes and reviews it, and then delivers that back to Claude.

351. 39:58

     And I have cut myself out of the copy and paste game of Claude said this and Codex said that, and like going back and forth in, in Tmux splits.

352. 40:05

     So I'm, I'm super happy about that.

353. 40:07

     Yeah. I would say Vercel is pretty skills forward. They've got a bunch of CLIs and stuff that are, are pretty interesting. So they... And they were like kind of the first on using some of the marketplace stuff, so, uh, check out their like open source skill stuff too as well.

354. 40:21

     Great question. Thank you. Okay.

355. 40:27

     That's what NanoBanana just made. [laughs]

356. 40:30

     Close. Close-ish. Yeah.

357. 40:33

     Awesome. [laughs]

358. 40:34

     Uh, but the fun thing about that is that you can ask it for any style.

359. 40:36

     Yeah.

360. 40:36

     So you can say like... I, I mostly do pixel art, um, so I'll say like, you know, old school pixel art, and uh, it's a lot of fun.

361. 40:43

     Yeah. All right. We are at time for this, uh, piece of it. Did anyone, uh, build a, um, a gen- a first like pass on the repo row skill that they wanna share?

362. 40:55

     Yeah? Yeah, I uploaded it.

363. 40:57

     Cool.

364. 40:57

     Awesome. If you wanna run, um, ./share.sh. I know. I, I uploaded it. Oh, cool. Sweet.

365. 41:04

     This, this guy, all right, wins the workshop.

366. 41:06

     All right. Uh-

367. 41:07

     Are you Sharif?

368. 41:09

     I'm Sharif. You're Sharif? Okay. Are you, uh...

369. 41:15

     Okay. I'll run, uh... Well, I'll just run all three of them real quick. Uh, so I'll run them on the, uh, skill. Or, sorry, this workshop.

370. 41:24

     Oh, what did I do?

371. 41:26

     That's pretty safe. Don't you wanna run it against like WorkOS or something, or the CLI?

372. 41:30

     Uh, yeah. I just realized that it loaded it locally into this one, not in a global way.

373. 41:35

     Oh.

374. 41:36

     Uh, I can do that. Okay.

375. 41:44

     Sorry.

376. 41:47

     I'm gonna give it the, our WorkOS CLI, and then I'll say, uh, repo roast Zack B on the, uh, CLI repo, and we'll see what it does.

377. 42:00

     That's a new verb, defending the home verb. [laughs]

378. 42:05

     Oh, yeah. You can c- you can, uh, customize those. So I, a lot of my, uh, spinner verbs are Lord of the Rings or The Office themed.

379. 42:12

     Okay.

380. 42:13

     Uh, so you'll see like defending as Gilliath and things like that. [clears throat]

381. 42:27

     I didn't, I didn't think that would work, but that's okay. It's running against the s- the, uh, workshop repo.

382. 42:31

     Okay. That's fine.

383. 42:38

     So it's running all of the commands, uh, that you gave, and while you're, while we're doing that, I will bring that up.

384. 42:50

     So this is Zack's skill. Nice, good description

385. 42:55

     Analyze repository health by running Git and file system scripts to find stale todos, churn hotspots. Yeah, that's good.

386. 43:04

     And then it tells it specifically how to find stale todos. Awesome. Hotspot files, largest files. Nice. Constraints, never be vague for evidence. Never present a finding without a script output or Git data backing it.

387. 43:18

     That's probably why it's running still. Oh, nice.

388. 43:22

     Um, yeah, scope.

389. 43:28

     Okay, Zack B. Okay.

390. 43:30

     Nice.

391. 43:31

     Nice skill.

392. 43:32

     You, uh, didn't tell it to, to just, like, be mean to you.

393. 43:36

     Yeah.

394. 43:36

     So that's-

395. 43:36

     Be super mean to Nick and Zack over on stage. [laughs]

396. 43:39

     Awesome.

397. 43:42

     Try Amy. She's pretty mean sometimes. [laughs]

398. 43:47

     All right. Awesome. We will, we'll run more of these. We've got more, uh, more things to get through. Uh, and we'll, we'll do this again, and we'll, we'll test another one.

399. 43:56

     Yep.

400. 43:57

     Uh, so moving into the next section, uh, we're gonna make that skill smarter.

401. 44:02

     So the first thing, uh, that you can do to make your skill smart, uh, is by providing more information to it. But this gets into the problem of the CLAUDE.md, where you can be extremely verbose in there and give it so much information about your repo, and you're just bloating the context window 'cause it doesn't really matter.

402. 44:19

     Well, you can do the same thing in skills, uh, but you can do it in a better way, and that is specifically with progressive disclosure. And I guess you could do this in a CLAUDE.md as well.

403. 44:28

     Uh, but all it is is just saying, like, "Hey, if you're thinking about doing testing for..." Excuse me. "... for example, uh, load this file that I have on testing and read through it."

404. 44:38

     Uh, and you just give it, like, a path, like a local path to TESTING.md or whatever. And that way, it's only gonna load that if it's actually doing, like, a testing skill or a testing task as part of the skill run.

405. 44:49

     If it's not, it'll skip that. And so you can, uh, specifically tell it, like, oh, in, in this example, uh, if you're doing, like, a scoring, like if you're, if this is a run where it's doing scoring, ro- run the scoring, uh, ah, load the scoring rubric-

406. 45:04

     Yeah

407. 45:05

     ... uh, and read through that so we explain to you how to score things properly. If we're not doing scoring, you don't have to load that, and we don't have to fill up the context window with all of that bloat.

408. 45:13

     Yeah. This also g- gets back to that gentleman's question in the back too of, like, you can imagine this pattern really scaling out. So the way that it actually did scale out i- i- even in our public WorkOS skills repo you can go and check out, we have multiple migration guides that we publish for various folks.

409. 45:25

     So, like, if you're coming from Auth0, we'll happily help you move off Auth0 to get to WorkOS, and then there's N number of, you know, competitors essentially that we've got migration scripts for.

410. 45:36

     And so in this case you could say, "Here's the migration skill," and the migration skill is a pointer to the specific reference, so you're not bloating your context window, it's just loading the two Markdown files it needs.

411. 45:46

     Yep. And if you look at the WorkOS skill, like it... We literally call it skill router, uh, in there, and it just has, like, a reference map. So if you're going to install AuthKit into Next.js, you should probably load the workos-authkit-nextjs.md file, uh, from the references.

412. 46:01

     And so if you're not working with Next.js, we don't wanna, we don't wanna load that and bloat that. We only load it when you need it for all of those.

413. 46:07

     Yeah.

414. 46:07

     And so this file is just filled with, uh, routing to the actual pertinent information that you need.

415. 46:19

     Okay. Um, another way, this is again, to some degree it's fuzzy math under the hood, right? If you really get down to, like, m- matrix multiplication. But, uh, nevertheless, another way to boost performance here is to kind of enforce confidence scoring, and one of the reasons that Nick's, uh, ideation plugin, which is open source that you can

416. 46:34

     go check out, works so well is that it has, like, an internal counter of confidence of how close am I to fully fleshing out all of the variables that this task requires before I can go and execute.

417. 46:45

     And it then forces, like, a iterative loop with the user of continuously asking additional questions until it gets to the point where it's like, I'm 95% or above confident I've mapped most of this problem space in my head, and now we can start work, and the result as, um, you know, as a result, the, the output is

418. 47:02

     likely better.

419. 47:03

     Yeah.

420. 47:03

     Um, and so y- that same concept applies here when you're, you know, building skills. You can, you can kind of add in that, that same functionality and say, uh, for this particular aspect of the code base you must always find this evidence and then get to a point until, you know, uh, the tests are either at this

421. 47:17

     level of coverage or you have this level of confidence on, on A, B, or C, um, and that's another way to essentially boost performance in the skills.

422. 47:24

     Yeah. Uh, it's, it's really important, like, like, like Zack was saying, it's just, like, kind of pulling that number out of nowhere. If, if you say, "How confident are you?"

423. 47:33

     And it's like, "I'm sure confident." Uh, well, why? And as you give it, like, ask it to, like, show more of its work as to why it's confident, it might be like, oh wait, I'm not actually as confident as I thought.

424. 47:44

     Uh, and so that's the whole thing is, like, trying to get it to think more. Uh, in the terms of, of, like, the, the ideation skill, uh, what it's doing is it's using that to assess that it has, like, a full understanding of what I'm trying to say because, like, I have a problem where I don't give

425. 47:58

     it enough information. I have the information, I know what I want, it's hard for me to express it to the, the machine in a way that it expects. Uh, and so it's using that confidence score to say, "Eh, I don't have, like, a full rounded understanding," and it loads, like, a whole rubric on what it means to

426. 48:15

     be confident on something. Uh, but then if it's not confident, it uses, uh, Claude's built-in ask user question tool to ask me a number of questions to pull that information out of me rather than me being like, "Oh, you're not confident.

427. 48:27

     Let me try and give you more insight." It's like, "No, I'm not confident because of these things, and then here's how you can make me more confident by answering these questions."

428. 48:36

     And a lot of times it'll just give me multiple choice on, like, you know, do you want this? Is this what you mean? This is the recommended approach I would take, but if you wanna go this other way, we could do it that way.

429. 48:44

     Uh, and so, like, we have that dialogue going back and forth with it, but it's all based around how confident is it that it understands what I want and understands how to do what I want.

430. 48:58

     Uh, yeah. And so then this gets back to, uh, just kind of in practice the way this works or at least how it has for me and what we kind of recommend is, you know, build an initial skill.

431. 49:06

     Maybe you're doing that yourself in Markdown, maybe you're using the skill builder in Claude and saying, "I need a skill to do X." You're doing it, you're using that skill for a couple of iterations, maybe a couple days, maybe a week.

432. 49:16

     Um, you look at what it produces and then, you know, keep in mind that as you're having multiple conversations with, say, Claude over the course of a week, all of those conversations are even getting saved locally to some degree in JSONL files.

433. 49:26

     And so you can, um, be honest with the evaluation phase about is this actually improving things, is it not? Where does the skill fall short? What are the edge cases it's not currently capturing?

434. 49:36

     What's the annoying thing that I've now discovered that I've been running it seven days that it's missing? And then you kind of iterate and... But again, you're still going faster because you come back to a state that's already working and you say, "These three edge cases are driving me nuts," and you also need to be able to,

435. 49:50

     like, review your own PRs in the future, right? And so then once that loop is, is done, you have a skill that's significantly more powerful, and then you can keep, keep on running from there.

436. 49:59

     But it's kind of like they're sort of evolving over time. Um, so they're, again, like, I think of them as, like, organizational units of where to put kind of, you know, work intelligence, and then over time if you're, if you're doing it right, they're getting better.

437. 50:12

     Yeah. When, when skills first came out, uh, Zack and I were actually at an on-site together in San Francisco and, uh, like, we woke up one morning and they're like, "Oh, s- introducing skills."

438. 50:22

     And we're like, "Well, this looks like every other Markdown file that they provided. [laughs] What, what's the difference?" And, um, uh, like, later that day we presented on, on skills like, I don't know, four hours later, and the, the one that I built to present that was a Claude skill, Claude skill, uh, that would analyze the...

439. 50:40

     Like, it wouldn't analyze your skill running 'cause nobody had skills like four hours into them existing, but it would analyze, "Oh, you just did this task with Claude. Let's go through and pick out what could've been...

440. 50:51

     What, what we could, like, encapsulate into a skill so that it can do that in an easier way." And, like, since then there's, like, meta skills and things like that that have come out where it will analyze the performance of actually how you're using Claude or how you're using the skills in Claude, and then it can use

441. 51:04

     that to feed back in, just like Zack was saying, just by looking at those JSONL files, they're these logs of, like, the conversations that you're having with Claude, and, uh, that can inform it on how to pr- improve things.

442. 51:15

     So for example, like in the repo roast, uh, if it's kind of n- being wonky about how it's pulling in Git information, adding in like the, the bang with, uh, like, the specific Git command that you want it to run to get log information, that's a way to improve it so that it doesn't have to iterate over

443. 51:32

     that and say... And you come, come back to it and say, "No, that's not what I wanted. I wanted it like this." Like, you, you can be more explicit with it, and that can be fleshed out by reviewing the performance that you had the first time or the first couple of times.

444. 51:44

     The, the other intuition I'll share is that, um, it's kind of like, in my experience recently, it's the types of nagging things that I find the most cognitive, like, resistance to doing every week that I actually need to turn into skills.

445. 51:56

     And so, like, a, a breakthrough moment for me was realizing that, like, context switching between Slack and focusing on code, and then going and ticketing, like, new asks in Linear was so disruptive to me that I just needed Claude to do that.

446. 52:08

     So now it just monitors, and when someone asks me for something new in Slack, it goes and looks at my Linear, and then if there's not a ticket for it already, it does do de- deduplication, adds a new ticket, and then I'm...

447. 52:19

     I haven't left my flow, right? I'm still able to focus. And so, like, that's kind of the intuition I have now is that, um, you can sort of... I think it's really powerful, and I think we're only at the very beginning of it, like analyzing your own workflow over time-

448. 52:32

     Mm-hmm

449. 52:32

     ... feeding it more information about how you actually work, and then letting LLMs, you know, do what they're really good at and compress down the, that, that actual time.

450. 52:42

     Is, is there a, a skill out there that you'd recommend for that?

451. 52:46

     Uh, was there-

452. 52:47

     Is, is there a skill... You mentioned that there is skills out there that may have skills to review your kind of past conversations and pr- propose skills or improvements.

453. 52:56

     Is there one that you use or would-

454. 52:58

     That one, there's not one for that, but, um... And I didn't do this myself the last time. This was, like, last week, but what I should have done is say, "Hey, Claude, use skill builder yourself," 'cause Claude's got that baked in skill-builder, I think it is.

455. 53:10

     "Use skill builder to, um, look back at my workflow and tell me where it's the least efficient." Right? And then, um, that's also pulling in connectors 'cause there's a Slack connector and there's a Linear connector, so that's where, like, the Markdown might be referencing, "You must always use the Slack connector to pull in this, and I only

456. 53:27

     care about these channels and direct mentions of my name." Right? Um, but yeah, I think it might even be faster in some cases to just say, "Here's where I work.

457. 53:35

     This is the tool that we use to communicate. Make me a skill that does that."

458. 53:40

     Yeah, and I-

459. 53:40

     Which is also, like, kinda crazy.

460. 53:42

     I think this is the one that I was thinking of, uh, specifically is Claude Meta Skill, uh, that helps you configure Claude, uh, including, like, setting up those skills.

461. 53:50

     Uh, I think this is the one I've used, but like Zack said, I've also just asked it to review its own performance and kind of go from there. Uh, one really great thing is, like, I built this pretty cool tool and I wanted to write a blog post about it, and, uh, it, it was all built with

462. 54:04

     Claude, so I was able to just go ask Claude, "Hey, remember that time we did this fun thing together? Let's reminisce about it." And we just, like, talked about it, and it, like, led to these anecdotes that I added to the blog post that I for- I completely forgot about, but they're...

463. 54:16

     Claude was very fond of that moment between us. [laughs]

464. 54:19

     That's not what's happening. You don't understand that, okay?

465. 54:21

     No, I don't. [laughs]

466. 54:22

     Under the hood, that's not what's happening.

467. 54:23

     Don't lie to me. [laughs]

468. 54:25

     Do, do you use any skills for memory, for maintaining a, a, like, memory state within Claude?

469. 54:30

     Great question. Uh, Claude has its own memory built in. Uh, and I... There's that AutoDream thing. I don't know if that's real yet or if it's, like, a, a thing that's coming, but it will actually, like, prune the memory.

470. 54:41

     And so I've been, like, focused on building around that, uh, but I've been building it on, uh, in Py specifically. Uh, and so, like, I built this, um... I, I built, like, what it would take to be a, a DX engineer at WorkOS as, like, a full agent using Py, and it's called Case, and, uh, it- ...

471. 54:59

     uses memory internally, like memory.md files, and it works across all of our open source repos. Uh, so it knows like React, and React Router, and Next, and Tan Stack Start, and all of those.

472. 55:09

     And so then it has like general memory files and then like framework specific memory files, and it goes in and prunes those and updates by doing, like as part of its flow, doing a retrospective at the end, and analyzing its own performance, and then saying, "Oh, I spun, I spun in a circle a bit for this.

473. 55:25

     I could've, like once I got to there, like I can just save that to memory so I know like this is the command I run next time to get the information I need," and it just keeps track of that.

474. 55:34

     I haven't built in like the full dreaming thing where it prunes that yet, but, um, I, I'm experimenting with it.

475. 55:40

     Yeah, and also I, I wanna play with the Obsidian connector more because I think that would be super powerful. I, I had a habit in the past of using Obsidian and just making a daily to-do with just the date as the title.

476. 55:50

     And then so I think writing to and reading from those vaults so that you could imagine saying, "Look back over the last week." Last week gets translated into what are those actual dates, it fetches those files directly, right?

477. 56:00

     And then it can also write consolidated memories. It's also worthwhile playing with things like OpenClaw, which I've, I've done, because that memory system that it ships with was surprisingly good.

478. 56:10

     Better than a lot of like stock Cloud or OpenAI's stuff. And so seeing how it does that with like daily journal.mds and then the consolidated memory, which I think the dream stuff is kind of pointing towards like consolidating memory over time.

479. 56:23

     Yep.

480. 56:23

     Um, but a lot of times the crazy thing about this is like the answer is one turn request with skill builder is the fastest way.

481. 56:31

     Pointing it to-

482. 56:31

     Yeah.

483. 56:32

     This is a good repo.

484. 56:32

     Yeah.

485. 56:33

     Yeah.

486. 56:33

     100%, yep.

487. 56:35

     Um, so we're gonna jump into the next, uh, piece of, of work on your side, and that's adding phases, uh, and confidence scoring to it. So adding progressive disclosure, uh, adding a confidence score, telling it like, "How confident are you in, in this?"

488. 56:48

     Or like, like, uh, we, we've got some examples, uh, of that potentially, like, uh, you know... What's a good example?

489. 56:57

     Um, how confident are you in this, you know, you installed AuthKit correctly?

490. 57:03

     Yeah, but I mean like for repo roast.

491. 57:05

     Oh, for repo roast?

492. 57:06

     Uh, you, you know, you gave me a bad score on, I don't know, Git commits. Why is that? Like-

493. 57:13

     Okay

494. 57:13

     ... have it dive down deeper than just-

495. 57:15

     Yeah

496. 57:15

     ... oh, it's-

497. 57:16

     This is our pattern of how we use Git commits. We always have our messages like this.

498. 57:19

     Yeah.

499. 57:19

     We're following these conventions, so then based on that, what's your confidence that this is correct to our repo?

500. 57:24

     So for example, you might use, uh, conventional commits at your work.

501. 57:27

     Yeah.

502. 57:27

     And if you find commits that aren't like that or you find a bunch of merge, uh, commits in there, for shame. Uh, but yeah, like different things like, like that you can, you can add, uh, specifics to and have that as be, as a, um, a progressively disclosed rubric that it can follow for those things.

503. 57:46

     Yeah. A quick housekeeping thing in case, uh, for any reason you, you're behind or feel behind, uh, you can run setup.sh and then checkpoint two to get to the same spot that we're at now.

504. 57:54

     Yep. Great.

505. 57:56

     Um, and then yeah, any other questions feel free to just shout out and I'll run you a mic.

506. 58:03

     We'll spend about five minutes here, uh, and then we'll move into the next section just to make sure we have enough time. [clears throat]

507. 58:14

     Do you wanna talk about, um, any of these topics, Zack?

508. 58:20

     I can talk about when confidence... When, when, um, yeah, confidence scoring saved us.

509. 58:25

     Yeah. What's that?

510. 58:25

     That, uh, was when we were working on the, um... When confidence scoring saved us.

511. 58:34

     The, uh, well, that was kind of built into the evals, uh, that we wrote. Like Cloud ships with a whole eval framework now that you can use. Uh, and it'll like spin up a GUI for you, or like a...

512. 58:45

     It'll create an HTML report, and you can see like before and after and all of this insight into how your, uh, skills are running, uh, and whether they're actually like improving Claude or making Claude worse at the task.

513. 58:57

     Um, but before that existed, uh, I was writing my own to do that, and, uh, it was all based on, on that. And so like let me, let me bring up the, um,

514. 59:09

     ideation skill. And I'll just say... Let's see. We'll go to the CLI,

515. 59:16

     and I'll say...

516. 59:21

     So for context, this is our WorkOS CLI-

517. 59:23

     Yep

518. 59:23

     ... that we're building in the, you know.

519. 59:25

     I'm on the main branch of that. I use Worktrees for that. Um, what's a feature that we wanna add? I wanna add a fun slash buddy command similar to how Claude Code shipped that for April Fool's Day.

520. 59:40

     I used, uh, a tool called WhisperFlow, uh, to go full Wally and not even type anymore. Uh, and I just press a button. This is how I code now.

521. 59:50

     Um-

522. 59:52

     Do you, do, do you pair that with the Claude's voice mode? Uh, yeah, I do. I've been on WhisperFlow for maybe a year now, and the thing I like about it is that it can, uh, input anywhere on, on, uh, Mac.

523. 1:00:02

     So, you know, if you're in, uh, some funky old like website in Chrome, it works there, it works in Safari, works on any app that you've got. As long as you can focus a cursor there, you can insert text there, and it's also fine-tuned towards like technical terms.

524. 1:00:16

     So you can say @userauthentication.ts and it'll come out correctly. Um, you can reference files, et cetera. So it's great. I m- I m- imagine that more and more of the tools are gonna get their own native voice, uh, over, and over time that's gonna become like a dominant like interface.

525. 1:00:31

     But right now WhisperFlow is like a pretty sweet experience. Yeah.

526. 1:00:38

     I turned on fast mode so it'll go faster. Um, yeah. It also does cool things like, uh, you can sa- say like when you're dictating into Slack, uh, be more casual.

527. 1:00:48

     When you're dictating into an email, be more formal.

528. 1:00:50

     Yeah.

529. 1:00:50

     And it will kind of-

530. 1:00:52

     It's sort of context aware-

531. 1:00:53

     Yeah

532. 1:00:53

     ... in the formatting that it'll put out. Like you can say it, it knows you're in Gmail or it knows that you're like writing code, you know, or requesting code.

533. 1:01:01

     So this is an example of the ideation skill. I gave it that, that simple command, and now it's saying, like, "Oh, what do you... Like, I don't fully understand what you mean.

534. 1:01:09

     Uh, what kind of fun are you looking for?" I'll say, uh, a visual gag. [laughs]

535. 1:01:19

     Uh, ASCII art gallery. Sure. Hidden Easter egg. Uh, yeah, we'll go listed but subtle. So, like, I b- I give it one sentence-

536. 1:01:31

     Right

537. 1:01:31

     ... and it's like, "Well, what do you mean by that?" And it's, like, pulling all of that out, out of me.

538. 1:01:35

     So, but there's the value in thinking. It's like, you know, the same way that a good engineer in a whiteboarding session would kind of draw the same stuff out of you.

539. 1:01:40

     Yep. And so right there, it did this confidence score. It's based on the problem clarity, it has a 20. Goal definition, 18. Success criteria, it doesn't really know what I'm asking for, so that's the lowest one.

540. 1:01:52

     Uh, scope boundaries and then consistency. So those all add up to 100, and I got a score of a 90 out of 100. So it doesn't, it's not gonna just be like, "Okay, I know what you want."

541. 1:02:00

     It's going to ask me, uh, some more things. Like, oh, we'll do that, and we'll just have minimal... Uh, I'll say zero config. I just want it to go fast.

542. 1:02:10

     And so now I'm at 96 out of 100, so it understands what I want, and now it's going to write a, um, a contract for me to read. I read and review the contract, and then it's going to build from there these phases that I can execute or these specs that I can execute in phases, uh, and

543. 1:02:28

     then go from there, uh, so that I can clear the context for each one and have, like, a fresh context going.

544. 1:02:33

     Yeah. Uh, the way I would say that is, like, is the math airtight? No. Uh, does it matter? No, because the value's in the iterative loop of, like, clarifying and, and clarifying your own thinking by, by responding.

545. 1:02:43

     Yeah. Oops. And so there's the contract that it's, it's loading.

546. 1:02:52

     Uh, and it tells, like, what success criteria means, the scope boundaries, what's in scope, what's specifically out of scope, any future considerations, how we plan to execute it. This is an easy one, just a single phase.

547. 1:03:02

     Uh, and so it's going to just create that spec for me, which it did here, and then I can run this, uh, and go. And so it was all gated on that, that confidence score.

548. 1:03:13

     Cool.

549. 1:03:14

     All right. Um, you wanna jump into-

550. 1:03:17

     Yeah, let's do it.

551. 1:03:18

     All right. Uh, we'll, we're gonna skip ahead into, um, the next section, and we'll have one more, one more thing, and we'll do some sharing, uh, after that one.

552. 1:03:28

     So kind of moving beyond the editor, we consider... We, we thought about this, and we're like, does that title make sense? Uh, skills beyond the editor, 'cause we're not really in an editor.

553. 1:03:36

     But, like, for us, we kind of are. Like, we don't open... I don't open Neovim nearly as much as I used to, uh, so I've lost my identity a little bit.

554. 1:03:45

     But, um, yeah, these skills, they really do work in a lot of different places. Um, another thing that you can do is, like, you can level up your skills in a number of different ways.

555. 1:03:56

     Uh, so, like, for this, uh, Repo Roast, for example, you could have, like, oh, I wanna know who the bus factor people are. So use, like, uh, get short log to understand who's committing the most, who's committing the most in specific sections of the, the code base, uh, and, you know, list out what the bus factor is,

556. 1:04:12

     uh, and how vulnerable we are to that. Commit crimes. Uh, this would be people who just have bad commit messages. It's so easy. You just tell Claude, commit it, and go. [laughs]

557. 1:04:22

     Uh, zombie branches. You could have, you know, all... List out all of the branches that never went anywhere, uh, or that are still hanging around. Uh, who is committing at 3:00 a.m.?

558. 1:04:32

     Who's, who's up the latest, uh, working and making us all look bad? And then this one is definitely something that you should d- you should add, and that's-

559. 1:04:41

     Read me

560. 1:04:41

     ... is my read me... Yeah.

561. 1:04:42

     Yeah.

562. 1:04:42

     Is my read me real? [laughs] Does it r- explain or describe real things?

563. 1:04:50

     Uh, yeah. And so again, the reason that this is so powerful is that it's no longer specific to any foundational model provider, right? Uh, you can define these skills, and then you can use them locally in Claude Code, but you can share them with your team, as we talked about with, you know, a Git-based, you know, plugin

564. 1:05:04

     architecture. But now you can also put them in Claude Desktop and Web, as we talked about with the recruiting team. Folks that identify as completely non-technical are loading, uh, specific skills and running them in their own sessions.

565. 1:05:15

     And sharing them.

566. 1:05:15

     And sharing them, right? And then now as we're finding, like, agent harnessing is, uh, harness is becoming more and more, um, relevant, and so things like, uh, Py, um, which is what OpenClaude runs under the hood, uh, you can load them there as well.

567. 1:05:28

     So it's, it's, uh, the value is really in, like, defining the discrete work block and then figuring out exactly which tweaks make it the most effective description of getting that work done and then, you know, sharing it with your friends and putting it on different boxes, um, without having to do much more than authoring some markdown and

568. 1:05:45

     possibly some scripts.

569. 1:05:47

     Yep. And skills, i- if you took a skill file, uh, like you took Repo Roast with that skill.md and any scripts or references and all that, but you took that folder s- uh, Repo Roast, and you zipped it, you'll get a .zip file back, right?

570. 1:06:01

     Rename that from .zip to .skill, and now a non-technical teammate can drag that into Claude Desktop and use that skill, and that's just how they're shipped. That's, uh, a really easy way to-

571. 1:06:12

     Yeah

572. 1:06:12

     ... to share them. Not a really easy way to version them. [laughs]

573. 1:06:14

     There's still, there's still pain too around, like, how do you handle-

574. 1:06:16

     Yep

575. 1:06:17

     ... sensitive, uh, you know, credentials in that case. Like, you, you don't do it that way. Please don't put it in the zip file.

576. 1:06:22

     For sure.

577. 1:06:22

     Um, but, you know, it's evolving, so...

578. 1:06:24

     But you can also use those marketplaces. Like, the Claude Marketplace works in, uh, Claude Desktop as well, so that's a, an easy way to, to share skills, um, if they are applicable to, like, non-coding workflows.

579. 1:06:36

     Yep. For sure.

580. 1:06:39

     And, uh, so some of the... Like, we've talked about this, but, like, one of the things that I really wanted to drive home is, like- With the WorkOS CLI, this is a, it, it's like a generic CLI that you can use, uh, to do, like, WorkOS commands in it, but, like, its flagship feature is this ability to

581. 1:06:56

     just run install. So if you have a project that doesn't have Auth in it, or you have, uh, like, other Auth in there that's not WorkOS, uh, you can just run WorkOS install in there, and it's going to politely remove the other Auth that you might have in there, uh, o- and then add in based on what

582. 1:07:12

     you are using, like if you're using Next.js or TanStack Start or whatever, it's going to figure that out and load that in there for you. And the CLI is using the Claude Agent SDK, which is like a programat- programmatic Claude Code that you can ship, that I can ship in the CLI.

583. 1:07:27

     Uh, and the smarts of that, all of the brains are actually Skills that are in the WorkOS Skills Directory, so it knows all about that. And the reason we did that is so that we just had, like, the, you know, two birds with one stone.

584. 1:07:39

     We have, we build the Skill, and we make it good, and then we prove that it's good by having the, the CLI run it. And the beauty of the CLI is, like, it's an easy command.

585. 1:07:47

     You just do NPX WorkOS install, uh, and we're, like, um, proxying all of the commands to Claude so that it hits our API token. Uh, and, and so it's an easy way to just, like, say, "Here's a zero friction way to get set up with it."

586. 1:08:02

     It'll even create, like, a WorkOS account for you, and you can go back later and claim it, so it's like five minutes and you're, you're set up. And all of that is entirely Skills driven.

587. 1:08:12

     Yeah. A- another place we're seeing, like, high leverage with this is, imagine blog writing. Uh, like, lots of folks on the team, as it's growing, like, wanna wr- write blog posts in a uniform way, but they don't know exactly how our CMS works, exactly the, the tone or format, and, like, the conventions that we use.

588. 1:08:26

     And that's the type of thing that you used to put in a Notion doc and then hope that you could inject it in someone's Slack and, like, force them to read it before they write something.

589. 1:08:33

     It's just easier to define that as a Skill so that they can interact with it and then get to 80% of that artifact without having to consult somebody else essentially.

590. 1:08:41

     Mm-hmm.

591. 1:08:42

     Um, code review, image generation. With image generation, too, you can also put additional parameters there to get, like, specific styles as well. Um, CI pipelines, and as I mentioned earlier in the talk, like, once, uh, Nick had published, uh, the, you know, public repo of, of WorkOS Skills, uh, the RAG pipeline was able to just start loading

592. 1:08:59

     them all as agentic tool calls, and performance on all those queries just jumped, uh, over just, you know, flatly chunking all documents and putting them in a vector database, for example.

593. 1:09:09

     And you saw the giant lobster outside when you came in, right? Like, that's all can be Skills-based as well.

594. 1:09:14

     Yeah.

595. 1:09:14

     So it's, Skills are just this uniform way that transcends the, the Claude Code or the Codex, uh, and is something that you can load anywhere at any technical skill level.

596. 1:09:24

     Yep.

597. 1:09:24

     So it's really easy. Uh, we talked about, uh, evals, like measuring the stuff matters. Uh, w- with the, the Skills, like specifically with the Next.js installer Skill, I actually found out through my evals that I was making things worse because I was overly prescribing what to do with Next.js, and Claude Code was just inherently good at working

598. 1:09:44

     with Next.js, and I was making it worse by being too dogmatic about what I wanted it to do. And it led to like a 30% drop, I think, in, like, overall accuracy based on these numbers I made up.

599. 1:09:56

     Uh, but I was able to use the, that, and I, I kinda think of evals a lot of, in a lot of ways like my Apple Watch. Uh, it tells me, like, my heart rate and, you know, how, how many calories I'm burning throughout the day.

600. 1:10:08

     Is it accurate? No, of course not. But it gives me a general, like, baseline of like, "Ah, I am more active today than I was yesterday," uh, and I can kind of use that to gauge where I go forward.

601. 1:10:18

     Is it accurate and would I, like, base my, my life on it? No. But it's a, a general, like, vector that I can, I can look at and see whether I'm improving or, uh, making things worse.

602. 1:10:29

     Yep. [clears throat]

603. 1:10:31

     So some Skills in the wild. Um, Zack, you've, you've made a couple of Skills, uh, that are, these are specifically, like, not, uh, code related.

604. 1:10:38

     Right.

605. 1:10:39

     Uh, but they, they're pretty impressive.

606. 1:10:41

     Yeah, so this is what I was talking about earlier just to show, um, what I am the most excited about is, like, taking what seems like an incredibly complex workflow and then just making it available as a Skill.

607. 1:10:50

     So this is, uh, as, this is an example where I have a, a Slack avatar that I built m- um, I had generated for me, like, months ago, and I just handed it to this animation Skill and I said, "Animate this in the most obvious way possible."

608. 1:11:03

     We'll see if that's actually obvious. [laughs] So taking a giant ball of energy and grimacing at it as one does. Um, but the point is, that was a single text prompt, uh, of like, "Make this person look like they're in Fallout holding a ball of energy," and then a- animating it.

609. 1:11:22

     This one, uh, this is the same exact Skill, so same Markdown file and Pyth- two Python scripts saying, you know, uh, the prompt was, "Child running through a field."

610. 1:11:34

     And there's also sound with this because it's, um, hitting the VO API. So again, the f- at first, uh, you know, Claude reads the Markdown Skill, says, "Okay, I understand what this is.

611. 1:11:42

     It's a, a sequence of two API calls I'm gonna make." The first API call is the user's prompt to make the static image. The second API call is the output of that, the static image, and then a new prompt that I write saying, "Animate this in the most obvious way possible," hitting VO with, uh, the VO API

612. 1:11:59

     with that, and then getting back an animation. But, you know, again, that's like 30 per- 30 seconds of generation time. And so I used this exact same workflow to, to do, like, all of the interstitial scenes in a, in a film recently.

613. 1:12:14

     Uh, and another example, uh, I mentioned this earlier, but the Remotion Skill. Uh, I have, I'm terrible at video editing. I don't know anything about it. Uh, but when I was working on the WorkOS CLI, I thought, "Oh, it'd be kinda cool to make, like, a fun video that I could use on Twitter to, like, demonstrate it

614. 1:12:28

     or, or talk about it." And so, uh, somebody was mentioning Remotion, and I just asked it to make this, and it put it together pretty much like this. Like, I asked it to use our, our actual logo rather than some made-up one.

615. 1:12:40

     Uh, but it even, like, understood, like, the output of, of the CLI and put all of this together into a demoable video, uh, that- ... showcases what it can do, and I didn't have to do that, uh, at all, and I, I looked super impressive without [laughs] knowing anything about video.

616. 1:12:57

     Yeah.

617. 1:12:57

     And also, like, the skill, when I said, "Do this," it loaded up a, like, localhost:3000 in my browser that was a full ReMotion video editor. And so I could see it playing on a loop in there, and it was, like, doing things, and I'm like, "Oh, you didn't use our actual logo.

618. 1:13:11

     Go use that." And I just, like, told Claude to do that, and it just updated, like, in real time. It was, it was so cool.

619. 1:13:16

     Yeah. So imagine, like, hooking this into your GitHub CI/CD flow, and then at the end of a big project or every time a milestone gets merged, you auto-update, you know, whatever document, and then even include a demo.

620. 1:13:27

     Um, it c- it can start to get pretty powerful if you orchestrate skills that are well-defined.

621. 1:13:37

     Is this the skill?

622. 1:13:38

     This is, this is exactly how that, that one works under the hood. So you can imagine, like, the one that I showed you that had the two YouTube videos.

623. 1:13:43

     So if it's called animated image, the first one's gonna be gener- generate a minimalist static image, and then, um, take that image and animate it via VO. And y- there's just two scripts.

624. 1:13:53

     There's one to generate an image here, and then there's one to generate the video. Um, but the skill file itself is, like, 30 li- lines of Markdown.

625. 1:14:00

     Yep. And that, uh, that Nano Banana one that I ran earlier, that was just, like, coming up with a, a creative enough prompt, like, taking the idea that I had, like, fleshing out the prompt, and then it passed it to a TypeScript file that called the Nano Banana API and got the image back.

626. 1:14:15

     So, uh, that skill is just basically like a, a simple LLM wrapper around this, uh, a- around a s- a TypeScript script that uses their API to, to go do that.

627. 1:14:26

     Yeah. So it's also, like, just broadly applicable to workflows. It's not just a dev thing, right? You can imagine if sales has a very specific way they have to reach out to people, or there's always, like, a type of report that you're generating for customers or prospects or whatever, um, all, all of this is, like, excellent, uh,

628. 1:14:39

     for use with skills.

629. 1:14:41

     So did anyone, uh, have a, um, a skill, a Repo Roast skill that they wanna share?

630. 1:14:47

     Yeah.

631. 1:14:49

     The Amy and Wolf from Ravenwolf one.

632. 1:14:49

     Okay.

633. 1:14:53

     Yeah, the Amy and Wolf from Ravenwolf skill, try that. You have to see the results.

634. 1:14:59

     Uh, which one is it?

635. 1:15:00

     Uh, number two. Or I uploaded another one. Number six is a newest one. Use the-

636. 1:15:04

     Okay

637. 1:15:05

     ... newest.

638. 1:15:06

     I'll do number six. Huh. All right, so while that's running... Oh, no, we ran the wrong one.

639. 1:15:21

     There we go. While that's running, uh, let's go look at it.

640. 1:15:29

     Oh, nice. Okay.

641. 1:15:30

     Ruthless honesty. I love it. [laughs]

642. 1:15:33

     Brutally honest. [laughs] With a heart of gold. [laughs] Awesome. I love the context, lots of, uh-

643. 1:15:42

     Thick files, very nice

644. 1:15:43

     ... yeah

645. 1:15:44

     Excellent

646. 1:15:44

     Constraints. And here, uh, the audience detection. You told it to load audience-guide.md. Here's that progressive disclosure about that.

647. 1:15:56

     This also just helps to keep your Markdown files manageable.

648. 1:15:59

     This is a 10 out of 10 skill.

649. 1:16:00

     Yeah.

650. 1:16:01

     Very nice.

651. 1:16:02

     All the credit goes to my assistant. [laughs]

652. 1:16:06

     This is awesome. All right, let's see if it gave us anything.

653. 1:16:12

     So it's, it's r- uh, grading the, the workshop itself. Um-

654. 1:16:18

     Six out of 10

655. 1:16:18

     ... six out of 10.

656. 1:16:18

     Brutal. [laughs] I feel that. [laughs] I thought we had-

657. 1:16:23

     Okay. Fair. Fair

658. 1:16:23

     ... I thought we had something going, but okay. [laughs] [laughs]

659. 1:16:26

     Ooh. [laughs]

660. 1:16:29

     Rough.

661. 1:16:29

     Hopefully you give us a little bit more of a, of a grade than that. [laughs]

662. 1:16:34

     That's awesome. Uh, some critical things.

663. 1:16:36

     Test suite isn't on fire 'cause it doesn't exist.

664. 1:16:38

     Yeah.

665. 1:16:38

     That's great. [laughs] [laughs]

666. 1:16:42

     1,200 lines of monolith. Yeah. Yeah.

667. 1:16:46

     Git identity crisis, Zach is two people. That's how it feels too. [laughs]

668. 1:16:50

     Yes. [laughs]

669. 1:16:51

     Hard-coded secret. That's okay. It's not really a secret.

670. 1:16:55

     That's awesome.

671. 1:16:56

     Love it.

672. 1:16:58

     Super cool. All right.

673. 1:16:59

     Yeah.

674. 1:16:59

     We got, uh, three minutes left if there's any other questions or, um, anyone else wanna share a skill?

675. 1:17:10

     So this is a skill that you can use, but more importantly, it's techniques that you can take and use to build your skills, uh, and build them up in different ways.

676. 1:17:18

     There's a lot more advanced topics that we can go to, go into as well. Uh, we mentioned, um, like sub-agents for example. Sub-agents is a great way to extend those skills without bloating the context and having it kinda do one-off things and then, uh, and then exiting.

677. 1:17:33

     Um, and the... Like, to take this to the next level, I really recommend, like, having Claude's own skill creator skill-

678. 1:17:40

     Yep

679. 1:17:40

     ... installed, uh, because you can just say, "Hey, I have this skill. Is it any good?" And it'll give you pointers. Or you can say, "Run some evals on it," and it'll run, like, a full eval test suite on it, uh, and tell you, "Yeah, it's good," or, "No, it's bad."

680. 1:17:52

     Yep.

681. 1:17:53

     And, uh, and can go from there. Uh, and then like Zach was saying, like, reflect on the transcripts. Reflect on how you're actually using the skills, and you can use that as insight to see how to improve the skills and the execution of those skills.

682. 1:18:06

     Uh, like for example, if somebody kicks off a skill and it's always asking questions about, uh, you know, a specific thing, maybe that's something that you can provide ahead of time.

683. 1:18:13

     Or if you see it, like, oh, it's going and doing, like, 10 tool calls, maybe you could, like, m- condense it down to one or two tool calls and pre-provide that information so it doesn't have to do that each time.

684. 1:18:23

     Yep. The sc- uh, plus one recommendation on the internal skill creator. And then the other thing it kind of, um, sugges- I mean, suggests to you to do over time is to think about the way you manage your context, even stuff that you used to think of as disposable, right?

685. 1:18:36

     Mm-hmm.

686. 1:18:36

     So, like, in the pre-LLM era we might have devved real hard at the keyboard all week, as I used to do, and then finally on Saturday, like, wipe it all away so that I can GitHub for myself.

687. 1:18:45

     Like, now all of that context is gold. Like, the conversation, especially what failed, especially what didn't go well, especially what was frustrating because now all of that is very rich context for a skill creator or refiner to mine and then build you a bespoke tool that's gonna solve that problem smoothly next week.

688. 1:19:02

     Yeah.

689. 1:19:02

     So-

690. 1:19:05

     And you can also, like, think of skills, like you could use that progressive disclosure to, like, disclose things to different audiences. So for example, you could, depending on who's running the skill, you could say like, um, get config user.email and, and figure out who this, the user is.

691. 1:19:19

     Uh, or you could do things like, oh, how many commits does this user have in there? They have 10,000 commits in here? Okay, we can really roast with them.

692. 1:19:26

     But this other person who has four commits, they're probably a new hire, maybe go a little gentler on them.

693. 1:19:30

     Be nicer. Welcome-

694. 1:19:30

     Don't scare them away from this project that they just spent-

695. 1:19:32

     Welcome to stay with me too. Question, yes.

696. 1:19:34

     Very quick, Zack, could you take me through again, uh, you were saying about the context switching, you'd somehow hooked up Claude with Slack and Linear?

697. 1:19:41

     Yeah.

698. 1:19:41

     So it sounded almost like it's constantly being able to read what Slack's doing.

699. 1:19:47

     Absolutely. I have it, um-

700. 1:19:48

     Is this a Claude CoWork or Cl- We, we use Cursor, so I don't know if we have the same.

701. 1:19:51

     Gotcha. Yeah. I'm using Claude Code now. It's possible to do it in Claude Code and, and Claude, uh, Desktop, but essentially I just have the connector in Slack, so I say, uh, I had to do GitHub or just had to do OAuth with Slack, and then it can read my Slack messages.

702. 1:20:03

     You can now run the loop command, at least in Claude Code, to have it, like, do that every 15 minutes if you want. And then you say in the prompt, "If there is not already a correlative Linear ticket, make a new one for me.

703. 1:20:15

     If there is one, and there's additional asks on this, you know, request, update Linear." And then by the way, you have a second terminal tab that's looping against your Linear state.

704. 1:20:24

     Kathleen works at WorkOS. E- earmuffs, Kathleen, I'm really working really hard. [laughs] Uh, you have the second one that's looping and looking at your Linear tasks and then, like, doing work for you, essentially.

705. 1:20:34

     But the, the main point was just that, um, yeah. Oh, sorry, we're at time. The main point was just, like, automate those loops. So that's our time. Thanks so much, guys.

706. 1:20:41

     Uh, thanks for being an awesome audience. Thanks for all the great questions. Really appreciate it.

707. 1:20:45

     Thank you. [audience applauding] [upbeat electronic music]