Skill Issue: How We Used AI to Make Agents Actually Good at Supabase

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone.

2. 0:15

   Uh, is everyone excited for the, the conference?

3. 0:18

   Yes.

4. 0:19

   Awesome. We got a full house here. Uh, it's very... I'm very glad to be here, very honored, uh, to be giving, uh, the opening or one of the opening workshops, uh, today.

5. 0:28

   If you've noticed already, uh, the title is slightly different, uh, from what we have in, in the schedule. I've basically done a rebrand, but the theme for the workshop will remain the same.

6. 0:39

   Uh, we went from skill issue to level up your skills. I've decided to move the skill issue title to, to the keynote that I'm giving, uh, tomorrow. Uh, if y- you'll have time to, to know more about what the keynote is going to be about.

7. 0:52

   Uh, but mainly, uh, it's, um, this workshop, uh, basically this workshop is what I've been doing in the last two months, uh, at Supabase, writing our own skills. And tomorrow I'm going to present how we put this actually into production and the lessons we've learned.

8. 1:09

   Um, so for everyone who's been paying, um, closer attention, you probably noticed that I'm, I'm running this slide deck on localhost. Some of you have already noticed. Uh, this is no coincidence at all.

9. 1:22

   I bas- uh, I essentially vibe coded the presentation, so if you see something off, uh, it was not my fault, it was Claude. Uh, but for, for you to...

10. 1:33

    If you don't believe me, you can see that, uh, or you cannot do, or you, you'll have to be a very, uh, Google Slides guru to have, uh, dark mode enabled.

11. 1:41

    So honestly, I like this layout better, so I think we're going with dark mode here. Um, if there's any, uh, light mode fans, uh, or there are the majority of the room it's light mode, I'm happy to switch back.

12. 1:54

    Uh, but for now, let's go with, uh, with this one. So to, to do a little presentation of myself before starting the workshop, uh, my name is Pedro. I'm, uh, uh, I'm from Portugal, uh, Lisbon, and I work at Supabase as an AI tooling engineer.

13. 2:10

    Essentially, my day-to-day is to, uh, think of how we can make the, the Supabase the most agentic friendly as possible, uh, and, um, improve the agent experience. So we-- You've probably heard about develop, uh, development experience, the DX.

14. 2:28

    Uh, we are more focused on the AX, which is the same thing, but for agents. Uh, in this workshop, we're going to, um, talk a bit about skills, uh, 'cause essentially, um, that's how we've been, uh, improving the, the performance of agents around, uh, a product like Supabase, uh, or a company like Supabase has multiple products.

15. 2:51

    Um, the secret sauce is, has been basically skills. So we're going to dive into how to write one, um, how to, uh, test it manu-- first manually, and then how to automate the testing with, uh, evaluations.

16. 3:05

    Uh, so to start with, how many of you have heard about skills?

17. 3:11

    All right. So almost anyone-- uh, everyone. Uh, so what I'm going to say, it's probably no news to you. Uh, skills are basically folders, uh, with, uh, instructions, um, and v- uh, files for you to, to run workflows, uh, repeated workflows, or give, uh, custom, um, essentially custom information to your agents, uh, or provides a, a new

18. 3:36

    set of, uh, of tools, let's say, in form of, of scripts. So there's a, a bit of a misconception about skills. Usually, the skill.md, the main file takes the, the, the spotlight.

19. 3:48

    Uh, but skills can actually be more than just the, the main file, right? Uh, so the, the main file is, uh, basically a Markdown file, uh, named skill.md, where the essentially the, uh, the main information about the, the skill lives.

20. 4:04

    Uh, it ha- it is composed by this front matter at the top, uh, which essentially has-- can have multiple fields, but the two, um, required ones are the name, which basically identifies the skill, uh, and then the description, which tells the agent what the skill does.

21. 4:22

    The main, um... What exactly the, the, the, the skills basically bring that, uh, tools like, uh, MCP didn't, uh, was this concept of, uh, progressive disclosure. Uh, progressive disclosure is basically when the agent, uh, or the, all the information about a subject is not loaded, uh, straight to context.

22. 4:45

    Uh, instead, you just load the exact amounts of information that allows the agent to, to choose to load the rest of the information once it actually needs it. So in this case, um, the, the skill.md file is designed like this, so the front matter, uh, will be loaded at first to the, to the context of the agent,

23. 5:05

    not the content of the, the file. Uh, this works as an envelope, so the agent has-- kn-knows from the description what the skill does, uh, and when, uh, it should load the, the rest of...

24. 5:17

    So when should it look for the information inside of the file. Uh, inside this file, you can also reference another files. Uh, usually these other files are either Markdown files or script, Bash, Python, whatever you will, uh, you would like, uh, to, to, to reference.

25. 5:32

    Um, starting on the reference files, you usually put them inside a reference folder, um, and they provide more information. You can think about a, a skill in this format as a book.

26. 5:43

    The skill.md, you can think of it as the index on steroids, because besides of having these links to the other files, you can think of them as, uh, the pages of the book or the other chapters.

27. 5:55

    Uh, you, you'll have, um, custom inform-- you can have custom informations and then also reference it, uh, to, to the other files. The reference files, they have nothing special about them.

28. 6:05

    They're basically like the mar- a normal, regular Markdown file. Uh, you can think of, uh, similar to skill.md file, but, uh, instead of being the main one, it's, uh, the one that, uh, that got referenced.

29. 6:16

    Uh, you can also fun, um, funny enough, you can also reference files inside of reference files, so you can make basically a graph out of a, out of a skill.

30. 6:25

    And for sc- uh, for scripts, I've actually talked about, uh, how the, how MCP and skills differ, uh, from each other, and we are basically comparing apples to, um- Uh, apples to oranges when it comes to MCP and skills.

31. 6:40

    Uh, one of the mi- misconceptions pr- mm, currently probably was already debunked. The debate now is more about MCP versus CLI. But when the, the skills were released, uh, back in, I think it was November or October last year, um, they basically started this debate about, um, well, it's, uh, should we use them instead of MCP because

32. 7:03

    if, if I can run-- if I can provide more information, more context to the, to the agent without actually loading every tool to, uh, to the context like the, like the MCP, um, and I can also have, uh, scripts so I can have actions just like, uh, I have a, on MCP tools, should we use them?

33. 7:20

    And the answer is, uh, you should use both, to be honest. Uh, if you're building anything that it's, uh, uh, an integration, you should use, uh, MCP, right? Um, anything that i-i- if your agent doesn't have access to Bash, you should ye- you should use MCP to integrate to your service.

34. 7:37

    Uh, skills actually just provide more context to your agent, right? And you can define workflows, everything that you would not-- that, that you don't have space to define on, on the MCP tools.

35. 7:47

    Uh, descriptions, you can define them on, on skills. Um, also regarding the comparison-- the, the debate between skills, scripts, and the skill-- uh, the MCP tools, the main difference is that tools don't need, um, an environment to, to run.

36. 8:04

    Uh, the agent ju- can just call a tool, knows how to call a tool, especially if the, uh, the MCP server is remote and the tool will run on, on server side.

37. 8:13

    While the sc- uh, the scripts, uh, well, they basically are loaded into your machine. They run on your local environment, uh, and they're tied to the whatever environment that you have.

38. 8:22

    So if you're running on Linux, they have to be Linux compatible. If you're running on macOS, the same. Windows, I'm not going-- I'm going to even start it about it.

39. 8:31

    Uh, but essentially, the, those are the main differences between the MCP tools and the, the scripts. Uh, hope is everything clear. If you have any doubts, uh, feel free to...

40. 8:42

    I'm going to have a, a little demonstration. This workshop is going to be more, more of a walkthrough than actually code along, but feel free to, to tag in.

41. 8:51

    Um, I have a, a GitHub repo, uh, prepared, so you'll be able to visit it and to explore it. Um, but if you have any doubts i-in any moment of the, of the workshop, uh, feel free to, to interrupt me or to raise your, your hands.

42. 9:08

    So moving to... See this. Exactly. This is-- So I tested this on a smaller screen. It was working. You can see it was web coded. So, uh, how do you test your skills, right?

43. 9:19

    So if this is just a markdown, how you test your markdown files, basically. Um, so to test a, an applica-- uh, a, a piece of code, it's, uh, it's already...

44. 9:30

    It's straightforward, right? We, we already know you have all sort of, uh, um, tests, uh, types. You have unit tests, integration tests, or you can test the whole flow or what we call end-to-end testing.

45. 9:42

    Um, well, e-essentially, when you're testing a markdown file, y-you can basically do exactly the same. You can be as granular if you, uh, as you, you want. Um, but usually since we have an LLM in, in the loop, you'll have something called evaluations.

46. 10:01

    So, uh, for those of you who haven't heard about evaluations or evals for short, um, they essentially, um, are a more, um, a non-deterministic way of testing the output or the behavior of an agent or a model.

47. 10:19

    Uh, you can test both an LLM or an agent, uh, with, with evals. Uh, essentially you'll be the, the most common structure. I'm going to be-- to, to, to present to you at the end, um, a framework for you to test your, your evals, like a very simple one where you can start and I'm going to, uh,

48. 10:37

    dive, uh, deeper on evaluations there. But essentially, um, e-- they usually are, um, made of an input, an expected output, just like a regular test, and in between you can, um, evaluate the steps that the agent took, the reasoning, uh, the tools that it, that it called, uh, which is, uh, normally more interesting and easy to, to

49. 10:59

    evaluate the, than just like, uh, a regex on the exact output since this is non-deterministic. So there's essentially, um, a framework, uh, to, that you can follow to test your skills.

50. 11:13

    Uh, this one was proposed by OpenAI on their system, uh, on their blog post called Systematically Evaluate, uh, Agent Skills. I think they released this back in January or February, so not that long ago, but all this is fairly new, so this is basically prehistory.

51. 11:30

    Um, i-it-- So you start, uh, by defining your metrics, so what you want to evaluate, uh, on your skills. Uh, if you're building a skill for your product, for example, what exactly do you want the skill to, um, to highlight to your, to your agent?

52. 11:45

    It's going to be to, uh, forward it to the documentation. Are you putting some specific instructions, specific workflows? So depending on what you want to, to evaluate, you start this, uh, eval-driven development, uh, so this test-driven development.

53. 12:02

    You start by defining the metrics, what you-- what exactly good, uh, means, uh, when it comes to the skill. Then you create the skill itself, right? So you write the skill.md file.

54. 12:14

    Oops. So you write the, the skill.md file, uh, any scripts, uh, alongside it, uh, the reference files, uh, if you want to. They're all optional. The one you-- The only re- only required is the skill.md file.

55. 12:27

    Uh, and then you, you went, uh, um, you move to the testing part. So you run the evaluations or, or you run it, um, manually. Um- I've, uh, recently heard the, the, the CEO of, of Braintrust, uh, during the podca- uh, a podcast.

56. 12:44

    I don't know, uh, how many of you know Braintrust?

57. 12:47

    Okay. Not, not as much the... not as many as the, um, uh, not as popular as Skills. Uh, but, uh, so Braintrust for tho- those of you who don't know, Braintrust is a platform that allows you to systematically run evals and provide you like, uh, the full picture, uh, of, um, of the, uh, of the, um, the

58. 13:09

    agent behavior during the, the evaluation, uh, scenario, right? Um,

59. 13:16

    just trying to think about, uh, another platform to compare with, but this is fairly new, to be honest. Um, so you can think of it as like an observability tool, uh, to, to check your...

60. 13:27

    the behavior of your, um, of your agent during a specific oc- controlled scenario, uh, which are the evaluations. Um, so you move to, to the testing part. Basically, you run a set of evaluations, uh, scenarios, uh, how you, um...

61. 13:43

    these are defined by the input and expected outputs tools that, that should be called. So basically, how the, you expect your agent to behave. Um, and then, uh, you move to the grading part.

62. 13:55

    So how did the, the agent do? Well, essentially, it's, this is very similar to a testing cycle, right? Uh, but now we instead of having a deterministic output, you can have, um, it's non-deterministic, it's an LLM in between.

63. 14:09

    Uh, but you can still have deterministic parts to evaluate on. And then you iterate basically and repeat. This is the... that's why this, uh, it's, uh, it's a cycle.

64. 14:20

    It's pretty similar to, uh, any of the, the, the test development cycles that we, that we have at the moment.

65. 14:29

    All right. So, uh, jumping straight to what we're going to do, uh, during this workshop. So we're going to write a, a skill. I've prepared, um, a little demonstration app, a demo app.

66. 14:41

    Um, it's going to be, um, a performance review application, uh, with, uh, four, I believe, uh, four employees. Uh, o- one employee, two managers, and one HR, um, representative.

67. 14:57

    Uh, and essentially we're going to... the s- there's some errors, uh, on the database side that we're going to find and fix. Uh, we're going to build a skill to help d- to guide the agents to, to, to fix them.

68. 15:12

    All right? And then at the end, uh, I have, as I said, a framework to test, um, automatically the, the same scenario that we're going to test manually, um, using, using evals.

69. 15:25

    Um, before moving to the demonstration, uh, how many of you have y- uh, have heard about or used Supabase?

70. 15:34

    All right. So almost, almost anyone knows or used Supabase. Uh, I've, I've seen some, some hands down, so, uh, still I'm going to give you a little brief. So, uh, Supabase, it's essentially a backend as a service.

71. 15:46

    Uh, you, you can think of it as the open source version of, uh, of, um, okay, I just can-

72. 15:55

    Firebase.

73. 15:55

    Thank you. Firebase. Uh, I w- uh, only Fireship was coming through, through my mind. [laughs] Sorry. Uh, to Firebase. Uh, and if you don't know Firebase, you're probably living under a rock.

74. 16:04

    Uh, no, but essentially it's the, it's a, it's a backend as a service. You can use it to build any backend as you, as you would like, uh, coming straight out of the, of the box.

75. 16:13

    Uh, we provide a database, um, for, for you to just plug into your application and run on Postgres, one of the most, if not the, the most popular open source solution out there for databases.

76. 16:26

    Uh, you can easily integrate authen- including authentication on your application, uh, running a storage to save files, um,

77. 16:36

    man- many other things, uh, edge functions, we- which are Lambda functions, uh, for those of you who come from the AWS environment, and so forth. So the demo application that I've built was built on top of Supabase, of course.

78. 16:49

    Uh, and so you can follow along, uh, here is the QR code. Uh, he- at the back, can ev- anyone, everyone, uh, scare, uh, scan the, the QR code, or should I make this bigger?

79. 17:02

    Bigger? Right. This. [typing] Okay. Just so everyone can see,

80. 17:19

    I'm basically editing the, the presentation at the moment as we speak. Let's see what Claude's,

81. 17:25

    uh, has to, to offer us. Bigger? [laughs] This is the cool thing of, uh, of web coding your presentations.

82. 17:44

    I really recommend. Uh, y- I probably spent the, the same or pro- or more time than y- than if I j- just, uh, used something like Google Slides, but at least it's more fun.

83. 17:54

    Uh, and Anthropic should be thrilled about it, for sure.

84. 17:59

    All right. Uh, let me know once everyone is, uh, in the... Is that the... You can see the repo. If you cannot see or scan the QR code, uh,

85. 18:13

    I should probably make the, the link bigger as well. [typing]

86. 18:35

    So you asked for a demo, here's the demo on my slides.

87. 18:40

    Uh, so, uh, mainly everyone u- here in th- this room used, uh, Skills, so I probably won't have to, to sell you, uh, the, the, the power of Skills.

88. 18:52

    Uh, but if you, if you're still a bit skeptical about Skills, this whole presentation was... W- without Skills, this whole presentation would be a lot, um, not pleasant, let's say.

89. 19:05

    Uh, be much uglier in a sense. Okay, you should probably see it now.

90. 19:13

    So basically navigates to, uh, to GitHub, [REDACTED:username], which is my, um, nickname, and improve skills workshop AIE Europe. It's a very long name. All right.

91. 19:33

    So is everyone at the, at the, uh, the GitHub re- repo at the moment? Okay. Everyone was... had no trouble. All right, so...

92. 19:44

    Uh, not this one. Uh. Right. So this is the repo that you should be looking at. Uh, essentially, it's, uh... I know it's, it's a big repo, but we'll go...

93. 19:55

    we're going to break it down. Um, actually going to

94. 20:01

    move... [sighs] Okay. Going to, going to move to VS Code.

95. 20:19

    All right. So here we have, uh, we have, uh, uh, two Next.js apps. Actually, the, the slides are also embedded here. Uh, the, the Next.js a- app that, um, that matters, it's inside demo, right?

96. 20:37

    And to give you an insight of what that looks like,

97. 20:42

    it's basically this. So it's a very simple application to... You can see that it's a vibe coded application, to be honest. Uh, the, the layout, it, there's nothing special on it.

98. 20:53

    Uh, you have, as I've described earlier, several, um, employees of this, uh, fictional company. Um, and you can think of it like a, an internet or a performance review application where, uh, you have all the information.

99. 21:08

    As an HR, um, employee, you have all the informations, uh, all the information about the, um, about the, the other em- uh, employees of the, the company. Uh, and you can change, for the sake of the presentation, you can change between,

100. 21:24

     uh, users, right? So what we're going to do, uh, is first, without a skill, we're going to try to implement a new, um, a new, a new view, uh, here.

101. 21:36

     We're going to implement the reports view. Essentially, this, uh, reports, uh, part of the, um, of the application is going to be, uh, is going to show, uh, both the salary, uh, and the, the average rating for the performance review of each department.

102. 21:54

     So, uh, so HR c- can be, um, can know what's, uh, uh, can have, like, an overview o- of the, of the whole company. Um, so before we start the vibe codes, 'cause, um, n- during these prese- these workshops, no one actually writes code anymore, uh, so of course, I'm going to vibe code it.

103. 22:17

     We're going to just break this, um, uh, this application down. So if we navigate to the dashboard, so nothing special to, to see. You have... It's basically the, the, the page, the first page, the main page that you, that you've seen, and then you have here the, the reports where we should

104. 22:42

     have, uh, yeah. Where you should have this

105. 22:49

     set view exists that is going to, um, uh... So I've, I prepared the back end. We're just going to, to, to create on the back end the view as a SQL view on the, the back end, on the, on the database, and then we should be able to, to see,

106. 23:05

     to see it on, on the application. So I've prepared, uh, pa, pa, pa, pa, where is it? Yeah. So

107. 23:15

     I've prepared the prompt, and we're going to

108. 23:24

     live test it. Fingers crossed that this work. Right. Uh, first let me navigate to, uh,

109. 23:35

     uh, pa, pa, pa, pa, pa, pa, pa, pa, pa. It's app.

110. 23:43

     Okay. Let's see. We have more control. All right. So essentially, or the ones in the back, uh, I'm just ask, uh, I'm going to ask Claude's to create a department stats view, uh, that shows the head count and the average salary broken down by department.

111. 24:02

     All right? So for, uh, HR to have a full overview o- of what's going on in the, the company. So we're going to hit, hit the prompt

112. 24:11

     and wait it, wait to see what it, uh, it should come up with. Uh,

113. 24:17

     right. Um, forgot about this part. I have, um, this MCP server configured. I- if you have, uh, you actually have... Let me take

114. 24:31

     I totally jumped the, the README. Uh, if you're follow along... Sorry about this. If you follow along, if you're following along, you can follow the setup guide, uh, to, to get a, to get your application started locally.

115. 24:45

     Uh, it's essentially is going to install the, the dependence-- clone the repo, install the dependencies, start locally your, your Supabase project. You don't have to have the CLI installed, we'll be using npx to start to run it as a, um, as a binary.

116. 24:59

     Um, just reset the, the database state so you start from, from scratch with the seeded data, uh, and then just run, uh, the, the app, uh,

117. 25:10

     as, uh, as the running npm run dev. Uh, should be, uh, available on localhost three thousand slash dashboard. Um,

118. 25:19

     you also have-- you'll have this mcp.json, um, file prepared. This essentially is pointing to the MCP server that we, we Supabase enable, uh, for local projects. No authentication required, so your agent should be able to just load it, uh, on demand.

119. 25:38

     This, um, uh, this MCP server expose, um, a set of tools. It's, um... I don't know how many of you have used the, the Supabase MCP server, but we-- I think we are currently have something along twenty, twenty something, twenty-nine tools, I believe, for the production one.

120. 25:57

     Uh, this one is, uh, a smaller version. Uh, it has twenty tools, but you can basically perform essentially almost a-a-any, anything that the, the, the one, uh, to connect to a remote project does.

121. 26:10

     Uh, basically lists the, the tables that you have, executes SQL, uh, straight on your database, apply migration, uh, and run the database, uh, advisors, and so forth. So essentially what it, what it, it started to do was to list my tables.

122. 26:26

     So I've asked for review. It's going to review the, the schema that I already have implemented. So I'll let you...

123. 26:35

     And now it's going to run the apply migration, uh, tool to create the view. So it's basically doing a schema change on my database, and it's going to create, uh, the, the view.

124. 26:47

     If we inspect the view, we're basically creating, um, create or replace a view, a department stats, the name that we gave, uh, and we're, uh, by, um, fetching all the information, uh, from,

125. 27:02

     I think department exact- uh, or from... No, from profiles, exactly, and group by department. Okay. Oh.

126. 27:12

     Made a mistake. It's going to try again.

127. 27:17

     Okay. It's going to test it. It's actually something that I really like about... All right. And here's the, here's our, uh, view on the database. So, uh, we currently have it on the database.

128. 27:30

     Let's see if that's also enables on the app. Hmm.

129. 27:38

     Okay, it's not. Then let's quickly, uh, I've [whispering] created a SQL view and what's the name I gave?

130. 27:52

     This is the... This is essentially the, the problem with live views,

131. 27:58

     is it usually doesn't go well at first try. Uh, with the...

132. 28:06

     I'm going to repeat the, the... [keyboard clacking]

133. 28:28

     And let's see if it, if it implements. If not, we can just run the, the SQL query for you to, to see as, uh, different users, uh, to, for you to see if everything is working, um, accordingly.

134. 28:39

     So for now, it's going to implement on the Next.js application, so we can have a nice interface to, to, to check the results. Um,

135. 28:50

     let me enable everything. Wait, let me just put on auto mode so I can continue to talk. So essentially the, the agent created the, the view,

136. 29:00

     test it, said everything is working accordingly. Uh, we should not ha-- The, the, the app was, um, the, the feature was implemented all good. Um, but we're actually going to see if it's actually-- if, if everything it's, it's good or not.

137. 29:18

     Um, so let's give it a... Let's give him some space, uh, not to pressure, not to pressure it, uh, to create the,

138. 29:27

     the, the feature. Let's, uh, just wait a bit more time.

139. 29:35

     In the meantime, if you're following along, you can also play with it, uh, change the, the layout.

140. 29:46

     I'm actually using the... using Claude Code. I don't know. Just doing the, a brief, um, survey here in the, during the workshop, how many of you are using Claude Code as well?

141. 29:57

     Oh, fairly almost everyone. Okay. Um, how many of you are using Cursor?

142. 30:06

     With Claude Code or with the plugin or... Okay. Yeah, at least one person. [chuckles]

143. 30:12

     Uh, we're going to have, uh, some, um, Cursor folks here, I think from Anthropic as well. Uh, OpenAI, uh, is going to be here as w- um, Gemini, of course, Google DeepMind is sponsoring the event.

144. 30:25

     So, um, we're basically going to have the whole gang here. Uh, okay. So we should be-

145. 30:35

     I'm trusting his word. All right? So it says that, uh, we should now be, uh... Have c- correctly displaced the, the department sets view. So let's see if that's actually true.

146. 30:46

     It looks like it. Yeah. So we now have these, uh, cards, uh, with the whole view of the company. So I'm logging in as Julia from HR. Uh, we can see that we have five people on the engineering team with an average salary of, uh, uh, 107K.

147. 31:04

     Uh, HR as well, well, only one person, w- would be Julia, and product has four people and that average salary. So, so far so good. Looks, looks okay. Let's see.

148. 31:18

     Uh, so this is like, this is sensible information, uh, the, the, the reports, right? So the, uh, we're expecting that the other, um, the other, um, employees will not have access to it, and even the managers only have for their departments.

149. 31:33

     Let's see if that's the case. So let's navigate to Bob. Bob, uh, is the head of engineering.

150. 31:40

     Oh, okay. So, so Bob also can see the, the performance reviews of both, uh, uh, the, the information of both HR and product. Well, it's not, it's not that bad, right?

151. 31:52

     Uh, it, um... Well, it's not ideal, but, uh, at least he's a manager, right? So, uh, he should be access to privileged information anyway, and, uh, ooh, doesn't like a, a transparent company.

152. 32:03

     Let's see if... Oh, hell, this is... Okay. Um, okay, this is, this is problematic. So we basically created a view. Uh, Claude, uh, said everything is working because as you can see, the information is here, it was created.

153. 32:18

     But, um, he missed something, uh, that his training data basically missed something, which was for pos- Postgres specifically, um, when you create a v- a new view, the,

154. 32:33

     a- and your, um, table has Row-Level Security enabled. So for those of you that know, a Row-Level Security allows t- uh, for you to define who can see the information on a spec, uh, on a specific row on a database level.

155. 32:47

     So without trusting the, the application, you can filter it directly on the, on the database. So in this case, we should be limiting the, the, the view of, uh, of the rows, uh, by user ID, right?

156. 33:01

     Uh, and the, the user role. So if a user has an employee, um, and has an employee role, he should not be, should not have access to the rows of the, uh...

157. 33:11

     That don't belong to the, to them, right? Uh, we have Row-Level Security enabled. If we navigate to our Supabase, um, migrations,

158. 33:24

     you can see, uh, Row Level, that we have Row-Level Security enabled

159. 33:35

     both, uh, on profiles and on performance reviews, right?

160. 33:41

     And on a performance review should be, should be about right. Uh, right. So we have

161. 33:50

     reviewer ID equal current setting, so it should work. Why is it not working? Well, when you create a view on Postgres, um, by default, the, the, the permission, um, it creates with the permissions or, or, or the, the credentials of the, the user that created the view and not, uh, with the, the, the credentials o- of, uh,

162. 34:14

     of the, of the table, let's say with the, the Row-Level Security. So basically, by default, it bypasses the Row-Level Security, uh, that you have in place on your... That you might have in place already on your, um, uh, on your, uh, on your table.

163. 34:31

     So for, for this to not to happen, we have to add a security invoker. Uh, we have to use a security invoker flag to

164. 34:43

     transfer the Row-Level Security policies or to enable the tr- the, the RLS policies on the view itself. So this is the why currently everyone can see everyone's, because the, the Row-Level Security policies were basically bypassed on the, on the view.

165. 35:00

     So, uh, for the sake of the, of the demonstration of this, uh, workshop, I've already created, um, a skill.

166. 35:12

     Uh, uh, I prepared the skill, uh, for, for the presentation. [clears throat]

167. 35:18

     Uh, and essentially the skill is three main po- main security points about Postgres, uh, for... That the agent should be aware of during the, the presentation. For this one specifically, I actually overfeed it to the exact view that we're creating, but models right now are smart enough to, uh, generalize this and if I wanted to, to create

168. 35:40

     a new view, you will be able to. Uh, essentially it has to create with this flag. Uh, since Postgres, uh, version 15, this flag was enabled, and every time it's enabled the, the Row-Level, the RLS policies, uh, um, are also enabled on the, on the view.

169. 35:59

     Um, as you can see, it's, um, it's actually quite human readable documents. Um, mo- most of you have already written skill, so I'm not going to, uh, dive, uh, deep into, into this.

170. 36:16

     Um, but as you can see, we have both the title. Uh, let me just move, move this. We have the title, I called it Supabase Security. Uh, and the description, uh, uses the, the verb use.

171. 36:30

     Uh, this is, uh, an insight that I, that I got from, uh, some experiments that I've did. Using verbs, mainly the, the ver- um, the verb use, uh, increases the chances of the skill being loaded, um, at least on Claude.

172. 36:46

     I don't know- If this is default behavior for, for Claude, if he was trained to recognize the, more, more easily recognize verbs, essentially "use," uh, but I found it more efficient to...

173. 36:59

     If you write "use" and then the whole purpose of the, the skill, uh, in front of it. And then a regular markdown, uh, list. We have the, the view case, uh, there, but also another, uh, checklist to the points, uh, for security, um, on RLS.

174. 37:18

     So public schemas should, should have RLS enabled by default. Uh, public schemas, uh, or exposed schemas are the, the, the, the database schemas that are going to provide information for the, the application that the user can see.

175. 37:32

     So for example, the, the users table, the profiles, the performance reviews, all this information is, is going to be fetched by the front end. Uh, it's completely secured 'cause, uh, Supabase makes it secure, uh, by, by allowing you to, to fetch information on, uh, from the front end.

176. 37:50

     Uh, but the, the key part here is that if you don't enable Row Level Security, you will not have this filter on the, on the table, and you will have to rely on the application logic to, to make the, the filter.

177. 38:03

     So enabling Row Level Security at least makes it safer, uh, for, for you as the backend engineer, uh, that the... you only expose the information that you actually want from the start.

178. 38:17

     And then a couple more things that I'm not going into.

179. 38:22

     So if we, uh, we can install this skill on this project, um, by running...

180. 38:31

     Where, where do I have the command? npx, yeah.

181. 38:37

     Oh. So I'll be... Oh. I'll be using... Where's the skill? Okay.

182. 38:50

     I'll be using Vercel's, uh, NPM package called skills. Um, curious to, to know how you guys have been packaging your skills. Have been... Have you ever used this, uh, this package?

183. 39:01

     Are you using plugins? Just... This one.

184. 39:06

     This one, yeah.

185. 39:06

     Yeah. This one, uh, mainly. Yeah, it became very popular, uh, a few months ago.

186. 39:12

     I, I think the only problem is it doesn't really, uh, adhere to the project boundaries. So you get like global skills and then if you want it only for your local project...

187. 39:22

     Yeah, you can install it both, uh, globally and, and on your project. Yeah. And also for multiple... as support for multiple agents. Uh, while plugins for now are still tied to the agent that is going to load them.

188. 39:34

     So Cursor has plugins, Claude Code has plugins. Um, I think other vendors has, have as well, but they're specifically distributed and made for those specific models. Uh, so we're using this one to install.

189. 39:47

     You can install any skill, uh, from a repo online that has a skill.md file or, or, um, you can use it to install the one locally. It will auto-detect the, um, uh, the location that you're trying to fetch from, uh, based on the format.

190. 40:05

     So in this case, we don't have any GitHub, we don't have any HTTP protocol there, so we have a dot, uh, slash, so it'll recognize that it's, um, uh, that it's a local one.

191. 40:16

     And for this, I'm going to, uh, uh, move to the main... Yeah. Okay. And on a good old-fashioned way, going to run on the batch.

192. 40:31

     So it's going to pop, pop this up, um, ask me which agent d- do I want to install on. I'm using Claude Code, so I'm going to install it on Claude Code.

193. 40:40

     If you're using any other, uh, any other, um, agent harness, you can also install it, as long as it's, um, uh, it supports it. Uh, I'm going to install it on, on a project level, so it's going to g- uh, in this case, going to create a .agent folder with the skill and, uh, link it to my

194. 41:00

     .claude, uh, slash, uh, skills, uh, folder as well, so the, so Claude knows how to, where to find them. I'm going to symlink and we're ready to install. So we, if we...

195. 41:12

     Oh, let's not expose my key. [laughs] Uh, I'll delete... This is just for the workshop, so I'll delete it afterwards. Feel free to use my, my free credits, uh, for the time being.

196. 41:23

     But essentially, created the, the, uh, uh, .agents. Uh, where is it? Yeah. So I also have some more things that we're going to see afterwards. Uh, but the essential part, it has the skill that I've showed

197. 41:40

     previously. Yeah, there it is. It's the, the skill. And it also created a sim, uh, a symlink, a symbolic link to the .claude. Uh, this is how the package works.

198. 41:51

     Uh, and this, this way allows to, to Claude to either search on .agents, which is becoming the standard, or on the .claude, uh, f- folder that it has.

199. 42:04

     So let's see. Let's run the same prompt again on a new session. Um, let me go back to the apps demo. Yeah. And start a new session. We should have this one enabled.

200. 42:20

     Yeah, there it is. So Claude is aware of, uh, the Supabase security skill now. For you to run skills, uh, you can either, um, just run your prompt and, uh, pray that, uh, uh, Claude imports your skill, uh, based on the description that you gave.

201. 42:40

     Uh, you can, uh, include the keywords "use" and then the name of the, the skill that you have on the prompt, and this will, uh, almost 100% of the times load your skill.

202. 42:53

     Or if you're using Claude Code, you can just slash- And write the name of, of your skill. And this a hundred percent guarantees that Claude, we're going to import the skill.

203. 43:05

     So for, for our use case or for the presentation, I'm going to... 'cause I cannot afford that it doesn't load the,

204. 43:19

     the, the skill. Let's... Wait, I need to,

205. 43:26

     uh, I need to reset the, the database to create the,

206. 43:31

     the view again. Um, going to be a workshop, uh, and it's npx supabase db reset --yes.

207. 43:50

     I'm just resetting the, the database, applying the migrations from the, from the start. Uh, I didn't-- uh, it didn't create any, um, any, uh, migration file. Uh, he applied direct-- the, the migration directly to, to the database.

208. 44:07

     Uh, so we now should, sh- should be good to, to go. It's going to, uh, bring down the, the database, uh, and create a new one, uh, based on the schema that we defined on the migration files and the seeded data.

209. 44:23

     Yes.

210. 44:25

     On, uh, loading the skill, I think when Anthropic first presented it, the whole point was to kind of load it magically in the background based on language. And now we have still to use the slash command or the user label skills to be trans- like, still missing a bit the, the initial promise.

211. 44:43

     Have you found ways to load skills where it understands, like, the topic going on without you having to articulate it differently? Do you all understand what I'm trying to say?

212. 44:56

     Like, the whole kind of how skills get loaded kind of thing is still pretty just we want to use that specific thing rather than be like understand that we are working with security or load, or pull up security just do it when it were into that.

213. 45:14

     Yeah. That's a, that's a fair point. So the, the... Your whole question or observation it's that the initial promise of skills whenever they were presented by Anthropic were-

214. 45:25

     Have they found any ways by, you know, actually getting work into that?

215. 45:29

     Yeah. Uh, so since this is, uh, this is like on the, the agent side, right? The agent decides when to load this. Uh, the best thing that you can do without explicitly either with the slash command or the use and then the name of the skill on your prompt, uh, is for you to play and, uh, play

216. 45:50

     around with the description, uh, and run a bunch of tests either manually or, um, or automatically, uh, [clears throat] to, to check what actually works and not for the, the ones that, uh, that you're expecting, the, the, the agent to behave, right?

217. 46:07

     So you define a bunch of s- uh, scenarios where you think that the, uh, skill should be loaded and when the skill shouldn't be loaded. Um, you test it out.

218. 46:19

     You can test it o- on your machine, like on this scenario, I don't want the, the skill to be, to be loaded. Prompt, uh, the, the prompt, uh, on, on Claude Code, let's say, and check if the skill was loaded or not through the CLI.

219. 46:32

     Um, and then play around with the description to see w- what actually works or not. Like this-- without actually explicitly call the skill, this is the best thing that you can do to, to, to, to test, uh, if the, the skill is being loaded correctly or not.

220. 46:51

     Yeah. It-- [clears throat] we're still at the, at the very beginning of, uh, like a, a very early stage, uh, of, of skills. Even for MCP, like all, all this, um, agent stuff, it's fairly new, so we're still, we're still standardizing things.

221. 47:08

     Uh, we're still figuring out what works and what doesn't. Progressive disclosure was something that no one was talking about, uh, six months ago, and now it's fairly... It's fair to say that it's one of the north stars of, uh, uh, agent, uh, development.

222. 47:23

     Uh, so in six months from now, probably co- could be another thing. So... Or skills could be the standard, or maybe Anthropic or OpenAI or someone else found a, a more efficient way to manage the context or provide more context to the, to the agent.

223. 47:39

     Uh, so we'll see, basically. All right. So the database was, was reset.

224. 47:47

     Okay. So at least now we have the view, but we don't have the information on your database. So now we should be able to run the same prompt again, but we...

225. 47:59

     but with the, with the skill. So if, if we hit the prompt...

226. 48:08

     Let me see. It was quite fast, I don't think.

227. 48:13

     Uh, yeah, but it didn't create one. Okay, let me try another thing instead of...

228. 48:24

     Uh, instead of this, let's use to create.

229. 48:36

     Let's see if it works now. Yeah, okay. So it loaded the skill. So now at least should have the context, uh, to create, uh, the, the, the RLS, uh, or the security invoker flag should be included, uh, uh, when creating the view.

230. 48:52

     Uh, and the steps should... The, the, the rest of the workflow should re- remain the same. So it, it will list my tables, right?

231. 49:01

     Exactly, identified the tables. And now if we look, if we look closely, we can see that we are, uh, we now have this, the, the flag here, uh, is going to be on the, on the migration.

232. 49:16

     So let's see if with the flag, uh, this is the expected result.

233. 49:23

     This is what, what happens when you write code a, a CLI, you know, I have the, the UI duplicated. Right. So it created the view.

234. 49:33

     We should be able to see it, but Alice shouldn't.

235. 49:40

     So what's happening? [chuckles] Wait. Okay. So, uh, do I have to reset? No.

236. 49:51

     Hmm. Interesting. Should have another... Uh, probably... Let me just see if I have it, uh, here.

237. 50:07

     Uh, where did I put it? User add count.

238. 50:19

     I'm going to cheat here. Let's say the both [keyboard clacking]

239. 50:27

     and the employee should be able to see information [keyboard clacking]

240. 50:40

     and, uh. [keyboard clacking] Okay. Basically, live troubleshooting.

241. 50:58

     What is not happening? Probably from a differ- a, a different, um, policy that I've defined here. Uh, but now it's going to troubleshoot. Let's see if the, the skill actually improves the, the, the efforts here.

242. 51:12

     If not, I have something on my sleeve, uh, 'cause, uh, if you're not aware of, uh, Supabase basically has, um,

243. 51:22

     database advisors that you can use, uh, to try to identify early, early on identify, um, some potential vulnerabilities or, uh, schemas that are exposed, information that might be exposed, um, before you're running into, into production.

244. 51:37

     Um, so if, if it can't figure out by itself, I'm going to include on the skill to also run the advisors, uh, to, to check. So this is the, the main part of skills is that you can...

245. 51:52

     Oh, that's... You can, uh, uh, see. Well, it's a, it's a very poorly written application, let me say. Uh, it's essentially the, the main part of skills, uh, it's not if, if this specific, um, demo works or not, is that you-- the, the, the behavior changed, uh, once the, it loaded the skill, right?

246. 52:13

     It created with the security invoker part. Uh, and with, with that, that just shows how powerful it is that, that you can create, um... You can change the, the behavior or, or guide the, the agent on demand basi- uh, based on the, on information that you, that you put.

247. 52:31

     You can think of the skill.mds as a prompt template that you can give to, to your agent. So let's just quickly troubleshoot. Oh, is it, it's even offering to apply a migration.

248. 52:45

     Let's see if it doesn't break my, my app.

249. 52:49

     All right. So it seems too complica- too complex. Anyway, going to, going to move. What was that? To a separate table.

250. 53:00

     Let's see. HR. Anyway.

251. 53:07

     Could, could you look at the context, like, uh, the /context in your, uh, just to see, like, how your context now looks like?

252. 53:15

     Good point. I have a fairly amount of skills. As you, as you can see, I've been playing around with them. I also have the, some of the pre-installed MCP servers for that c- um, that Supabase enables.

253. 53:30

     Uh, but essentially, ski-- uh, it would be more interesting if you, if I've just, um, if I've compared the, the context, uh, from before and after loading the skill.

254. 53:42

     So right now, skills take one point three, uh, thousand tokens, uh, on my context, right? Uh, as, as you saw, I have more than, than just this one, uh, skill.

255. 53:54

     But the skill was loaded, so the whole information inside skills.md, it was loaded to, to context. If we clear and run the context again,

256. 54:07

     the, the skill amount... So this skill is not, it's not enough to, for you to see. But as you can see, the s- the skills take quite, um, less space, uh, that, that the MCP, uh, would, uh, from

257. 54:27

     its... All right. Oh, okay. I have a newer, um, version of the Claude Code. So for those of you who are not aware of this, uh, Anthropic recently released the tool, the, the tool search tool, uh, which is a mechanism for, for Claude Code to load tools on demand.

258. 54:45

     So it doesn't load... Basically progressive disclosure, but for MCP tools, right? Um, the ke- the main difference between MC- uh, this progressive disclosure or the, the tool search tool, um, on, uh, on Claude Code and skills is that the progressive disclosure is, uh, built by design, uh, for skills.

259. 55:05

     So it's like already baked into the structure of the instance of the skill, while on MCP it's still not a standard for all tools. So it works for Claude Code, but for many other clients it won't.

260. 55:17

     It will just load all tools straight to your context. So, um, this is a, for now, a thing, uh, for just, um,

261. 55:29

     for, for just Claude Code. If you're interested about it, uh, we are going to have the, the founder or one of the co-founders of the MC- of MCP speaking on the 10th, so on Friday.

262. 55:38

     Um, he's going to, to give a, a brief overview of the, the MCP roadmap, um, which if it's something, if, uh, anything... If nothing changed since last week, uh, when he presented this in New York, uh, on the MCP Dev Summit, it should bring this, uh, this progressive disclosure part, uh, to, to tools to, to bring it

263. 55:59

     to the protocol itself. So-

264. 56:04

     Excuse me.

265. 56:04

     Yes.

266. 56:05

     Um, let's say that we have a very large da- database and, uh, we have to, to load, uh, in the context the schema of it, of this database because we, we have to query the database, uh, using agents.

267. 56:21

     Mm-hmm.

268. 56:21

     Okay. In your opinion, is it better to use a skill or, um, an MCP, for example, for, um, to, to load this schema but progressively?

269. 56:36

     Okay, uh-

270. 56:37

     Is it possible to use the schema to progressively disclo- um, uh, load the schema of this big database in your experience?

271. 56:47

     Y- yeah. Oh, yes. Okay. So, uh, uh, is your question more about h- how should we access it or the whole architecture of this, uh, pipeline to import the, the data progressively?

272. 56:57

     I j- I just want to, to ask to a, a- an agent, uh, to, to query the database and, uh, obviously, uh, uh, the agent, uh, uh, must know the, the schema of the database before...

273. 57:13

     Or not. [laughs]

274. 57:16

     Y- oh, s-

275. 57:18

     How can you teach the, the agent to, to query the database?

276. 57:23

     Hmm.

277. 57:23

     Using the skills, using the, uh, an MCP

278. 57:27

     server or something like that.

279. 57:29

     Okay.

280. 57:30

     And, uh, if you use the skills, if you decide to use the skills to, uh, to load the context of the agent with the schema of the database, is it possible to progressively load the schema within the context?

281. 57:46

     Hmm. Okay. Gotcha. Um, so l- let me break, let me break the, the situation, uh, let me break down the situation for you. Here you'll, you'll have, um, essentially two parts.

282. 57:59

     One is, uh, what's going to be on the, the context or what's going to be loaded and the, the specific information, uh, that you want to, to have, um, on your, on your scenario.

283. 58:12

     Uh, and the second part is the actual mechanism, the, the extraction mechanism that you're going to use to load the information from the database. So for the second part, to, to load the, the information from the, the database, you can either use, uh, a script, so a skill that invokes a script or an MCP tool.

284. 58:31

     Um, I would advise to use an MCP tool because you can, uh, use it. If, if you're using on production or on remote project, you don't rely on your local environment, you don't have to manage the keys, um, and the tool, it's already standardized and, uh, you already have the, the authentication baked into the protocol so the

285. 58:51

     agent never manage the, the authentication au- uh, token. It's on the... It just runs the tool, um, and, and it works for the, for it. To progressive disclosure the, the information on the database, it will...

286. 59:04

     You'll have to, um... You can include it on, on a skill. Yeah. Uh, you'll be using the MCP tool, so on the skill you'll probably s- state that use this tool to load.

287. 59:16

     And in the tool implementation, you have to enable it to not load to progressively load it, right? So to load into chunks. Um, it might be just enough from the, the, the tool parameters.

288. 59:30

     Um, the agent should figure it out by itself that if you put a parameter called buffer, for example, should be able to load it in chunks, right? Uh, instead of the whole table.

289. 59:41

     But if you want to have 100% sure that it's going to load into chunks and use it properly, I would also package with, with a skill and describe it how, how intends to, to use this, uh, uh, this tool.

290. 59:55

     So this is actually how both skills and MCP play along together. It's the, the tool to enable this connection, this integration, and the skill to describe how to use it.

291. 1:00:06

     Yeah. This is how I, I would implement this, this type of system. Uh, thank you for, for the, for the question. Uh, and it got me the opportunity to, to basically talk about, uh, how, how to use both m- skills and MCP and not put it, uh, uh, one against e- each other.

292. 1:00:23

     Um, so now as I promised, we should be moving on. Um, I'll have to t- to give it more time, uh, to, to figure out, uh, 'cause I've... I basically during the, the workshop, uh, when I was preparing the workshop, I've, I gave it a bunch of vulnerabilities.

293. 1:00:39

     So if I just kept it simple in that one, the demo should pro- would probably work. Um, since I have more vulnerabilities, uh, exposed that I, if I had time, I would, um, try to solve it.

294. 1:00:51

     Uh, it didn't for, for the moment. But, uh, uh, but you c- you, you saw on both, uh, scenarios that the first one didn't have the security, uh, flag, security invoker flag, and the second one had.

295. 1:01:04

     So at least we can, um- In-- We, we can imply that the, the, the skill was doing something. It, uh, the agent saw the information on the skill. It, it, it merged with the system prompt or stored it near, near the system prompt and, um, changed the behavior accordingly.

296. 1:01:26

     Uh, to test this... So if you want to move this, um, this part, the, the skill into production, right? So it works on your machine. It's a, it's a tale older than time that it's working on my machine, but I don't know if it's going to work on your agents, on your machine, uh, on your, um, environment.

297. 1:01:45

     So to have this, uh, to, to test this or to automate this testing and f- with this, we can unlock having a pipeline, for example. If you change, uh, one thing on your skill, uh, how can you reliably tell that it's, uh, it keeps doing what you're expecting, didn't break the previous flow?

298. 1:02:03

     So if I, uh, change one of the checklists, how can I ensure that the, the other ones were still working, right? So for th- this is where evals, um, could step in.

299. 1:02:16

     So, uh, evaluations, it's a very broad term. You can basically evaluate anything. Since this is a Markdown file, it's a free text file, you can evaluate basically anything. Uh, so it's f- um, fairly difficult for you to-- The most difficult part to create evals, I would say, is actually coming up with the scenarios, 'cause you would first

300. 1:02:37

     have to, to know what's the expected behavior, uh, of your, of your agents. Um, so coming up with representative, actually good s- scenarios that represent a fairly amount, that cover a fairly amount of, uh, use cases that you want to, to build are the most difficult.

301. 1:02:55

     Um, and there's still not a standardized structure to create evaluations. You can use, um, you can test it, um, or i- by importing a, a bunch of prompts and expected outputs, uh, from a CSV file, from a JSON file.

302. 1:03:13

     You can use tools, uh, like Braintrust or Langfuse to test its, uh, um, and to, to have a, uh, an analytics and an observ-observability layer on top of it.

303. 1:03:24

     Uh, for this presentation, I followed the, um, I followed the, what Agent Skills, uh, open standard defines as, uh, to, to design the, the test cases. So if you're not aware of this website, this is the landing page of the Agent Skills Open Standard, uh, to try to standardize what a, a skill is and how it should

304. 1:03:48

     behave. And they basically propose a very simple structure, local way to test the, the skills organized by... You'll have an eval.json that essentially has

305. 1:04:03

     a set of evals, so a, an array of ev- uh, of eval scenarios. Um, you'll put the prompt that you're going to give the agent, the expected output from the agent.

306. 1:04:14

     Uh, this is only if you have an LLM-as-a-judge. Uh, this is a technique used for non-deterministic evaluation. You, you would have, instead of a human, you can give the outputs o- of a, of a vel- an evaluation run, uh, to another LLM, say it-- define a su- a success criteria, and let the, the LLM who's, who's doing

307. 1:04:37

     the, uh, whose role is, is to judge in this case, that's why it's called LLM-as-a-judge, uh, to give it a grade, basically. So this is one part that you can automate on your evaluations for non-deterministic, uh, workflows.

308. 1:04:52

     You can either assert if a tool was called, or you can give the, the results to an LLM and non-deterministically try to, uh, get the, the agents to, to grade the, the performance of the other agents.

309. 1:05:05

     So you basically have agents evaluating agents. Um, so I followed this,

310. 1:05:11

     this structure. I gave the same, um, the same input here, right? Uh, so the, the agent that is going to run this evaluation is going to get the same input that we, that we had.

311. 1:05:25

     The expected output, it's that the security invoker, uh, it's true, so it's, it's present on the, um, on the app. Uh, sorry, on the view. And I have, and then I have a bunch of, uh, assertions that, in this case, um,

312. 1:05:39

     uh, I'll, I'm going to check it, uh, um, deterministically, right? I prepared a Python script that ess- essentially just resets the, the state of the database. So we, we ensure that, uh, since we're running this locally and not on, uh, isolated container, like a Docker container, for example, uh, we have to make sure that the systems always

313. 1:06:04

     starts, uh, from the same ground. So I'm going to reset the, the app. Uh, if you want to, to run the, the evaluations as well, you have to pick your own Anthropic key, uh, create, copy this.

314. 1:06:17

     You can follow the, the README inside the, the Supabase security. Uh, here you'll have how to set this up. Um, but then I, I will run the, uh, the Claude Code CLI on it, uh, I think it's on print mode or...

315. 1:06:33

     I can't remember what it, what they're called. But essentially, like we, uh, I will run it as a binary, headless. Um, so the agent will receive the, um, the, the prompts that are, that's on the evaluation, uh, as the task to perform.

316. 1:06:50

     And I'm also going to, to give the condition. Uh, we're going to test two conditions, one with the skill and another w-without it.

317. 1:07:00

     And essentially, so for you to, to see when the condition... Run Claude. This is where the Claude Code will run. And if the condition is with skill, we're going to load the skill.md into the, the, the system prompt, right?

318. 1:07:16

     Um- If you, if you would actually li- uh, would like to mimic the behavior, you would run this on the Docker container. You will put the agent skills, uh, on the .claude/skills, um, directory inside of a Docker container and let organically let the, uh, Claude codes, uh, find them and use them.

319. 1:07:37

     Uh, for, for this presentation, this is a very simple setup. I've just basically appended to, to the system prompt. So, [humming]

320. 1:07:52

     so we're going to run the evaluations. Uh, do I have the other... Yes, I do. Okay. I think we run it on the

321. 1:08:06

     database. [murmuring] It was not finding the...

322. 1:08:24

     How is it not finding the, the skill for Supabase? Uh, no. [lips smack] No. Okay.

323. 1:08:36

     Oh, um, I have... Wait. I know what's going on. I have the,

324. 1:08:41

     the wrong name. Let me change it. All right. So we started by running with the skill. So the first result that we should get, uh, is the, um, the with the skill.

325. 1:09:00

     It stopped. Now it's running without it, and then we're going to compare it. This will output a workspace iteration one, um, folder, and we can compare it both the outputs of, uh, with the skill and without it.

326. 1:09:18

     Uh, while the without skill is loading, let's just quickly inspect what, what the, uh, with skill, uh, output gave. Um, and essentially you can see that it created this, uh, the, the view with the security invoker.

327. 1:09:32

     And then we have this grading.json file with a bunch of information, like the, the assertions that we, we've put on the eval-- we've set on the eval.json. Uh, we have them here, and we can see

328. 1:09:46

     that for this one, graded as the f- as failing even though that created. Where is it? Not found. [lips smack]

329. 1:09:58

     Uh, the view has security certainty. Oh. Okay. I'm actually evaluating something wrong. So the problem here now it's, uh, with the... Yeah, with the skill, uh,

330. 1:10:12

     view. So since, uh, I was expecting this to, to create an PG class, uh, RL options instead of just inspecting the, the view, it's giving me that, uh, it failed.

331. 1:10:25

     But the key part... Has it finished? It's not finished.

332. 1:10:34

     Still running, take a long time. Could be... Okay.

333. 1:10:43

     And now we can inspect. Okay. So, uh, this is actually good, good insight. So, um, with these results, this is the, the tricky part of, uh, of writing evals.

334. 1:10:59

     Um, so as the, uh, as like normal tests, uh, y- the results will depend on how you implement them, right? It's ju- just code. Uh, so if you're evaluating something wrong or some-- or not the, the expected behavior, you're going to have wrong results.

335. 1:11:17

     It might not be because the, the system is, uh, is not working. So, uh, we've tested manually and see that with the skill it created with the security, um, the, the security flag, we can actually just inspect it here with the skill created.

336. 1:11:32

     Let's see if, if on this one. Surprisingly, this time it did. It's a non-deterministic, um, non-deterministic behavior of, of Claude. Um,

337. 1:11:44

     but since I was evaluating something wrong, right, uh, I was expecting it to create the... or, or inspecting, uh, the, um, a meta schema to check if the, the, the view, the, um, the security invoker was there or not, instead of just inspecting the view directly.

338. 1:12:02

     Um, the results came a bit off. So it said that with the skill it failed, and with

339. 1:12:12

     the, um, without the skill it, it passed. So, and if, if we inspect the both outputs, they're basically the same. So with this ju- just to show you how tricky it is to, to write evals because it's...

340. 1:12:28

     Although this can happen on, uh, with the, with the regular, uh, tests, um, it's easier to catch because the, the output is deterministic, right? It's just code. Um, here, if you're handing to, to an LLM to evaluate, it can, uh, sometimes hallucinate.

341. 1:12:46

     So to finish, uh, 'cause we're also almost running out of time.

342. 1:12:53

     To sum up the, the structure, this is the one that they recommend. Uh, I find it very easy to implement to, to getting start with. Uh, later on you can move on to more, um, complex, uh, s- uh, evaluation scenarios like running on a Docker or in a sandbox, uh, to guarantee that, uh, you get a, a

343. 1:13:14

     fresh environment, uh, with just one skill that you're testing on your set. Um- But essentially is y- you would just put two conditions with and without the skill, compare the results and see, uh, run, run them on the harness, the agent harness that you would like, and compare the results, uh, out there.

344. 1:13:33

     This is basically your very first, uh, evaluation pipeline to, to test the skill automatically.

345. 1:13:40

     From my end, that's all. I hope you find, you found this, uh, workshop useful to, to, uh, get your skills leveled up and ready to productions. I'm going... As I said in the beginning, I'm going to give, uh, a keynote tomorrow.

346. 1:13:55

     A keynote, no. A talk tomorrow about how we've implemented, uh, uh, and created the, the Supabase skill for the product itself, how we're keeping it main- maintainable while ensuring that provides value, and how we're, uh, testing it, uh, into production.

347. 1:14:12

     Thank you. [clapping] Anyone has, uh, any doubts, questions? I'll, I'll also be... Yeah.

348. 1:14:23

     Uh, so I have a question about, uh, like the number of skills that you typically install on your environment. Because with this progressive disclosure, it seems like we can basically keep adding different skills and the agent will automatically, um, basically...

349. 1:14:42

     The agent will automatically find them. Uh, do you have any recommendation on how many skills to have, or is there any limit, or we should just basically keep adding, uh, and it will magically work?

350. 1:14:58

     Yeah. Uh, I, I'm probably not the best person to talk about this 'cause, uh, it's easy for, for you to, um, get into this rabbit hole or j- of just like...

351. 1:15:07

     Especially when you're experimenting getting a bunch of skills as you, as you saw. I had plenty of them, um, installed globally and, um, I think it's fair to say that I don't use them all, uh, on a daily basis.

352. 1:15:20

     Um, but it depends if, if you're using them on your local machine, I think it's pretty, um, it's going to be pretty easy for you to, um, get this messy, uh, environment where, where you have, uh, all of them installed or most of them installed.

353. 1:15:38

     Um, for loc- for your local environment, I wouldn't... For now, since it's all very experiment, in my personal opinion, I would not, uh, constrain myself on like, um, space management or context management about this.

354. 1:15:52

     The progressive disclo- disclosure, it's a very powerful thing that you can explore in this case. You... Sure, if you have skills that you don't use, uh, you're going to have them, uh, fill your context window, but the descriptions are so small that you can afford to not delete them if you don't want to.

355. 1:16:09

     Uh, into production, treat them as, um, any artifact that you would have on your CI. Uh, so keep it clean. Uh, into production, into your CI, um, I would keep them only the, the exact skills that you're, that you're using in that specific case.

356. 1:16:27

     Yeah. Uh, another piece of information that I, that I could give you on the production part is that it's now more and more common for you to, uh, also export skills or make skills available on your repos, um, as like a- another piece of documentation.

357. 1:16:43

     So treat it, treat skills that you put into production as actual document, as you would treat documentation. So it's important for you to keep them updated, uh, included on your...

358. 1:16:53

     include the, the updates workflow on your claude.md or on your agents.md so y- you make sure that if anything changes, um, you will change the s- the, the skill as well, like you would do on, on the documentation if a feature or a workflow changes.

359. 1:17:09

     Um, y- from time to time, you can also create a w- um, a job to, to check if the skill, uh, is still, uh, running a fair workflow. Uh, if somehow you could check if the, the skill have been loaded by your users, um, i- in a...

360. 1:17:28

     If, if it haven't been loaded by a sk- by your users for a long time, w- does it still make sense to have it there? Um, so yeah, th- this is basically the, the piece of advice that I could give you for skills into productions based on my experience.

361. 1:17:43

     Uh, for the rest of it, you'll have to come to the, to the talk tomorrow to learn how we're putting it into production on Supabase.

362. 1:17:52

     Any more questions? I'm going to be around throughout the, the whole event, so if you catch me, um, if you, if you c- if we cross paths, feel free to, to ask me anything.

363. 1:18:03

     Tell me about what you're building. Love to see. If it's with Supabase, even more thrilled to hear about it. Um, and from my end, once again, thank you very much.

364. 1:18:12

     You've been lovely today. For, uh, 9:00 a.m., pretty co- good energy. Uh, so just from my end, enjoy the, the rest of the, the conference and we'll see you around.

365. 1:18:22

     Thank you. [clapping] [outro jingle]