Combine Skills and MCP to Close the Context Gap

Talk transcript

1. 0:00

   [upbeat music] All right, I get, I have, uh, the green light, so we can get it started.

2. 0:18

   Uh, hello, everyone. Uh, you might have noticed that the, the title's, uh, changed a bit from the one that we have in, on the schedule. That's because when I've submitted the talk, uh, there's the MCP versus skill debate was still going on, was a hot topic.

3. 0:32

   I think now we settled on they're b- both different. They all, uh, have their own roles, and now the, I think the debate is more on the MCP versus CLI.

4. 0:40

   So I thought it would be more useful to come and explain how we've wrote our Supabase skill, uh, and the lessons that we got from writing, uh, the, this document, 'cause I've never spent more time writing a single document since I've wrote my master thesis.

5. 0:57

   Okay? So I know that writing a skill sounds simple, but it can be very complex, especially when you have a complex product like Supabase. Um, for starters,

6. 1:09

   I'm Pedro. I'm, uh, an AI tooling engineer, uh, at Supabase. I'm an MCP enthusiast, AI in general. Uh, feel free to connect me on LinkedIn. Uh, and I'm al- also a co-founder of the Lisbon AI Week in, in Lisbon, will be on, uh, late October this year.

7. 1:26

   And if we're talking about me at the moment, I usually prefer, uh, doing this on a dark, darker, um, darker mode. I don't know. How many of you prefer dark mode over light mode?

8. 1:39

   The majority. I thought so. So let's do this presentation on dark mode instead. [laughing]

9. 1:45

   So we, I think we can all agree that agents are already smart enough, right? They can b- do very cap... They are very capable of doing mundane tasks by themselves.

10. 1:56

    But, uh, spe- when you present a task about something that they've haven't seen yet, or you've updated since they were trained, like your product, for example, they need the right guidance.

11. 2:08

    Uh, for ex- for example, uh, at Supabase, we noticed that they would usually either miss some, some security pitfalls that we have, uh, like role-level security instructions that, uh, they have to set, uh, to, to basically not expose your app.

12. 2:22

    Um, they could just usually operate on stale knowledge, uh, on their training data, and they are very lazy to, and very stubborn to admit that they don't know, uh, and they do need to find fresh information.

13. 2:35

    Um, and also, we would like to guide them on specific workflows that we think are the most optimized for agents, uh, on our product. So for starters, how many of you know or have readen a skill before?

14. 2:51

    Okay, so what I'm going to say is pr- probably not new for the most of you, but just to get an introduction on skills. So skills, um, are folders containing instructions, scripts, and resources that agents discover, right?

15. 3:04

    Um, progressively they discover. This is the main selling point of skills. Uh, and they, they have this envelope called front matter, where they have the name and the description.

16. 3:13

    This is how the agent is going to decide when to load the skill. Then they have the actual instructions inside the file, the main file called skill.md, and then optional bundled resources like scripts to perform actions, um, or reference files that the main file can reference for more information that doesn't have to be loaded immediately to context.

17. 3:35

    Uh, so we tested out, uh, at Supabase. We experienced giving the same agent, uh, in this case was Claude Sonnet, uh, 4.6, uh, the same prompt for a simple task.

18. 3:48

    Like, we had an app, like a collaboration, a collaborative app, and we wanted to create this new SQL view, right, on top of a table that already had row-level security enabled.

19. 3:58

    So the users could only see the information that belonged to them. Um, we gave, uh, we gave them, uh, just the, the MCP, uh, o- on one, uh, condition, and the MCP plus the, the agents, uh, the agent skills.

20. 4:14

    And the result was, well, the expected. Uh, if you don't know in Postgres, if you create a s- a skill over, uh, a table, uh, on top of a table that has RLS enabled, if you don't explicitly pass that, um, that flag over there, the security invoker equals true, it will bypass the RLS.

21. 4:33

    So basically, the view will expose data that is not dis- exposed by default on the, uh, uh, on the table. So the agent with the skill, with the knowledge, was able to get this, uh, uh, th- this information implemented correctly and safely, while the one that only had access to the integration to the, to the m- to

22. 4:53

    the MCP tool did not. So for this, we decided, well, just like this, we wanted to, uh, enable agents to know how to work correctly with, uh, with Supabase.

23. 5:07

    So we decided to, we are announce- actually announcing today, uh, this, um, this Supabase Agent Skill that I've been working on on this couple, uh, couple passive months. And to make things official, I'm actually going to try something.

24. 5:22

    Wait. I'm going to live tweet it on stage. So it's live. [clapping]

25. 5:37

    All right. So, uh, what exactly is this skill about, and what lessons can I share with you? So if you're building a skill for your product, uh, you can build one like we did.

26. 5:47

    And happy to discuss the, all details, like this is free text- text, and we haven't achieved a, a standard for it. So happy to, to chat about it later.

27. 5:56

    Uh, so to break down on some principles that we, um, converge to, the first one is that- Don't duplicate information. Yeah, like treat s- uh, skills as a document, uh, documentation for yourself, and you will not do- uh, um, duplicate your documentation, right?

28. 6:12

    You already have documentation on your product. Just point the agent to it, to the most up-to-date. You'll have to be very stubborn with the model to go to ask them to go f- uh, s- search the, the web or your documentation.

29. 6:26

    Provide the guidance, the tell where to and how to find the documentation, but be very persistent with it, uh, to go for it.

30. 6:36

    We're also, uh, did, uh, run a little experiments, and this is, uh, still, um, well, as I said, uh, an experiment and yet to be, um, uh, bit, uh, um, we're still figuring out the details.

31. 6:50

    But we also announced quite recently, and you can see it, you can read it on our blog, we're expo- we're basically exposing our documentation through, uh, SSH. The main reason behind it is that the agents can now look for the documentation like it was a file system.

32. 7:06

    So they're very familiar with file systems in general or navigating them, finding files and information on, uh, using Linux-based, um, tools. If we expose, if we give them this interface to also do this but remotely, uh, w- our premise is that they will have easy, um, ease to, to navigate the, the agent.

33. 7:26

    So would also would love a- after the talk or during the, uh, the conference to see and to hear your opinions on, on this idea. The second principle that I got, that I ge- I have for you is that if something can get skipped, it will be skipped.

34. 7:44

    What I mean by this is that besides, uh, new information on, uh, searching online, so agents, uh, like fetching information online or tool calling, it's expensive for, for agents, so they mostly defa- uh, default to their training data.

35. 7:59

    The same is true for reference files. We've noticed that, uh, even if, uh, when the, the agent loaded the skill, uh, it will, uh, even if it had reference files there, uh, it will be very lazy to load them.

36. 8:14

    And even if it loads one reference file, uh, if your problem requires more than one, the information that it's more the, uh, in more than one file, uh, it will most likely will not load two files, right?

37. 8:26

    It's almost impossible and not, not even starting on three or four. Um, so you have to be, um, very critical about what you put on your skill.md file from the beginning.

38. 8:38

    You put information that it's likely not to change, like in our case, a security checklist about Supabase, um, that we didn't really want it the agent to miss at all.

39. 8:49

    Uh, so we decided this cannot be on a reference file. We actually started by putting it on a reference file, and it usually missed it, so we put it on the skill itself.

40. 8:57

    So if you have any, um, information that the agent can just not miss and, like, defines your product, goes to the skill.md file. Do not afford it to put on, on a reference file.

41. 9:11

    And lastly, uh, the third principle that I have for you when writing a product skill is to be opinionated. You know your product the best. You know how to work with it.

42. 9:21

    You know how your users, or you should know how your users are using it. Um, don't be afraid of guiding the agents on what, on workflows, uh, that you think are the most effective when working with your product.

43. 9:35

    In our case, for example, uh, managing a database schema, right? I haven't, I actually haven't, uh, asked this and should have i- in the, the beginning. How many of you know Supabase and what Supabase does?

44. 9:46

    Okay. For, so for the ones who, who don't know, we're basically a back-end service. We provide, uh, a storage b- database, authentication, and other f- uh, features that you would need to create a, uh, back end, uh, out of the box.

45. 10:00

    So we provi- or as I said, I pro- we provide a database, and the agent can interact and manipulate your, your schema, right? Uh, we found that this, uh, for our platform was the best workflow for the agents to efficiently, uh, manage the schema.

46. 10:16

    So in this case, it run, uh, direct DDL operations, like change the schema freely on your d- um, development or staging database. Once you're happy about it, we provide, uh, an advisor who are basically linked, uh, to give any, uh, security or performance issues that the database cou- could have, fix them, um, and only then create the

47. 10:40

    migration file. So this will prevent the agent to create a migration file every time it changes the schema. We found this li- to be the best, um, workflow to manage the schema, so for us, it should be in the, um, in the skill when working with, with, with Supabase.

48. 10:59

    How we tested this skill? So we're living, uh, a very interesting times where we now can test free tests. We, we can now test, um, documents. We can now test documentation, right?

49. 11:10

    Uh, this would be, uh, completely bonkers to think, uh, years ago. Now I've basically been testing a markdown file. And how did we do this? Uh, through evals. Uh, so for those of you who don't know evals, evaluations for, short for evaluations, are basically, uh, tests that you can run m- mostly like you would run, uh, on

50. 11:31

    your CI. Uh, but the, uh, now instead of evaluating code, you're evaluating an agent, an LLM, and its behavior, what tools it's co- it's calling, wha- what's the, its reasoning, right?

51. 11:41

    And so we ran, on an early stage, we ran, uh, a set of six sp- specific scenarios for Supabase, so Sup- ongoing Supabase projects in different, uh, situ- uh, in different scenarios, right?

52. 11:54

    And we ran it against four different, uh, agents, uh, from two vendors, um, in three test conditions. So we wanted to test on a baseline, so no MCP, no skills, with just the MCP server and with the MCP server plus the skill.

53. 12:09

    All this was done based on a task completeness score, uh, four graded score, uh, that run on Braintrust. If you don't know them, they're around. They're sponsors of this, uh, conference.

54. 12:21

    Go, um, go to their booth and, and talk about it. It's a very cool product. So the results are, uh, are out here. Um- The skills plus the MCP, uh, outperform any other conditions on every model that we, we tested.

55. 12:36

    So we tested on, on Claude Code for Opus 4.6 and Sonnet 4.6, and also on Codex for GPT-5.4 and GPT-5.4 mini. Uh, so I think we can conclude that it's pretty na- unanimous that the skills were actually improving the performance, the task completeness score, uh, because they were providing the right guidance to, to the agent.

56. 12:59

    So we had already the tools. We already had an MCP server. We just needed the right guidance on how to operate with, with Supabase. Uh, it's agent a- agnostic.

57. 13:09

    Uh, most, uh, more and more agents are adopting this, uh, open standard, uh, on, on skills. Um, and currently, as I said, uh, in the beginning, the bottleneck is not the, uh, the context.

58. 13:22

    It's the guidance. So if you can take something from this talk, uh, when building a skill for your product, is that point for your single source of truth. Point to your documentation, basically.

59. 13:36

    Be opinionated. You know your product. Don't be afraid to, to show it. And start minimal. Any model vendor, let's say whether that's Anthropic, OpenAI, on any blog post about skills, you will read start s- uh, start minimal, start slow, and then iterate, expand.

60. 13:55

    Don't be afraid to create new versions for it.

61. 14:00

    So if you want to know more about it, we, uh, we actually wrote a, a blog post. Uh, it's live today. You can check it out on my Twitter account, on, on Supabase, um, blog.

62. 14:12

    Uh, or you can just run this, uh, this command and install your, um, install it on your project and start to s- to, to use them now. That's all.

63. 14:23

    I'll be around. And once again, thank you very much. [audience applauding]

64. 14:32

    I do have one more thing to show you. So we're running a g- a, a giveaway. If you want to, to have a chance to win a Mac Mini, uh, just scan the, the QR code,

65. 14:44

    uh, sign up for Supabase, and good luck.

66. 14:49

    Do I have time for any questions, if there's any?

67. 14:53

    Yeah.

68. 14:54

    Yeah? Okay. Yeah.

69. 14:55

    Uh, I actually have one. Uh, so since we're all moving into RAGs right now, uh, I'm just wondering how much of a demand do you guys see on vectorized databases, uh, on Supabase, right?

70. 15:08

    Hmm. It's a, it's an interesting, interesting question. So, uh, I mean, it depends. The, the... You, you, you're asking about, like, uh, how many customers are-

71. 15:19

    How much are you, um, having demand for vectorized databases?

72. 15:22

    There have been more and more, uh, as the, as the days passed. Like, um,

73. 15:28

    the use cases for, for vectors are mainly for embeddings, right?

74. 15:31

    Yeah.

75. 15:31

    Yeah. And semantic, uh, uh, uh, there are many use cases for, for embeddings. The one that I'm most interested about is, uh, um, semantic search, right? That you could use to provide even more context, uh, for example, through the SSH, uh, exposing, uh, the docs through the SSH.

76. 15:49

    You can now, instead of naively, uh, let them navigate with the, the bash tools, you can augment the tools, the already known bash tools, into providing some sort of, uh, semantic search.

77. 16:02

    So I do see a very big potential on vectors. On our data, definitely customers have been exploring more this, uh, this solution. Thank you for the question.

78. 16:14

    Thanks.

79. 16:16

    Yeah.

80. 16:17

    So thank you for the talk. Very interesting. How do you distribute the skills inside your organization? Do you,

81. 16:25

    do you just pass them? Do you have a repository? You have repository or do you use some package manager?

82. 16:31

    That's, that's an amazing, that's amazing question because, uh, currently one of the downsides or the, I would say the, the constraints of using skills is their distributions, uh, distribution system, right?

83. 16:43

    Uh, we're still finding the- there's still some players trying to reclaim, uh, the either the registry or the way to distribute it. So Vercel came up with the skills package.

84. 16:53

    Uh, we're seeing plug-ins, right, that you can bundle with MCP servers and other things. Uh, but they are model specific. So this is to distribute in general, skills in general.

85. 17:04

    It's already a pro- a problem for, for itself that we haven't solved it yet. Um, internally, we have, uh, we are packaging the skills on the, the repos them- themselves.

86. 17:14

    So if you want to, uh, create a plug-in, you just, uh, create a .claude, uh, plug-in or a .cursor plug-in or whatever in that repo, right? Uh, and then it's available, or, or it's discoverable, uh, if the, if the, the repo it's, it's, it's open sourced or, or you have access to it, uh, you can use the

87. 17:35

    skills package to, to fetch it. So yeah, this... And this is how I've seen the other, uh, companies, uh, uh, distributing their skills. Like a repo, a skill. Trying to package the, the skills into the, the knowledge.

88. 17:49

    Thank you.

89. 17:50

    Thank you. Are there any more? I think we have time for one more. Yeah. Yes.

90. 17:58

    I recently built something to build skills. Should we do a collab?

91. 18:02

    Sure. Let, let's talk after. All right. So once again, thank you very much. It was a pleasure to be here. I'll be around. Thank you. [audience applauding] [upbeat music]