We Vetted 2,000 AI Skills Before They Reached Developers

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone. Good afternoon. Today, I'm going to talk about how we vetted two thousand AI skills before they reached the developers.

2. 0:23

   But before I-- before that, I'm Lucas Palma, but many people call me LP. I'm the Product Security Manager at Nubank. The product security structures, uh, structure that within security, looking upon how we make code safe and supporting engineers, product managers, and everybody into making our products safer.

3. 0:48

   I have, uh, over a decade of experience in financial services, engineering background, also a lot of years working here at security and a close relationship with the part that I love, which is innovation.

4. 1:04

   So before beginning, I believe I wanna bring to you, uh, why are we here. So one thing that's important for all of you to understand is that the--

5. 1:18

   now that we are using AI everywhere, even though-- e-even with, uh, coding, one thing that, uh, is important is that the AI skills are being part of the developer workflow.

6. 1:32

   And this, this might bring some risks because although they look like configuration, they behave like supply chain dependents, like, uh, for example, libraries and others. So what we made here was to build a security review system in order to check if these skills were safe or not to be used before deploying them.

7. 1:58

   So the lesson that I want to bring you here by the end of this presentation is that we should be protecting the whole workflow, not only the code that's being generated.

8. 2:10

   All right. So what I mean about the supply chain part is that, uh, traditionally, the supply chain has, uh, packets, containers, models, and so on. But now in the AI era, it doesn't have only that.

9. 2:26

   It still have the traditional part, but it will-- it also includes skills, plugins, MCP servers, agent rules, and much more things to be acting as supply chain.

10. 2:42

    And where AI skill fits into this, uh,

11. 2:46

    I believe that before going to that, it's important for everybody to be on the same page on what is an AI skill. So an AI skill has, uh, there is-- the-- normally, the developer is using AI tools in order to generate an output, which will be code most of the case.

12. 3:06

    And within this AI tool, there are, are a bunch of things that can be embedded. One of them are the AI skill. So with this skill, we can have, uh, a capability to a model or to an agent, uh, bundling some instructions, some context in order to have better guidance over what it can be done.

13. 3:28

    But there is also an impact over that because somebody can create their own skill and share with others. So when we do that, this first person is guiding over the code that's being generated by the other person, and then c- that can be dangerous.

14. 3:47

    And since we are here talking in the AI in Finance track, it's also important for us to understand that we are in a regulated environment. So from one side, there are, are developers wanting better, faster coding, more context to have less repetitive work.

15. 4:09

    But on-- but from the other side, even more because of the regulate part, we need to be aware of the audit- auditability, of looking upon credentials, safety by default, and many other security aspects.

16. 4:26

    And keeping that balance is hard, right? So some people might say, like, "Are AI skills dangerous?" So

17. 4:38

    I brought here a few examples of what do I mean by AI skills being dangerous. So first, uh, one thing that can happen is that when people are describing what a skill can or cannot do, it can, it can ask for it to retrieve a token or something, and it will begin using that token, uh, hard-coded, which

18. 5:01

    will go to logs and so on, and it can generate a data leak in the future. Another thing that can happen is also the person to, uh, instruct the AI to use shell commands, and then this skill will be used by another person.

19. 5:18

    And when they use on their shell, a lot of dangerous things that, that can happen and a lot of files being modified and so on. And there is also permissions.

20. 5:28

    So depending on how the skill was configured, it might have excessive permissions, much more than what was needed. And even a typo can make some dangerous stuff depending on who is using that such skill.

21. 5:47

    So first thing first, what we did, uh, initially is that how do we share skills among ourselves, uh, how the engineers will be sharing these skills. So Uh, we went through the marketplace solution, so the skills are being canonically shared among, uh, marketplace with the plugins, including the skills among them.

22. 6:13

    So it's a internal marketplace where people can discover new skills, and that's our boundary where we are trying to make it safer. So what happens is that when someone creates an skill, it's, uh, will open the pull request, and normally it will go to the marketplace.

23. 6:32

    But we made a step before that, like a CI step, where we created a tool that's called Skill Vector. And this is-- this-- what this tool does is to check if this skill is safe or not to be used, uh, using a lot of assessments that I will bring it here.

24. 6:51

    And also classify those risks, uh, request remediation, and so on.

25. 6:59

    So what v- Skill Vector does in, in a single page is that when a skill is created or changed, not only during the creation phase, one thing that's important is that the engineers are able to use it locally and also be iterating until the skill is being considered safe before upload it.

26. 7:20

    And after them upload the skill, we also runs it again because we can ensure that the engineer has run locally or has run the most updated version. So we also be scanning that after the upload.

27. 7:37

    And then we have some determinate checks for the, uh, easiest parts to check some, uh, easy risks using regular, regular expressions and so on.

28. 7:48

    After that, when we check that we need better context, we then use LLM. Uh, it's important to have this hybrid approach with LLM checking the, the context, but also with the determinist because you know how LLM is, depending on the temper-temperature that was set, sometimes it will check that it's a risk, sometimes it might not.

29. 8:12

    And then all of these findings are reported in the PR that was opened to upload the skill, so it will improve the usability since the engineer will have the-- in the same PR what has to be changed before uploading the skill.

30. 8:27

    And another good thing that we made that's important is to have a serif with all of this so it can be consumed by our security tools as well and generate a report on the risks and be part of our vulnerability management program.

31. 8:44

    So depending on the severity, depending on the policy, uh, the skill can require some remediation, can be blocked, all of this before the marketplace distribution.

32. 8:59

    So there is the local scan, the pull request, the deterministic scanner, then there is the LLM review, PR feedback, serif, and then the decision. Will we use it? Will we allow it?

33. 9:12

    Will we allow it but it requires remediation? And so on.

34. 9:18

    A few examples of what we have ex- uh, scanned here. It's, uh, a non-exhaustive list. So we are looking upon if there are some unsafe instructions, if there are some drift befi- within the behavior that the agent has, if there are some destructive shell commands that I commented earlier, if there are some file modifications that shouldn't be

35. 9:40

    there, uh, credential requests, how are they being done? Some data being exposed, uh, unintentionally. If there are permissions that are overbroad, risky, MCP usage, and much more. These are, these are the main ones.

36. 9:58

    And so getting back to the title, we have scanned on that over two thousand skills. Uh, now there is much more than that, but this is the baseline that I brought for you on this presentation.

37. 10:13

    Uh, inside this we have identified, uh, more than one thousand and half, uh, risks. So not that one thousand and half, uh, skills had risks because a, a single skill can has many risks.

38. 10:29

    But these were the total risks that we identified over, uh, this amount of skills. And one thousand of them were probably, uh, remediated right after.

39. 10:43

    And there are a few of them that were really risky that we were able to block before going to the marketplace. So we also had made a ba-- a historical scan looking upon the skills that were created before the Skill Vector we implemented.

40. 11:03

    Uh, over there we were able to identify new, uh, risks as well and put it down into the vulnerability management program so it can be re- could be remediated.

41. 11:17

    A few lessons that I want to bring here as well. So

42. 11:22

    what things that worked well is having both the deterministic scanners for known risk patterns, but also LLM review for, uh, behavior, checking upon the destructive commands, uh, looking upon the credential checks as well, uh, having the output in serif, and adding comments on

43. 11:47

    PRs. And things that needed improvement and we worked, uh, during the process were also, there were some risks like comments that we were treating equally, but depending on the comment it can be more or less risky.

44. 12:02

    Also some signals that were weak and didn't have much context that were, uh, more troublesome than helpful.

45. 12:11

    There is also the prompt level ask for confirmation. I, there is a next slide about that, that I will go deeper. That's an important one. Also,

46. 12:22

    uh, there were some warnings that seemed, uh, harmless, but only if it was running locally. If they were going to production, then they could be impactful, and they-- we had also to look up on that.

47. 12:37

    Uh, if the finding had, hadn't some clear guidance was troublesome as well. And last but not least, we know that other people could create other marketplace. So how can we proactively scan, check there is a new marketplace, and put Skill Vector into it as well?

48. 12:56

    So regarding the prompt level, that's something that's important for you to know. People sometimes will add the instruction like, "You need to ask for confirmation," but the AI may a-ask confirmation for itself.

49. 13:10

    So from your perspective, there is a human in the loop. But for the AI perspective, there is-- has been a confirmation, and that's okay. Another AI has confirmed it, and let's go.

50. 13:21

    So that's something that we were scanning as well, looking upon having proper human in the loop, looking the tool that's in be-executing, if it's going through the approval gates, and so on.

51. 13:33

    Having hooks. And within this, as I said, it's a plugin marketplace. Skill is one among many things that there is into that. So there are things that we can reuse from this lesson.

52. 13:50

    So for example, treating these, uh, supply chain is important. Reviewing what's being uploaded to the marketplace before it goes there. Letting developers to run these checks locally. Enforcing these checks that are being run locally also in the CI.

53. 14:08

    Having deterministic checks together with the LLM checks. Uh, looking upon dangerous actions and prompting, uh, uh, and having enforcement when they happen.

54. 14:23

    So next steps o-over here is that I'm talking a lot about skills here, but a lot of these, as I said, could be applied to plugins, to MCP servers, rules, hooks.

55. 14:35

    So all of this that I'm saying here, we also have the MCP Vector, the rules checks, and so on. That's also applicable here, but with different risks. Uh, having also different gates depending on policies that we're implementing, depending on the marketplace as well.

56. 14:54

    Have some enforcements on tool level here, enforcing that there are audit logs, trusted gateways, and so on. And also, last but not least,

57. 15:06

    having the trusted, uh, trusted AI marketplace is very important, so we can have a canonical way to scan and share knowing that them are being safe. And that's not about, only about the skills that are being created by people, but it also includes the third-party skills or plugins and so on.

58. 15:28

    So if someone downloads something and wants to use, it's important to upload it on the marketplace so all of this scanning can be done and check if it's safe or not to be used.

59. 15:39

    And also that way, allow other people to use in a safe way.

60. 15:45

    And that's it. Uh, I'm sharing here my contact. There is, uh, my LinkedIn profile if anybody wants to contact, talk more about that. The QR code will bring you to my profile.

61. 15:58

    If you don't want to type, no problem at all. And I hope you, you've enjoyed the talk. [clapping] [outro jingle]