Balancing Innovation with Security & Safety

Talk transcript

1. 0:00

   [on-hold music] I'm Don Bosco Durai.

2. 0:18

   I'm the co-founder and CTO for Privacera. Uh, very recently, we open sourced our solution for safety and security for GenAI and AI agent. Um, I'm also the creator and PMC member of the open source project Apache Ranger.

3. 0:32

   Uh, it does, uh, data governance for big data. It's also used by most of the, uh, cloud providers like AWS, GCP, as well as, um, Azure. Uh, so today, I'll be mostly talking about how you can build a safe and reliable AI agent.

4. 0:53

   So before I get started, let's get some of the terminologies, uh, uh, standardized. Um, from my perspective, AI agents are autonomous assistants. Uh, they can do their own reasoning, they can come with their own workflow, and they can, uh, call task for doing some actions, or they can use tools to get, um, uh, make API calls.

5. 1:12

   So tasks are more specific actions. Uh, they may be able to use LLMs, or they may, uh, also call RAGs or, or tools. While tools are functions which can be used to get data from the internet.

6. 1:28

   Uh, if you have databases, it can go and get data from the database. If you have, uh, service APIs, it can call those things also. And memories are contacts which are shared between the, uh, agents, the task, and the tools.

7. 1:48

   To give a visual representation, um, there could be multiple agents, and a agent may have access to multiple task. There could be multiple tools, and as these tools can talk with APIs and DBs.

8. 2:01

   So one thing that you need to know out here is, um, most of the, the, uh, agent framework today, they are run as a single process. What that really means is the agent, the task, the tools, they are in the same process.

9. 2:15

   That means if the tool needs access to database, that would needs-- need to have the credentials. Uh, if they want to make API calls, it needs shared tokens. So those credentials are generally are service user credentials.

10. 2:29

    That means they are, have super admin privileges. And since they're all in the same process, uh, one tool can technically access some other credentials which is in the same process.

11. 2:39

    Similarly, [clears throat] if you have task or agents which has, uh, prompts, all the things that's running within the process, any third-party library, uh, they can also access it. So those sort of makes this entire environment a little bit unsecure, right?

12. 2:53

    So there's a little bit of a zero trust, uh, issue out here. Uh, the agents, the task, uh, they talk to LLMs. The-- If you don't have a secure LLM, then that is another [clears throat] area where these things can get exploited.

13. 3:07

    Um, if you take agent on its own, by definition it's autonomous. That means it will call their own, uh, make, make up their own workflow depending upon the task. [clears throat]

14. 3:19

    So, [clears throat] so that actually brings in another set of challenges which we call in the security is unknown unknown. So you really don't know, like, what the agent is gonna do.

15. 3:28

    So it's very non-deterministic. So because of this, the attack vectors in a typical agent is pretty high considering from some of the traditional software.

16. 3:41

    So what are the challenges because of this, right? So there are multiple challenges. So if you look from the security perspective, if the agent is not designed or implemented properly, that can lead to unauthorized access, uh, also data leakages of your sensitive information and confidential information.

17. 4:00

    Right? Safety trust is also another biggest challenge. Uh, if you are using models which are not reliable or if your environment is not safe enough, if someone goes and change the prompts, that can also give you wrong results.

18. 4:15

    Compliance and governance is an interesting thing. Most of us are so much busy even just getting the agents working, we're not even worried about a lot of the other things that are necessary for making your agent enterprise-ready.

19. 4:27

    So interestingly, I was just talking to one of our customer this Tuesday. They're one of the top three, um, credit bureau. So they built a lot of agents, but their biggest challenge right now is to take it to production.

20. 4:39

    For them, they consider a AI agent as similar as to a human user, and when they onboard a hu-human user, they go through a training, and they have a lot of regulations they need to adhere to, right?

21. 4:51

    They have data from California residents, so they-- they have to make sure anyone who's accessing, uh, California resident data, they should not be using for-- uh, if the user has not given consent, they should not be using it for marketing purpose.

22. 5:03

    They have international, so their Europe data. So who can access the data? There's regulations around it. And also there are a lot of regional regulations. So when they consider even a AI agent similar to a human, so they have a onboarding process, they have a training process, and they want to make sure the agents are also following

23. 5:21

    the regulations, right? So without that, they can't go into production. And we as AI engineers, we're still in the early stage, so this one of the things which is out of our radar right now.

24. 5:33

    So now how do we really address this thing, right? So those who are in security are, are associated with security or compliance, there's no silver bullet. Uh, the best way to do, have multiple layers of solutions.

25. 5:46

    So these are some of the things that I have in my mind. Like, so you can split it into three different, uh, layers. The first layer is what is the criteria to even put your agent into production?

26. 5:58

    Right? What are you need to do, right? Uh, we talk about evals, but most of them we're only talking about evals for how good your model is, how good your response is, it is hallucinating.

27. 6:08

    But you also need to have evals which are more security and safety, uh, focused. So we'll go through some of those things, but the, the goal of this eval out here is to come with a s-- risk score.

28. 6:20

    And depending upon the risk score, you can decide whether you can even promote this agent to the production. And the agent may not be necessarily you are writing it.

29. 6:27

    It could be a third-party agent, so it has to go to the same criterion.

30. 6:31

    The second is enforcement. Um, eval generally tells you how good is your agent built, and enforcement is the one who actually doing the enforcement or implementation. So you have to make sure you have a pretty strong implementation.

31. 6:47

    If your implementation is not good, your eval is going to fail. Essentially, you can't go to production. And third is observability. Uh, particularly in the, uh, world of agents it's a lot more important because there's so many variables involved out here.

32. 7:01

    Like, you cannot really catch all of them during development or initial testing. So you have to keep track of how-- what-- how it is used in real world and how you can react on it.

33. 7:10

    So I will go through some of those things in a little bit more detail.

34. 7:15

    So, uh, let's start with the evals itself.

35. 7:19

    Right? Um, if you look into traditional software development, uh, there is already a process. There is-- there are gating factors that tells you how you can promote your application into production, right?

36. 7:33

    So if you start with basic things like, uh, when you're writing your code, you have to make sure you have the right test coverage, right? When-- If you're building, uh, Docker containers, you have to do the vulnerability scanning.

37. 7:47

    If you're using third-party software, you need to make sure you're, you're scanning for CVEs. If you find high or medium risk or critical risk, you try to remediate that before you can go into production, right?

38. 8:00

    Uh, you do pen testing to make sure there's no cross-site, uh, scripting and other, um, vulnerabilities. The same thing applies for AI agents also, right? You need to come with the right use cases.

39. 8:12

    You need to make sure you have the, the right ground truth, so that when you are doing any changes, you're changing the prompt or you are, uh, bringing a new library, a new framework, a new LLM.

40. 8:23

    You want to make sure your baseline doesn't change,

41. 8:28

    right? Uh, if you are using third-party LLMs, make sure they are s-- uh, not poisoned. They are s-- uh, they have been also scanned for vulnerability. Uh, if you're using third-party libraries, which almost everyone is using it, make sure they also meet your minimum criteria for vulnerability, right?

42. 8:46

    And similarly to pen testing, uh, you should also do testing for your prompt injection. Make sure your, um, your, your application has the right controls so it can block them.

43. 9:00

    And most of the LLMs are already doing it, but not necessarily all LLMs are doing it.

44. 9:06

    The other e-eval is around, around data leakage. Uh, this also is pretty important, particularly in the enterprise world, because when you are building enterprises, you're building agents which does generally what a human would do, right?

45. 9:19

    If you're building a agent for HR, uh, that will certain functionality. If I am employee, I can request for, um, uh, to get my b-- uh, salary benefits. But I can't do the same-- I can't get for someone else.

46. 9:35

    But if I'm HR admin, there's a possibility I may be able to access someone else's salary benefit, right? How do you make sure your agent is not leaking data?

47. 9:44

    There's no malicious user who can exploit some of the loopholes you have. So you'd have to do these evals upfront before even you can put tho-- your agents in the production.

48. 9:54

    Uh, similar to data leakage is unauthorized actions. Um, most of the agents, even though, uh, are read-only, there are also now agents coming which are trying to change things.

49. 10:03

    They'll, they'll do some actions. How do you make sure those are also done by the right person or the right persona? And runaway agents, um, uh, those who are-- uh, work on agents already know, like the agents can go in a tight loop and for various different reasons.

50. 10:20

    It could be a bad user prompt or just the prompt for the task, or the agents are not-- cannot address those things. So you have to make sure you test for such scenarios before you put your agent into production.

51. 10:33

    So the goal of this is to come with a risk score at the end of the day, so that it gives a confidence that can you put this into production.

52. 10:40

    And the next one is gonna be around enforcement.

53. 10:45

    As I said, your risk score is gonna be depending on how good is the enforcement. And particularly in agents, um, you're working in almost like a zero trust environment, right?

54. 10:54

    Because you have libraries which can access anything, right? Uh, if you are accessing certain of your backend systems which have sensitive data, how do you make sure the wrong user is not a-accessing it?

55. 11:07

    So, uh, from the security control, there are a lot of other things which I'm not gonna talk today, like, uh, detecting prompt injections and moderation. But focusing on enterprise-level thing, uh, you have to make sure you have the right authentication authorization.

56. 11:22

    Uh, this is pretty important because when you look at the environment, when a user makes a request to agent, it goes to task and eventually goes to tools and makes a API call to a, a service or a database.

57. 11:36

    If you don't have a right authentication, someone can impersonate someone else and may be able to s-steal confidential or sensitive information. And the second is authorization. If you have the authentication done properly, then you have to make sure the access control is applied properly.

58. 11:53

    And this is also important because agents have their own roles, and as a agent, they can do certain things. So you have to make sure they are not going beyond what they're supposed to.

59. 12:04

    And at the same time, if you have agents which are trying to do something behalf of another user, then you have to make sure the, the user, the-- that person's role is enforced.

60. 12:16

    So if you're accessing the database, it shouldn't access anything which the user does not have permission to or making API calls. So, so that's why authentication, authorization are s- super important.

61. 12:26

    Without that, um, obviously there could be a lot of other issues.

62. 12:31

    Approvals is interesting because, um, in the traditional world, we only have workflows. Uh, if I request for, uh, uh, leave, my manager will approve it. It's already built into the system.

63. 12:43

    But in the case of agents, you don't need to have a human all the time. Y- your agents can do most of the things automatically, right? So does it-- if you do it, design it properly, you could have another agent which all it does is just, uh, uh, looks for approvals and making sure the results are right.

64. 13:00

    And you can also put thresholds, how much these agents can approve automatically. And you can put the proper guardrails to make sure if it goes above a certain, uh, limit, it can automatically get a human in the loop.

65. 13:16

    So, uh, just to reiterate this one [clears throat] because it's pretty important, is when it comes to authentication, authorization, it's not just about doing the authentication at the point of entry, at the-- where you're making a request.

66. 13:30

    It-- You have to make sure the user identity is propagated across everywhere. If you're making a-- calling a task, or the task is calling a tool, you have to make sure the user identity is passed on to the, the, to the last point where it's actually making a data access or making API calls.

67. 13:47

    And that point, you have to make sure you're able to enforce the right policies and access control.

68. 13:54

    And the third is, um, observability. [clears throat] So observability is pretty important in the agent world because, as I mentioned, the traditional software, once you build it, it's-- then it just works.

69. 14:08

    Uh, you just have to make sure it is, uh, there's no, um, new vulnerabilities coming in because of some library update or something like that. But in the world of agents, um, there are many different variables involved.

70. 14:19

    One is the models change very rapidly. Um, you're-- if you're using an agent framework, that is also keep evolving, right? You're using third-party library, that may be-- start behaving differently.

71. 14:31

    Um, the another important thing in an agent is it's very subjective to what the user is entering. Uh, you may have tested with a certain assumption, mostly sunny day scenario, or I want to apply for, uh, my leave.

72. 14:45

    But the end user may use entire different, um, uh, um, text to ask the same questions. How do your models gonna behave best? So you have to keep monitoring to see if the user inputs or anything that changes, how the responses are coming in.

73. 15:02

    And also to make sure how much PI data and other confidential data has been sent out, because if you see some anomaly, you have to be able to ready to, yeah, act upon it.

74. 15:12

    Um, the other thing is, obviously, you can't monitor each and every request, right? As the number of requests increases, it's just not possible. So you have to start, uh, putting, uh-- defining thresholds and metrics.

75. 15:27

    So what that really means is, uh, uh, to start calculating, uh, uh, counting how many failure rates are out there. Once you know you have a certain failure ra- rates which is within your, uh, tolerance is fine.

76. 15:40

    But if it, if it goes above that, you can automatically create an alert and look into it. And the failure rates could be because of, uh, misbehaving agent. It could be a malicious users trying to, um, um, compromise the system.

77. 15:54

    Then anomaly detection and, is another interesting thing. Uh, I don't think we are anywhere close to it yet. Uh, but it's very common in, uh, the regular traditional software in the security side.

78. 16:05

    There'll be always something like user behavior analytics, where they look at the user and see whether they are within the, um, uh, um, standard operating thing. With agents coming in, uh, there'll be more and more of anomaly detection, whether agent is behaving within the, [clears throat] uh, accepted boundaries.

79. 16:26

    And all those things will end up with a security pros-process that will give you near real time saying how good your agent is actually performing in life. So that gives you to-- a bit of a confidence.

80. 16:41

    So to recap, uh, as I said, there are three things. One is preempt you. How vulnerability are eval to make sure that you get the right risk score, which gives you the confidence whether you can promote the, uh, um, agent to production.

81. 16:55

    If you're using third-party agent, whether you can use it in your environment. Um, second is proactive enforcement. Uh, make sure you have the right guardrails, you have the right enforcement, you have the right sandbox so that you are able to run the agent in, in a secure way.

82. 17:11

    Um, make sure you have the right observability so that you know at real time or near real time how good your agent is performing, and if there are, uh, some anomalies, you can go and quickly fine-tune it.

83. 17:25

    So, um, just as I said, we open sourced our, um, uh, safety and security solutions. Um, it's called PAIG.ai. Uh, security and, and compliance is a pretty vast field.

84. 17:36

    I don't think any single company can do it. Uh, so we are looking for, uh, design partners and contributors who can help us in our journey. So if you're interested, please reach out to, uh, me at [REDACTED:email_address] or connect me in LinkedIn.

85. 17:51

    Thank you. [audience applauding] [upbeat music]