How to Build Trustworthy AI

Hi, my name is Allie Howe. I am a vCISO for Growth Cyber. We are a business that helps other companies build trustworthy AI. We sit at the intersection of AI security and compliance.

Today, we're gonna be talking about a variety of different topics, namely what is trustworthy AI? What goes into building trustworthy AI, and why you should care about trustworthy AI.

So to start off, who even needs trustworthy AI? Why do we care about it? Well, it's been in the news a lot, whether you realize it or not. All the way back in two thousand twenty-three, we all saw that case with the Chevy Tahoe incident where a user was able to, um, be offered a Chevy Tahoe from

a chatbot for a dollar. So that chatbot did not operate as it was intended to by the company, and it was in a position to be taken advantage of by a user.

In another instance, in twenty-twenty four, Slack was able to be tricked into leaking data from private channels via a prompt injection. So again, that, that system did not operate as intended and had some pretty strict consequences as well for probably Slack AI and the companies, whoever, whatever company was had that data leakage happen.

Very recently, a couple of weeks ago, we saw Darth Vader, an NPC being released into Fortnite by Epic Labs. That was really interesting. I think this is the first case of a voice agent being used in a video game.

So users were able to interact with an AI Vader and ask it all sorts of crazy questions. You know, at, at first, Vader exhibited a lot of bad behaviors, sort of reminiscent of the Microsoft Tay chatbot back from two thousand and six.

It was saying things that were racist, homophobic. Um, it's, it since has improved dramatically. Um, that's not what we're diving into specifically today. But as you can see, there's many cases in the news where AI was not necessarily trustworthy.

So it's something that is happening quite often and something that needs your attention.

There's a lot of debate around who is responsible for trustworthy AI, and it really boils down to you. You are the one responsible. There was a lawsuit the other day on May twentieth, so very recently, where a radio host was suing OpenAI over false statements that were generated by ChatGPT.

However, ChatGPT was able to get the case dismissed, or they won the case, or OpenAI did not end up in trouble simply because, um, it states that ChatGPT, um, can, you know, make wrong outputs from time to time, and it's up to the user to understand that and to proceed with caution.

So if you're using AI, it's likely you that will be responsible both on paper and also just from a brand and reputational standpoint. You are the one that needs to be aware that you are likely responsible and will be taking the fall for anything your AI application does that's incorrect, wrong, or inappropriate.

So when it, when we talk about building trustworthy AI, that's something that both product, engineering, and security teams are focused on. Product teams probably care that your AI application is outputting the right topics, it's relevant, it's generally helpful.

Security teams are probably thinking about, you know, your AI application's not gonna be saying anything that's going to be inappropriate or off topic as well, but they're also looking out for things like prompt injections and jailbreaks.

Engineering is helping out with these as well, probably cross-functioning. It's also thinking about things like cost and latency as well. So it's a really big team that comes together to be able to put together trustworthy AI specifically.

And the recipe for trustworthy AI is AI security and AI safety. So what's the difference? AI security is how does the outside world harm my AI application? AI safety is how does my AI application harm the world?

And we'll talk into... We'll go into detail about both of those shortly.

So there's this new paradigm out there that AI engineering has introduced. Traditionally, we had DevSecOps where our scanning tools, security tools like our SaaS tools were integrated within our CI/CD pipelines.

They are able to capture most of the vulnerabilities such as software dependency, supply chain issues, and insecure code. But now, thanks to AI engineering and, you know, data scientists and machine learning engineers, they don't work in our traditional CI/CD platforms.

They work in things like Databricks. They work in things like Jupyter Notebooks. So we need a new model for what AI engineering DevSecOps looks like and how we're actually going to build trustworthy AI.

Traditionally, there's been a big focus on shifting, but thanks, thanks to prompt injections changing rapidly, AI models, um, being deployed and, and switched out rapidly, there's now a big focus on shifting right for AI security, which is why runtime security is particularly important.

I mean, all three of these boxes here, build, test, and run are. Build, test, and deploy are. But with AI security specifically, especially because it's so non-deterministic, having something on the rightmost side at the time of runtime is incredibly important.

You know, no longer are we just caring about shift left. We're really worried about this entire life cycle. So on the leftmost side, we've got our build. That's where we're gonna be doing some sort of like model scanning, thinking about model provenance, looking at, um, AI or ML BOMs.

So that's the machine learning security operations. It's kind of a play on the term DevSecOps. That's important. And then in the middle, we've got AI red teaming. We're going, going to test our AI applications for both AI security and AI safety concerns.

And then on the rightmost side, we've got AI runtime security where we can validate AI inputs and outputs as they come into our AI system and our models.

So DevSecOps is out, MLSecOps is in. So basically, MLSecOps is machine learning security operations. As I mentioned before, MLSecOps is able to look into and take into consideration places the traditional DevSecOps does not.

As I mentioned, AI engineers live in Databricks and Jupyter Notebooks, not in traditional CI/CD pipelines. So it's important to focus there as well and look for exposed secrets that could be in those Databricks or Jupyter Notebooks.

And also part of MLSecOps is understanding things like model provenance. So you know, where... What it... Who built this model? Where did it come from? What data was it trained on?

Those are helpful things to think about for compliance as well. If you have, you know, access to that data the model was trained on, if you're supposed to be using it or safeguarding it, understanding maybe, you know, maybe a nation state made that, uh, made that model.

What are the implications of that? Is that something you need to be worried about? But one of the biggest risks from models that you may be using, especially like open source ones that you can get your hands on, are model serialization attacks.

A model serialization attack is when models are-- code is saved into the model at runtime, uh, sorry, at time of serialization, and then when you go to de-serialize the model, that code will automatically be run.

So you're now talking about arbitrary code execution, and we see this with the pickle serialization format. That's one of the most well-known ones for serializing models. However, if you look at the pickle's documentation, they do say that the module is, is not secure, and you're not supposed to unpickle data that you do not inherently trust.

So you're getting these models, say, from a model repository or a model zoo on the web, and you're just downloading it. You need to scan those to see if they have any unsafe operators in them, if you're at risk to model ser-serialization attacks.

Model serialization attacks run as soon as the model is de-serialized. So that arbitrary code could be causing data loss, credential loss, or model poisoning. So it's really easy to scan models.

So I really encourage you to do this and be practicing this within your organization. One example is ModelScan from Protect AI, and Protect AI also created this MLSecOps community where you can learn more about MLSecO-SecOps.

I've personally learned a lot from it, so I really encourage you to check it out. But I can show you ModelScan really quickly. This is the repo. It's open source, free to use, really easy to download, just pip install it and then run it like this.

They also have an example of a model serialization attack within this code base that you can go ahead and check out. So as you can see here, when the model gets saved, we are adding our unsafe payload in there, which is this command to basically output the AWS secrets that we have.

And so when this model gets run or loaded, we can now see that this AWS keys has been outputted, which, you know, then we have a credential leak. That's not great.

And then if you had used ModelScan to scan your model before, you know, this had happened, you would have seen that there is a critical vulnerability, unsafe operator in use here.

So that's one example of, you know, a potential solution for scanning your models that's open source free that you can use. There's other ones out there as well.

They also have a partnership with Hugging Face. So, um, ModelScan is used within Protect... within Hugging Face to do scans of files and model data. So if you ever see something that's like unsafe here, you can go learn about it.

So keep a lookout for that as well as you're, as you're pulling models off of model zoos.

So now that we've talked about the leftmost side, which was MLSecOps, let's now talk about the middle, which is AI red teaming. In AI red teaming, we can use that to simulate both adversar-ad-adversarial threats and also AI safety concerns.

So during AI red teaming, we can test for things like prompt injections, jailbreaks, but we can also test for AI safety concerns. Like, so if a user asks, you know, "How can I build a bomb?

How could I create chemical weapons?" Those are things that your model should not, you know, be answering. It should be safeguarding against. It also should be, you know, it shouldn't be biased.

It shouldn't do anything that's like homophobic or racist like we saw in the Vader example from the very beginning. These are things we can both test during AI red teaming, and we should be continuously testing our models because, you know, as we know, models change as users interact with them, not just with code deploys like traditional software

has. Another benefit of AI red teaming is you can use them to influence a runtime guardrail. So if we see that, you know, our model's particularly vulnerable to this type of prompt injection, or it's continuously saying like this sort of racist thing, well, we can block certain, um, topics and prompts that would elicit those responses to make

sure that our model behaves as expected during runtime. Another benefit of AI red teaming is the ability to compare LLMs to each other. In some cases, models can have backdoors built into them where you're not going to see this during sort of just for testing.

But if you start to compare LLMs to each other, you might pr... start to see differences that might suggest that one model has a backdoor built into it, whereas another does not.

So if you s... If you ask every single model the same question but one had a really different response, you might s... be like, "Okay, maybe this model, like one either doesn't work appropriately.

Maybe this was by design. There's some sort of backdoor. There's just deceptive behavior that was built into this model, so I'm gonna use a different model instead." Um, so there's a lot of reasons to do AI red teaming that can be particularly helpful both with model selection and helping you influence what runtime guardrails to put in place

during runtime. So in terms of runtime security, if there was one area to invest in, I would pick runtime security because AI, uh, red teaming can be particularly time-consuming and also expensive to...

If you're going to like retrain models based on the results, it might be particularly difficult or a lot of overhead for your organization if you're starting, you know, an AI security practice.

AI runtime security, um, is easy to implement. Typically, it's done with just including an API or installing another like Python module. Um, it's pretty easy to get deployed with a lot of benefit.

So We should really focus on AI security, runtime security because it's important to shift right. That's where we're gonna see things like the prompt injections being thrown at our models and different types of prompt attacks.

These can be indirect prompt injections where say you have a system that is using RAG or calling out and scraping a website. Web data can have hidden prompt injections in them.

Same can documents that you've used with your RAG setup. So you could get something that way. You could also have a direct prompt injection straight from the user to the chatbot or to the AI agent.

You can also see jailbreaks as well, which in text those look like things that are semantically strange, pretty chaotic looking. This is an example here on this slide of a jailbreak from Lucara AI.

And also at runtime, your application can see off-topic or unsafe prompts, such as ones that might, you know, get your application to output something that is unsafe. So something that's like, you know, instructions for how to build a bomb or something that's, you know, inappropriate or racist or, you know, something that's, it's bad to say.

You can check that at runtime, both on the input side, looking at the prompts, and then either deflecting it and not allowing your model to answer it in the first place, or if your model answers it and something that's incorrect or inappropriate comes back, you can go ahead and block that from being ever sent to the user.

So it's a super nice solution to have in place to make sure that your AI you've deployed is behaving in a trustworthy manner.

So let's take a look at that Vader solution from, or setup from the beginning, where we've got that Vader NPC running, um, in Fortnite. It was a really interesting setup that I tried to imagine.

So this is just an estimated guess of what their architecture looks like. So basically, the users that are in, using Fortnite are sending proximity audio, user audio. There's like k- other contacts being sent to the servers likely about like player skins or other player configurations.

And also, of course, the user's audio feed where we see the, where they're talking to Vader go into probably Fortnite servers. And then it's being passed to ElevenLabs for voice-to-text transformation, eventually being passed to Gemini to craft Vader's response.

And then we're gonna see Vader's response being sent back in text to ElevenLabs, then sent back as audio to Fortnite servers, and eventually all the way back to the user.

So there's a lot going on here, and if I was gonna be inserting AI runtime security, I'd probably insert it as close to the model as possible. So if we get something that's like unsafe, if a user, user said something that was maybe against user terms or conditions or, you know, the user said something inappropriate, maybe we

don't wanna like allow Vader to answer that. So we can validate our inputs here at this, at this location. But let's say we answer Vader's question, um, or topic, and it did come back as something that was off topic or inappropriate, we could flag it there.

So what I'm talking about right now is AI safety concerns, where the user has said something inappropriate. We don't want Vader to behave in a way that's inappropriate either.

So we could check for that. But of course, we can also check for, you know, prompt injections, jailbreaks, other, you know, AI security concerns as well. So AI runtime is both for AI safety and also for AI security.

You could, of course, also, you know, include AI runtime security here as well before we even get to ElevenLabs. So if the Fortnite servers come back and say, "Hey, like, you know, this looks like something that's inappropriate."

Like, I don't even wanna do a call to ElevenLabs. There's another API call. I'm sure it's expensive as well because they're using other services. So you could add it here as well.

Same here, checking it again after it comes out of ElevenLabs, checking the audio, making sure that that still looks good. However, there's trade-offs at play here. We have to think about cost, latency, and accuracy.

That's what a good AI runtime security solution has. It has all of those things: low latency, low cost or, you know, acceptable cost. It's also like highly accurate as well.

A lot of AI runtime security solutions are backed by really well put together and established AI research teams, which are constantly finding new prompt injections, new jailbreaks. That's not something you have to worry about when you're building your product, trying to keep up with the latest jailbreaks and prompt injections.

You can rely on your vendor for that.

So just to show you an example of AI runtime security at work, this is Pillar. It's an application security lifecycle application, which I have been using to do some of my work.

This is just an example of one. There's also different AI runtime security solutions out there. But basically, I had a, an application that I was using for

finding the, finding patients that would be suitable for ALS cli-clinical trials. And as you can see here, this is a, this is supposed to be a acceptable use case here, where we are getting patients in our database we would recommend for different studies that we found on the web.

So basically, the goal of this agent or this multi-agent system is to output a list of patients suitable for each trial. It's not supposed to answer questions about individual patients or modify patient data in the database.

So in this example, I have asked this system to change a patient's FEC percentage in the database to fifty. But Pillar has blocked this for me, so my agent's not going to answer this because I have configured my guardrails to restrict this topic and this key and keywords related to this so that we can make sure to

not change, um, patient data in the database, and this application can behave as expected. So a lot of guardrails are comprised where a lot of guardrails and applications or vendors will allow you to look out for things like PII, making sure that that's not being either input or output depending on your configuration, making sure we're looking for

things like AI safety concerns, toxic responses. But a, a really strong advantage of AI runtime security platforms or some of them is the ability to add custom guardrails. Um, so like this one in this example where I've got this very specific functionality where I don't want my system to be able to update database information.

So I'm able to use that

For this, which of course is a security concern. But I could also restrict topics around, like, you know, say if I was Tesla and I didn't wanna recommend Ford, for example, 'cause that's a competing car company.

I could restrict that topic to make sure that my outputs are outputs that are reflecting my business goals as well as my AI security and AI safety goals. So runtime solutions can be particularly helpful and impactful in that way.

And then once we have an AI security runtime solution, we can go ahead and verify that in different GRC platforms. So as I mentioned before, I help with compliance as well.

Um, so if we're doing all this work to build trustworthy AI, we might as well demonstrate it, and we might as well show that to our customers so that they can trust in what we build and also help us with our sales cycles.

So for instance, I have this risk in my risk register, which having a risk assessment is something that is required for SOC 2 at least annually. So if I have this custom risk of, you know, what if my company reputation suffers damage due to a harmful or off-topic output from an AI application?

In Vanta, I made a custom control for validating AI outputs.

So in this control, I have submitted evidence that I'm going to be validating all of my different AI outputs. And so I can go ahead and throw that into a trust center and show that I'm passing controls for validating AI outputs and inputs.

So if someone comes and wants to, you know, work with my company or buy my solution, they can go ahead and go on my trust center and see not only controls that are applicable to SOC 2 but additional controls that I have created for AI security that shows, "Hey, you know, I'm using a runtime s- solution.

Um, so I mean, I'm, I'm clearly taking AI security seriously. And so maybe you wanna buy from me, or maybe you wanna work with my company, or maybe this, you know, helps you not send me a, you know, 200-question page security questionnaire," for example.

So something to call out if you're, you know, if you're building with AI and you are taking AI, trustworthy AI seriously, and you're building it, might as well show it, might as well use it as a competitive advantage in your sales cycles.

So if you're not convinced you need to build with trustworthy AI yet, here are some other reasons why you might wanna take it seriously. Cybersecurity risk and business risk have never been more aligned.

So in the past, you could have built a product that was insecure by design, shipped it, made some revenue with it, and then added security later. However, with AI, we're not seeing that.

Making sure that AI applications output the correct outputs, that they are aligned, they are safe, they are on topic, that is as much of a cybersecurity concern as it is a business concern.

AI that outputs things that are off topic or irrelevant, it's not gonna be revenue generating. So getting AI security and trustworthy AI right from the beginning will not only be helpful for your security program, but it will be helpful for revenue as well.

AI will also make missing cybersecurity best practices worse. So you aren't tracking things like, you know, what data you have trained on, where you're getting different models, where you're...

h-how you're taking care of supply chain risk. AI's only going to magnify that and make that worse. Um, we're also seeing an increasing regulatory environment for compliance expec- compliance perspective.

We're seeing ISO 42001 crop up, which is the first international standard around AI regulation or a compliance framework, sorry. And then the EU AI Act, of course, came out, and that's what this picture is about on the right.

There was a case where an AI company was putting together a database of, uh, faces that they had scraped off the web, and they ended up getting fined by the EU for about twenty million, thanks to EU AI Act.

So it's important to take that seriously. If you are going to accidentally output, um, patient data if you're building a healthcare AI application, then you might be finding yourself with a HIPAA violation.

So that is important to take into consideration. But also different guidelines that are specific to whatever you're building, whatever your industry is. So for example, the... even the FDA came out with, like, AI and ML guidelines earlier this year.

So it's important to keep in mind what sort of regulations your company might be subject to. So it's, it's worth getting ahead of those either regulations that exist now or will exist down the road by building trustworthy AI today.

Trustworthy AI is super important because it's going to unlock a lot of revolutionary innovation. So for instance, if we think about the healthcare industry, we can't use solutions that help us identify new proteins that can be able to create entirely new organs for transplant or allow us to use models that were trained on one point three million

cells and has a lot of, like, very confidential data in them. We can't take advantage of the amazing technology that we're gonna be able to create as a society if what we're building isn't trustworthy to begin with.

These systems need compliance. They need AI safety. They need AI security. So if that doesn't motivate you to take trustworthy AI seriously, I hope it does, because what will end up happening is we're going to be able to create things we never thought possible, but only if we had trust in place in the beginning.

So the bottom line is you are responsible for building trustworthy AI. We've seen the news headlines. It happens all the time. We've also seen the lawsuits where it's often the user of the AI that's the one that's responsible.

So, you know, don't wait to get yourself into a lawsuit. Start building trustworthy AI today. And trustworthy AI is AI security, which is how does the world harm your AI application, plus AI safety, which is how does your AI application harm the world?

You can build trustworthy AI by incorporating MLSecOps practices, by red teaming your AI applications, and by incorporating AI runtime s- security solution at the time of deployment in your AI system.

So thanks for watching. If you wanna find me, here's my handles. But it was super awesome to deliver this discussion to you today. If you've got any questions, please let me know.

Happy to reach out, answer more questions online. But thanks.