Trust, but Verify

[on-hold music] Hey, everyone.

Thank you for coming. I am Shreya Rajpal. I am the, uh-- I'm one of the co-founders and the CEO of Guardrails AI. Uh, and today, we are going to be talking about trust but verify, which is a new programming paradigms that we need as we're entering GenAI native, uh, application development.

Uh, before we get started, a little bit about me. Uh, uh, as I mentioned, I'm currently, uh, [REDACTED:username] Guardrails AI. In the past, I've spent about a decade or so working in machine learning.

Uh, previously, I was the machine learning infrastructure lead [REDACTED:username] Predibase, which is, uh, an infrastructure, uh, machine learning infrastructure company. I spent, uh, a number of years in the self-driving car space working across the stack of self-driving.

Uh, and before that, did research in classical AI and deep learning.

Awesome. So we're seeing this massive explosion o-- in, uh, AI applications over the last year. Uh, there's a lot of excitement and, you know, that is also why so many of you guys are here attending this.

Um, we have folks from AutoGPT, uh, which, you know, really took the world by storm and opened up the possibility and all of our minds with, like, what AI can do.

Uh, we've seen, like, a lot of really awesome applications in mental illness, uh, sales, uh, even like software engineering.

Uh, this is a relevant graph. This is basically search interest for artificial intelligence over time. Uh, and you can really see that peak, uh around where, uh, ChatGPT came out.

Uh, but if you think about, like, where a lot of the reality is or a lot of where the value lies today, uh, even though generative AI applications have seen fastest adoption compared to a lot of these other consumer applications, uh, their retention right now tends to be lower.

Um, so these, these are some graphs I, you know, uh, borrowed from this really fantastic article by Sequoia. And you can really see that retention for AI first companies versus the one-month retention, you know, for, uh, for non-AI first traditional software companies.

So why is this the case? Um, a common symptom, uh, that a lot of people experience as they're working with generative AI applications is, uh, my app worked while prototyping, but it failed, you know, the moment I tried shipping it out or even the moment like someone else tried te-testing this.

It just behaved very unreliably. Um, but the root cause of this symptom is that machine learning is fundamentally non-deterministic.

Um, for those of you, um, you know, we're gonna, like, dig deeper into what that really means. So I'm guessing that a lot of you here have worked with traditional software systems before.

So if you think about, like, a database and querying a database, uh, to get a question about how much was, you know, the spend of X user over the last month.

Every single time you hit that database API, you are going to get what is the correct response, right? And correct really means like representative of whatever your true data actually is.

So this is completely irrespective of like uptime, um, you know, um, uh, and availability, et cetera.

This fundamental property allows you to really build these, like, really complex software systems which, like, power our world today. Um, but if you think about, like, machine learning model APIs, this is not really the case.

Because of, you know, fundamental like, um, stochasticity that is, like, part of machine learning systems. Um, for a lot of you that have worked with generative AI systems and LLMs in the past, you'll see that even if you ask the same question across, like, multiple times in a row, you're going to end up seeing, like, different responses.

And the-- because of this, being able to build these, like, really complex systems, uh, that talk to each other, that rely on previous outputs, et cetera, becomes harder because you have this issue of, you know, like, compounding errors that really kind of explodes.

Um, this is just, you know, like, diving deeper into the problem a little bit. A lot of, like, common issues, uh, as you work with these problems. Hallucinations, that's a very buzzwordy, uh, thing that a lot of us here are familiar with.

But there's a lot of other issues like correct structure, uh, you know, their vulnerability to prompt injections. Um, and all of this is exacerbated by the fact that unlike all other previous generations of programming, the only tool that is really available to you is English, right?

It's just the prompt that you can really work with.

So, um, we end up in this scenario right now and in the current, like, time that we're in, where use of LLMs is limited wherever correctness is really critical, right?

Um, I love GitHub Copilot. It's on my badge as my favorite tool. Uh, but if GitHub Copilot is wrong, you just kind of like ignore it and move on.

Uh, same as like ChatGPT. The chat interface is really, really great because it's iterative and you can give it feedback and, you know. Uh, if it's incorrect, you can tell it, like, why it's incorrect, and it can, you know, maybe give you something that's more, uh-- that's more appropriate.

Um, but this is not the use case for a lot of, like, really high value critical applications. And so how do we add correctness guarantees to LLMs, uh, while still retaining their, like, flexible nature, uh, you know, that really, uh, uh, allows them to adapt so well to so many tasks?

Um, so I'm gonna add this, like, quick quote here by Alex Gravely, who is, uh, the creator of GitHub Copilot. It's a very simple idea, which is that, uh, "Add a constraint checker to check for valid generation.

Um, on in-- on violation, inject what was generate-- what was generated and the rule violation and regenerate." Um, so once again, we're trying to think about, like, how programming paradigms change as we are working with this fundamentally non-deterministic technology.

So this is something that, you know, wasn't, uh, needed for the longest time because we were working with, like, deterministic systems, but becomes very relevant now

Um, so interestingly, this tweet was actually pretty recent. Uh, but Guardrails AI, the open source framework that implements this and kind of like f- builds a framework around this strategy has existed, um, for a little while longer, uh, from the beginning of this month-- uh, from the beginning of this year.

Um, so Guardrails acts as a safety firewall around your LLMs, and this kind of fundamentally introduces, uh, a novel paradigm that once again wasn't as necessary in the previous generations of software development.

So this is what a lot of the software development like architectures for applications that you might build, you know, uh, may look like wherein you have like some application, and then in that application, you have a prompt that gets sent to an LLM, and then you end up getting like some output or some response back.

Um, this is the new paradigm that we propose, um, and that Guardrails kind of like, uh, implements as a framework wherein every output that you get back passes through a verification suite.

And that verification suite looks [REDACTED:username] all of the functional areas of, uh, you know, inconsistencies or risks that you are really sensitive to as an application builder, which may be very, very different from, you know, um, if you're building a code generation application whereas if you're building like a healthcare chatbot, right?

Uh, so maybe like containing PII or PHI, like sensitive information might be something you wanna check against or profanity, uh, filtering that out. If you're building a commercial application, you might really care about the fact that there's no mention of any competitors.

Like if you're, uh, building a McDonald's chatbot, like nobody should be able to get your chatbot to say that Burger King's the best burger in town. Um, making sure that any code that you generate is executable within your environment, uh, as well as, you know, summarization or free-form text generation is true and grounded in the source that

you know to be, you know, correct and not just hallucinated from the model. So each of these ends up being an independent check that runs as part of this like comprehensive verification suite that allows you to build trust in the models and the, uh, ML applications that you're building.

Uh, so the paradigm that we propose is that only use, um, large language model outputs if your verification suite passes. On failure, you can really hook into this very powerful, uh, capability that LLMs unleash, which is, you know, their ability to like self-heal.

Uh, which is that if you tell them why they're wrong, they can often correct themselves, and you can kind of go through this loop again if you have the, you know, latency budget or the-- even the dollar budget or the token budget to implement this.

Um, I'm gonna like go over this very briefly, but under the hood how Guardrails does this, um, is that it allows you to create what we call guards, uh, from, you know, different inputs.

So you can use like either a declarative model spec, uh, such as like, um, uh, you know, like XML or Rail. You can use Pydantic models that implement like specific validation criteria and structure, or you can use string implementation.

Uh, you can create a guard from all of these components. If you want, you can add information about, you know, your prompt as well as the LLMs you wanna use.

Um, and then you create this [REDACTED:username] initialization, but [REDACTED:username] runtime, this guard will basically surround your LLM callable and then make sure that everything that you're sending in or getting out of the LLM is valid and correct for you, right?

Um, so for example, uh, if your output is valid, you end up sending the output back to your application. But if it's invalid, uh, you go through this loop of, uh, looking [REDACTED:username] which constraint is violated or which check is violated.

And then if on violation, uh, you have a set of these policies including like re-asking, which we touched on earlier, uh, filtering or fixing, which is programmatically trying to correct outputs, uh, falling back on some other system, uh, so refraining from answering or, you know, just no op where you don't actively take an o- action, but you

log and store what the outputs of those checks or verification was and like why, uh, that particular check failed. And then you only do this like on-- uh, you only return the output once you know you can trust whatever came out of the LLM.

Um, so within this framework, what Guardrails AI does is it's a fully open source library, um, that allows you to, A, create custom validators. Uh, it orchestrates the whole va-validation and verification process for you, uh, to make sure that, you know, you're not taking on this like, uh, really kind of like often latency-intensive task of doing validation

and make sure that it's done as efficiently as possible. Um, it's a library and a catalog of many, many commonly used validators across a bunch of use cases. Uh, and it's a specification language that allows you to compile your requirements into a prompt so that like whatever specific, uh, validators you wanna use are automatically turned into a

prompt so that you know that, you know, those requirements are also being compu-- uh, communicated to the LLM.

All right. So a common question, why do I need this? Why can't I just use prompt engineering or, you know, a better fine-tuned model?

Um, so okay. So for some reason my, um, rendering here is weird. Um, but controlling the outputs with prompts, uh, including using retrieval augmented generation, which basically injects specific context into your prompt, uh, doesn't act as a guarantee, right?

Um, LLMs are stochastic. Even if you do all the prompt engineering in the world, there's nothing guaranteeing that those instructions will be followed. Um, we actually did this as an experiment for an unrelated thing where we used LLMs as evaluators.

Um, we ran the exact same experiment five different times, changing like absolutely zero parameters with zero temperature and saw like different numbers across our benchmark, which is, you know, really fascinating and wouldn't really fly in like previous generations of machine learning.

Um, and then second, prompts don't offer any guarantees. LLMs don't, you know, uh, always follow instructions.

Uh, the alternative is also like controlling, uh, the outputs with models. Uh, so first of all, it is very expensive and time-consuming to train a model. Uh, in my past life, this was basically what I've done my whole life.

Uh, and I was so frustrated with this whole process as I joined a startup, uh, where my job was to make this, you know, this process easier like as a function.

Uh, but it still requires like, you know, compiling a lot of dataset which is expensive, uh, training a model over a bunch of hyperparameters, um, and then serving it.

Um, and then if you are, if you aren't doing that and you're using like an LLM that's hidden behind a commercial API, uh, you typically don't have any control over model version updates.

Um, so I've kind of seen this where, you know, I, I mentioned like validations get compiled into prompts. So I've kind of like observed where commercial models will get updated under the hood.

Uh, and so prompts that might have worked for you in the past will stop working, uh, just over time.

Uh, so how do these guardrails work under the hood, right? Uh, there's no like one-stop sh- uh, one-stop shop solution for h- for a guardrail here. It really depends on the type of problem that you're, you're solving.

Um, so a very reliable way, if possible, uh, for, for implementing a guardrail is to ground it in an external system. So let's say you're working in a code generation app.

Uh, a really good way to generate more reliable code is to actually hook up the output of the LLM into a runtime that basically contains application-specific data. So we, um, tried it for a lot of text-to-SQL applications, which is something that is supported as a first-class citizen in, in Guardrails.

Uh, and we found that this re-asking framework where you hook it up to, you know, a sandbox that contains your ta- database and your schema, um, really substantially improved the correctness of the SQL queries that you got.

You can also use, uh, rule-based heuristics. Uh, so really looking into like, okay, if I'm, let's say, trying to extract, uh, an interest rate from a really long document, I, I always must know that interest rates, you know, end with like, uh, percentage signs, and so that can be a clue that I must always be retrieving.

Uh, you can try to use like traditional machine learning methods or high-precision deep learning classifiers. Uh, so really you don't need the full power of an LLM to solve, you know, really basic constraints.

So, uh, trying to find like, is there, uh, some type of toxicity in this output? Uh, does some type of output contain, you know, uh, advice that is harmful for my users or is misleading my users in some way?

Uh, you don't need, um... My favorite analogy to use is you don't need like a jackhammer to crack open a walnut. So if possible, you know, some of the guardrails should use like smaller classifiers that are much more reliable and deterministic, um, uh, a- instead of, you know, using LLMs.

And then finally, you can also use LLM self-reflection.

Um, all right, so we're gonna walk through this example of how this works in practice, uh, for building a chatbot, uh, where you wanna generate correct responses always. Um, so let's say you're an organization that has certain help center articles, and you wanna make sure that, um, you always generate, you know, y- you-- your users can ask

questions over those help center articles in a chatbot, and you always generate like correct responses, where correctness means no hallucinations, uh, not using any foul language, so don't swear [REDACTED:username] your customers, um, and never mention any competitors.

Now, how do you really prevent hallucinations? Like that's a very fundamental question, right? Um, provenance guardrails. Uh, provenance guardrails essentially mean that every LLM utterance should have some, some leaning in a source of truth, right?

Especially if you're building like retrieval-augmented generation applications. Uh, you make the assumption that, okay, I gave it this context. I hope it's using the context. What you wanna make sure is that every output that is generated, you're able to pinpoint to where in the context, uh, you know, your response kind of came from.

So this is one of the guardrails that, you know, is-- exists in our catalog of guardrails. Um, under the hood, there's a few different techniques that we employ. Uh, we use embedding similarity.

Uh, we also have like classifiers that are built on traditional NLI, like Natural language inference models, uh, and we use LLM self-reflection.

Um, this is a very brief, uh, um, you know, snippet of like how to configure a guard, uh, when you can essentially like select from this catalog which guardrails you wanna use.

So we've, we've used provenance, profanity, no references to peer or competitor institutions. Uh, and then you essentially wrap your, uh, LLM call with, you know, the guard that you've created.

Uh, so very briefly, let's say you get some question which is like, "How do I change my password on your application?" Um, you have like some prompt that you know is constructed from your retrieval-augmented generation application.

Um, but because LLMs are very, very prone to hallucinating, there's like-- it hallucinates where the setting exists for you in your, uh, you know, uh, in, in the response.

Uh, when this passes through your verification suite, the provenance guardrail will essentially spike and will cause, uh, the LLM to, you know, like go through this like re-asking loop, where a re-ask prompt will automatically be constructed for you via guardrails, which will like pinpoint which part is hallucinated, uh, give it the context again and ask it to

correct itself. Uh, and then finally, the re-ask output, uh, you know, it tends to be more correct. And so we can kind of see here in this toy example, uh, that the output is, you know, corrected for you.

And finally, verification passes, and you can send this back to the output.

Uh, very briefly, more examples of validators that you can, uh, create or that exist. Uh, never giving any financial or healthcare advice, making sure that any code that you generate is usable, never asking any private questions from your customers or mentioning competitors, um, no profanity, prompt injection, et cetera.

Um, and then just to summarize what Guardrails does for you, custom validations, uh, orchestration of verification, uh, a catalog of commonly used guardrails, as well as automatic prompt compilation from your verification checks.

Uh, to follow along, you can look [REDACTED:username] the GitHub project, which is [REDACTED:username] [REDACTED:username]. Uh, our website with our documentation is guardrailsai.com. Uh, or you can follow me or the p- project on Twitter, uh, and that's for my LinkedIn.

Awesome. Thank you so much, everyone. [audience applauding] [upbeat music]