# Trust, but Verify

Shreya Rajpal, Guardrails AI

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/9-vGxMoUM9Y-trust-but-verify) · [Video](https://www.youtube.com/watch?v=9-vGxMoUM9Y)

## Trust, but Verify: Put a Validation Boundary Around LLM Outputs

A prototype can look reliable until someone else uses it. Application-specific checks and corrective retries give model outputs a clearer path into dependable software.

Familiarity with model API calls and retrieval-augmented generation will help; the Python example uses callable functions and a bounded retry loop.

### The prototype works—until someone else tries it

[0:15](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=15s)

Why does an AI application work for its builder, then fail when someone else tests it? That is the practical problem behind Shreya Rajpal’s trust-but-verify approach. Introducing herself as co-founder and CEO of [Guardrails AI](https://guardrailsai.com/), Rajpal brings experience leading machine learning infrastructure at Predibase, working across self-driving systems, and researching classical AI and deep learning.

The excitement is easy to understand: AutoGPT expanded expectations for autonomous applications, while new tools appeared in mental health, sales, and software engineering. Rajpal’s search-interest graph shows a peak around ChatGPT’s release. But rapid adoption does not necessarily produce lasting use. Drawing on Sequoia’s [Generative AI’s Act Two](https://sequoiacap.com/article/generative-ai-act-two), she contrasts fast adoption with weaker month-one mobile-app retention among AI-first applications than traditional incumbents.

![Slide pairing a Path to 100 Million Users chart with one-month retention bars grouped into incumbents and AI-first companies.](https://ai.engineer/api/talk-essay-media/2776aac06782daf321c1e71bf86bc2771f7519804462b307b05dbca2177411db)

“The reality”: adoption speed and one-month retention for consumer applications.

The engineering symptom is a gap between a convincing prototype and reliable behavior outside the developer’s own tests. Rajpal attributes that gap to the nondeterministic behavior of machine learning systems: a successful response during development does not establish a dependable contract for future responses.

### Variable outputs become compounding errors

[2:50](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=170s)

Consider a database query asking how much a particular user spent last month. With the same underlying data and a correctly specified query, the answer represents the stored records. This is a question of correctness relative to the data, separate from uptime or availability. That predictable relationship lets one software component consume another component’s output.

A model API offers a different relationship. Repeating the same question can produce different responses, and downstream components may depend on whichever response arrives. Once those outputs feed further decisions, errors can compound. The failures include hallucinated content, incorrect structure, and vulnerability to prompt injection. In the prompt-centered development approach Rajpal describes, developers try to control these behaviors through English instructions, without an independent enforcement mechanism.

Human-facing interfaces can absorb some of this uncertainty. A developer can ignore a bad GitHub Copilot suggestion; a ChatGPT user can explain what is wrong and ask again. Applications where correctness is critical cannot always depend on that human repair loop. The goal is to retain a model’s flexibility while adding explicit checks before its output becomes an application decision.

### Insert verification before consumption

[5:50](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=350s)

Rajpal introduces a proposal she attributes to Alex Graveley: check a generation against constraints, then, on a violation, return both the generated content and the broken rule to the model and ask it to regenerate. [Guardrails](https://github.com/guardrails-ai/guardrails) provides an open-source framework around this pattern. Instead of sending a model’s raw response straight back to the application, route it through a **verification suite**.

The suite reflects the application’s risks. A healthcare chatbot and a code generator need different checks, even if both call the same model.

- **Sensitive information:** detect personally identifiable information or protected health information.
- **Language and commercial policy:** check profanity and competitor references. Rajpal’s example is a McDonald’s chatbot that should not recommend Burger King as the best burger in town.
- **Executable code:** check whether generated code works within the target environment.
- **Source grounding:** check whether a summary or free-form response is supported by the material the application treats as authoritative.

These are independent constraints, assembled into a suite appropriate to the product.

**Release an output only after the required checks pass.** If a check fails, feedback can help the model correct itself: identify what went wrong, request another response, and validate again. That correction is a possibility, not an assurance. Each retry also consumes latency, tokens, and money, so the application needs a budget for the loop.

![Side-by-side standard and Guardrails flows show prompt, LLM API, and raw output; Guardrails adds verification checks, a pass exit, and a failed-validation path back to reconstruct the prompt.](https://ai.engineer/api/talk-essay-media/fc91f1657a6940f3b465e031db4881c83cd10d469744fc3348188d9d50356ba7)

Guardrails adds verification and a failed-validation loop around LLM output.

### Separate the check from the failure policy

[9:10](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=550s)

A **guard** packages the requirements around a model call. In the talk’s API, guards can be constructed from XML-style RAIL specifications, Pydantic models with structure and validation criteria, or string-based definitions. Prompt and model information can also be supplied during initialization. At runtime, the guard surrounds the LLM callable and validates material entering or leaving it. These are the historical interfaces discussed in the recording; current package examples use a different API surface.

When an output is invalid, the next step depends on the failed constraint and its configured policy.

| Policy | Response to failure |
|---|---|
| Re-ask | Request a corrected generation |
| Filter | Remove offending content |
| Fix | Apply a programmatic correction |
| Fallback | Use another system |
| Refrain | Decline to answer |
| No-op | Record the failure without corrective action |

**Validation and enforcement are separate decisions.** A no-op can preserve the check result and failure reason while allowing execution to continue. It therefore does not implement the strict release gate described above. An application that requires rejection must configure failure handling accordingly.

The framework handles more than calling individual validators. It supports custom checks, supplies a catalog of common validators, and orchestrates verification to manage its latency cost. It also compiles validation requirements into the prompt, communicating the desired constraints before generation while retaining the checks afterward.

### Context is not a compliance check

[11:43](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=703s)

Why not use a better prompt or a fine-tuned model? Prompt engineering can communicate requirements, and **retrieval-augmented generation**, or RAG, can inject relevant context. Neither operation checks that the resulting response actually follows the instructions or uses the supplied material. Providing evidence and verifying adherence are different operations.

Rajpal reports that five runs of the same LLM-evaluator experiment, with no parameter changes and temperature zero, produced different benchmark numbers. She does not identify the model, benchmark, or magnitude of variation. The observation reinforces the need to examine actual outputs rather than treating a fixed prompt and sampling configuration as a guarantee of compliance.

### Training and model updates add operational work

[12:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=768s)

Controlling behavior through model training introduces another set of costs. Rajpal connects this to her earlier work simplifying machine learning infrastructure: assembling a dataset, experimenting with hyperparameters, training, and serving remain substantial work even with better tooling.

A commercial model API shifts those responsibilities to a provider, but it can also limit the application developer’s control over model-version changes. Rajpal describes prompts that had worked previously losing effectiveness after an underlying model changed. Requirements encoded only in prompts are consequently exposed to changes in how a model interprets them.

### Choose a checker that fits the constraint

[13:49](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=829s)

There is no single implementation that makes every guardrail reliable. When possible, ground the check in an external system. For generated code, that can mean a runtime containing application-specific data. In text-to-SQL, a sandbox with the database and schema can supply concrete feedback for a corrective retry. Rajpal reports that coupling this SQL sandbox with re-asking substantially improved query correctness, without supplying a numerical result. A runtime check provides evidence about the query’s behavior; successful execution alone does not establish that it answers the intended business question.

Other constraints call for smaller mechanisms:

- **Rules-based heuristics:** for interest-rate extraction from a document whose rates use percentage signs, check for that expected notation. The rule is useful because it encodes a property of the source format.
- **Traditional machine learning:** use a targeted model for a narrow classification problem.
- **High-precision deep learning classifiers:** check for categories such as toxicity, harmful advice, or misleading content. Rajpal favors smaller classifiers where suitable instead of invoking an LLM for every constraint.
- **LLM self-reflection:** ask a model to assess an output when the task calls for that form of judgment.

The choice follows the evidence needed to evaluate the constraint, rather than defaulting to the largest available model.

![Guardrails branches to grounding via external systems, rules-based heuristics, traditional ML methods, high precision DL classifiers, and LLM self reflection.](https://ai.engineer/api/talk-essay-media/2d41e8542b3d6ab0228b0a5be7637547f36104e149af94ad89da65df5b17aa0c)

Five approaches to implementing guardrails.

### Define correctness for a help-center chatbot

[15:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=948s)

The concrete application is a chatbot answering questions over an organization’s help-center articles. Its requirements combine factual support with product policy: responses should not hallucinate, swear at customers, or mention competitors. A single general judgment of whether an answer is good would obscure these separate failure modes.

**Provenance** asks where a generated claim came from. After retrieving help-center context, the application should be able to trace the response back to supporting material within that context. This checks support against the supplied sources; it does not establish universal truth or repair an incorrect source article.

Rajpal names embedding similarity, natural-language-inference classifiers, and LLM self-reflection as techniques used for provenance checks. Embedding similarity looks for related source material, while inference and model-based judgments can assess the relationship between that material and the generated response. The guard configuration selects provenance, profanity, and competitor-reference validators, then wraps the LLM call with the resulting guard.

### Repair an unsupported password-change answer

[17:46](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1066s)

A customer asks how to change their password. The RAG application constructs a prompt using retrieved material, but the model invents the location of the password setting. The answer may sound plausible while still failing to describe the actual application.

The correction proceeds through a specific sequence:

1. The provenance check identifies the unsupported portion of the answer.
2. Guardrails constructs a re-ask prompt containing the failed response, the relevant context, and an explanation of the unsupported content.
3. The model generates a revised response.
4. The verification suite checks that response again before the application releases it.

In Rajpal’s toy example, the revised answer passes verification. The key transition is from an unchecked candidate to a checked response, with corrective generation between them—not an assumption that a second answer must be right.

The following Python function expresses that control flow for the same password question. The caller supplies the retrieved help-center context, a model callable, and a verifier that returns concrete violation messages. It keeps a failed candidate out of the return path and stops when its retry budget is exhausted.

```python
from collections.abc import Callable


def answer_password_question(
    context: str,
    generate: Callable[[str], str],
    verify: Callable[[str, str], list[str]],
    max_reasks: int = 1,
) -> str:
    if max_reasks < 0:
        raise ValueError("max_reasks must be nonnegative")

    question = "How do I change my password on your application?"
    base_prompt = (
        f"Help-center context:\n{context}\n\n"
        f"Question: {question}\n"
        "Answer using the context. Do not use profanity "
        "or mention competitors."
    )
    prompt = base_prompt

    for attempt in range(max_reasks + 1):
        candidate = generate(prompt)
        violations = verify(candidate, context)
        if not violations:
            return candidate

        if attempt < max_reasks:
            feedback = "\n".join(f"- {item}" for item in violations)
            prompt = (
                f"{base_prompt}\n\n"
                f"Previous response:\n{candidate}\n\n"
                f"Failed checks:\n{feedback}\n\n"
                "Correct the response to satisfy these checks."
            )

    raise ValueError("No response passed verification")
```

The verifier remains the consequential component: the loop can enforce its decisions, but it cannot make an inadequate provenance check more accurate.

### Make application policy executable

[18:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1124s)

The same pattern extends beyond help-center grounding. Validators can enforce restrictions on financial or healthcare advice, check generated code, detect requests for private information, and address competitor mentions, profanity, or prompt injection. Each requirement needs an appropriate check and a deliberate response to failure.

Guardrails packages four responsibilities around those requirements: custom validation, orchestration of verification and re-prompting, a catalog of reusable validators, and compilation of requirements into prompts. That division lets the application communicate what it wants before generation and decide what it will accept afterward.

![Summary slide with four green checkmarks listing custom validators, prompting to verification to re-prompting, commonly used validators, and a specification language for communicating requirements to an LLM.](https://ai.engineer/api/talk-essay-media/b1a4332e33d7d6ba1441505f45715655c71fcf1608cb9cac73fb406865dd16bf)

Guardrails’ recap: custom validators, verification orchestration, a validator library, and a specification language.

For implementation, Rajpal points to the repository under its historical name, `ShreyaR/guardrails`, which now redirects to [guardrails-ai/guardrails](https://github.com/guardrails-ai/guardrails), and to the [project website and documentation](https://guardrailsai.com/). The practical work begins with specifying the conditions under which an output is acceptable—and what the application should do when those conditions fail.

## Resources

- [Guardrails AI repository](https://github.com/guardrails-ai/guardrails) — Source code and installation examples for composing validators around LLM inputs and outputs.
- [Generative AI’s Act Two](https://sequoiacap.com/article/generative-ai-act-two) — Sequoia’s September 2023 account of AI adoption, mobile-app retention, and the transition from demos to useful products.
- [Reducing Hallucinations with Provenance Guardrails](https://guardrailsai.com/blog/reduce-ai-hallucinations-provenance-guardrails) — A November 2023 walkthrough of embedding and LLM-based provenance checks, with historical Python examples.
- [The Guard](https://guardrailsai.com/guardrails/docs/concepts/guard) — Documentation for wrapping model calls, validating existing outputs, inspecting results, and limiting re-asks.
- [Use on_fail actions](https://guardrailsai.com/guardrails/docs/how-to-guides/use_on_fail_actions) — Examples of configuring what happens after a validator fails, including logging without blocking.
- [Shreya Rajpal on RAIL and corrective validation](https://www.latent.space/p/guaranteed-quality-and-structure) — An earlier interview and transcript exploring Guardrails’ origins, XML-style specifications, SQL validation, and re-asking.

## Slides

### Diamond sponsors

Fixie and Supabase are the diamond sponsors.

[0:09](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=9s) [Diamond sponsors slide with Fixie and Supabase logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/b3edd6c6072875ff1282697e19201c1cfe0bada6599bffeb282f6fe6e5368755.webp)

DIAMOND SPONSORS

FIXIE
supabase

### Trust, but Verify

Shreya Rajpal introduces new programming paradigms for GenAI-native software.

[0:19](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=19s) [Title slide for Shreya Rajpal’s talk on programming paradigms for GenAI-native software.](https://assets.ai.engineer/sites/aiecode2025/sha256/ec8eeae2eb1aafab07eb1380a2abf73580e560a6251f97ca51a6432ea82d33cd.webp)

Trust, but Verify

New programming paradigms for GenAI-native software

Shreya Rajpal
Guardrails AI

Guardrails AI

### About me

Shreya Rajpal traces her path from AI research to self-driving cars and ML infrastructure, before becoming CEO and cofounder of Guardrails AI.

[0:38](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=38s) [Shreya Rajpal’s portrait, current role, past work and career timeline from 2014 to 2023.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ff5b6ea22ce0495ff9387ebfddfa9f879c933413382d82d62225d166ace04fa.webp)

About me

Current
CEO & Cofounder @ Guardrails AI

Past
ML Infra lead @ MLops Co,
ML @ Self driving cars,
Classical AI & Deep learning research

Timeline: 2014, 2018, 2021, 2023
Research & publications in classical AI, Deep Learning
ML for Self driving cars
1st hire, ML Infra Lead @ MLOps Co
Guardrails AI

Guardrails AI

### A cambrian explosion of applications in AI

Headlines point to Auto-GPT, mental health and sales as AI application areas, alongside a post declaring software engineering as it has been known dead.

[1:04](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=64s) [Article headlines and a social media excerpt about AI applications in software engineering, mental health and sales.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa8fc38495b0d7875f462c0a78db0b767970792f0d224d5c7e2f8b1f00255924.webp)

We’re seeing a cambrian explosion of applications in AI

FORBES > INNOVATION > ENTERPRISE TECH
EDITORS’ PICK
Auto-GPT May Be The Strong AI Tool That Surpasses ChatGPT
Bernard Marr Contributor
Apr 24, 2023, 02:33am EDT

Seriously, software engineering as we have known it is dead in the water.

This AI is better than at least 95% of coders. And to the extent it is not, it delivers 1000x productivity improvements to complement them.

We have now officially entered the Age of AI.

CAN A.I. TREAT MENTAL ILLNESS?
New computer systems aim to peer inside our heads—and to help us fix what they find there.
By Dhruv Khullar
February 27, 2023

Harvard Business Review
Sales | How Generative AI Will Change Sales
Sales
How Generative AI Will Change Sales
by Prabhakant Sinha, Arun Shastri, and Sally E. Lorimer
March 31, 2023

Guardrails AI

### Interest in Artificial Intelligence over time

Search interest in artificial intelligence stayed low for years, then rose sharply and peaked around ChatGPT’s release before settling below its peak.

[1:33](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=93s) [Line chart showing search interest in artificial intelligence rising sharply in 2023.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd5931e2f9354207a02d3437fc60c7c2ad845498fd2832c16b490641b7c17f2b.webp)

Interest in Artificial Intelligence over time

Y-axis ticks: 25, 50, 75, 100
Timeline: Oct 7, 2018; Apr 5, 2020; Oct 3, 2021; Apr 2, 2023
Note

Guardrails AI

### The reality

Sequoia’s charts show ChatGPT reaching 100M users in ~2 months, while AI-first apps had 42% median one-month retention versus incumbents’ 63%, based on US mobile cohorts averaged over 12 months.

[2:02](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=122s) [Charts compare time to 100 million users and one-month retention for incumbent and AI-first apps.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4da68b51d1f6b33e087aef88e8d70b7605def62bf95aad17a2574139ee32fae.webp)

The reality

Path to 100 Million Users (stylized)
~2 months  9 months  30 months  41 months  55 months  61 months
Y-axis ticks: 0, 25M, 50M, 75M, 100M
Months from launch: 0, 10, 20, 30, 40, 50, 60

One Month Retention

Incumbents
YouTube 85%
Instagram 73%
TikTok 69%
Snapchat 67%
WhatsApp 63%
ROBLOX 61%
tinder 54%
duolingo 50%
Candy Crush 48%
63% Median

AI-First Companies
ChatGPT 56%
runway 45%
Lensa 39%
Remini 35%
42% Median

Data from mobile apps only. Averaged over the past 12 months of cohorts in the US.

Guardrails AI
Source: https://www.sequoiacap.com/article/generative-ai-act-two/

### Why is this the case?

An LLM application may work in a prototype but behave unreliably when someone else tries it. The speaker attributes this handoff failure to machine learning’s non-determinism.

[2:21](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=141s) [The slide describes an LLM prototype failing after handoff and attributes the symptom to nondeterministic machine learning.](https://assets.ai.engineer/sites/aiecode2025/sha256/a18404024befc40b75b48084b295298d8e51306906ca9ffe78e1b38283ffaff4.webp)

Why is this the case?

Symptom

“My LLM application worked while prototyping, but failed the moment I handed it off to someone else.”

Root Cause

Machine learning is fundamentally non-deterministic.

[illegible]ardrails AI

### Software APIs are deterministic...

The speaker uses a database API to illustrate responses that reflect the underlying data, then shows how predictable behavior supports systems combining databases, messaging and caches.

[2:49](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=169s) [A software API diagram shows repeated requests and checked responses.](https://assets.ai.engineer/sites/aiecode2025/sha256/9479cba387eebc00bccdbf47c93332aafcbc437e4402f67d4cd42e1dac5766de.webp)

Software APIs are deterministic...

Software API

[illegible]ardrails AI

[3:30](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=210s) [A software API returns checked outputs beside a diagram of a distributed software system.](https://assets.ai.engineer/sites/aiecode2025/sha256/87b83791c2c2cf4555b429488360ff863038ef8855a9f0c7caf14a4504acf9e6.webp)

Software APIs are deterministic...

Software API

Database
Replica
Replica
Load Balancers
Kafka
Notification Service
FCM
APNS
Cache

Guardrails AI

### ML model APIs are not deterministic

ML model APIs can return different responses to the same question. The crossed-out system illustrates how errors can compound when components rely on earlier outputs.

[4:05](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=245s) [An AI model API produces a mix of checked and crossed outputs; a large red cross overlays a distributed system diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/072463f516094f8e72c076b62c99ababfc39d07b2041b0271f8f911d8d0ffa8b.webp)

...but ML Model APIs are not

AI model API

Replica
Load Balancers
Notification Service
FCM
APNS
Cache

Guardrails AI

### Getting correct outputs is hard

Hallucinations, falsehoods, incorrect structure and prompt injection make reliable outputs hard. The speaker says developers have only the prompt to work with.

[4:21](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=261s) [The slide lists hallucinations, falsehoods, incorrect structure and prompt injection, and says the prompt is the only tool available to developers.](https://assets.ai.engineer/sites/aiecode2025/sha256/107530e16ea8f4b3b1e5b0d5569d2176f9f9f3be451acfa6bf38bcc3b62e6d42.webp)

Getting ‘correct’ outputs always is hard

Some common issues:
• Hallucinations
• Falsehoods
• Lack of correct structure
• Prompt injection

Only tool available to devs is the prompt

[illegible]ardrails AI

### Use of LLMs is limited when correctness is critical

Users can ignore or correct mistakes in interactive tools, but critical applications need more. The speaker asks how to add correctness guarantees while retaining LLMs’ flexibility.

[4:54](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=294s) [A projected slide says use of LLMs is limited when correctness is critical.](https://assets.ai.engineer/sites/aiecode2025/sha256/9708fa76f0efe8b0e6722db13a1960fc4a980b1c47db1ead111162e45d0603fe.webp)

Use of LLMs is limited
when “correctness” is critical.

Guardrails AI

[5:15](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=315s) [Slide states that LLM use is limited when correctness is critical and asks how to add correctness guarantees.](https://assets.ai.engineer/sites/aiecode2025/sha256/2444f0662f2482ec28e56678067459b21ece136a800b3973ee6c12a8d2038876.webp)

Use of LLMs is limited
when “correctness” is critical.

How do we add correctness guarantees to LLMs?

[illegible]ardrails AI
10

### Constraint checking and regeneration

Alex Graveley proposes checking LLM output against constraints, then feeding the generated output and rule violation back into the model to regenerate it.

[5:53](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=353s) [A post describes checking generated output against constraints and regenerating after a violation.](https://assets.ai.engineer/sites/aiecode2025/sha256/a16bec118a1edd4be35339aaf402c17fc7fbcc844d71918653ccd8a0d59faf79.webp)

Alex Graveley
@alexgraveley

Simple LLM technique that helps a lot (but you might not be using): add a constraint checker to ensure valid generation. On violation, inject what was generated and the rule violation, and regenerate.

Guardrails AI

### Guardrails AI project

Guardrails AI validates LLM outputs for structure, type and quality, then can reask the model when validation fails. The README describes a .rail specification and a lightweight wrapper around LLM API calls.

[6:28](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=388s) [A project README describes validation, corrective actions, and structured output for language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/71fa4bd5384a338c8014ca1bf836a9f7134907db4bc585f5e03010641716d20f.webp)

🛤️ Guardrails AI

discord  twitter

Guardrails is an open-source Python package for specifying structure and type, validating and correcting the outputs of large language models (LLMs).

Docs

Note: Guardrails is an alpha release, so expect sharp edges and bugs.

🧩 What is Guardrails?

Guardrails is a Python package that lets a user add structure, type and quality guarantees to the outputs of large language models (LLMs). Guardrails:

- does pydantic-style validation of LLM outputs (including semantic validation such as checking for bias in generated text, checking for bugs in generated code, etc.)
- takes corrective actions (e.g. reasking LLM) when validation fails,
- enforces structure and type guarantees (e.g. JSON).

🚒 Under the hood

Guardrails provides a file format (.rail) for enforcing a specification on an LLM output, and a lightweight wrapper around LLM API calls to implement this spec.

Guardrails AI
https://github.com/ShreyaR/guardrails

Interface text:
README.md
+ 99
Contributors 32
+ 21 contributors
Deployments 49
github-pages 5 days ago
+ 48 deployments
Languages
Python 99.8%  Makefile 0.2%

### Guardrails AI acts as a safety firewall around your LLMs

Guardrails adds verification after the LLM’s raw output. The diagram routes passing results onward and failed results back to a reconstructed prompt, with checks for risks such as PII and profanity.

[6:42](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=402s) [Slide states that Guardrails AI acts as a safety firewall around LLMs.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5a563778551b4436e634bf51d650bac3381082573faebdc65c070c2fbd547b3.webp)

Guardrails AI acts as a safety firewall around your LLMs

Guardrails AI

[7:15](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=435s) [A standard prompt-to-output flow is compared with a Guardrails flow that validates output, passes valid results and reconstructs prompts after failed validation.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb4ee788e32ca3bdf5decfaf916d99b79ea7b6150cbf894ea838f49cb16b2b06.webp)

Guardrails AI acts as a safety firewall around your LLMs

Application Logic
Standard
LLM Logic
Prompt
LLM API
Raw Output

Application Logic
Guardrails AI
LLM Logic
Prompt
LLM API
Raw Output

Verification Logic
Contains PII
Contains profanity
Mentions competitors
Code Executable
Summary similar

Fail Validation
Reconstruct prompt
Pass Validation

Guardrails AI

### Guardrails AI acts as a safety firewall around your LLMs

Guardrails checks raw LLM output for PII and profanity and whether generated code is executable. Outputs that pass continue; failed validation sends a reconstructed prompt back to the model.

[7:51](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=471s) [Standard LLM flow alongside a Guardrails verification flow with pass and retry paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/5bb2a5039a8756cdcbb7bb661d52c16afc6a94404fabfbe40ebf5184dfac2f38.webp)

Guardrails AI acts as a safety firewall around your LLMs

Application Logic
Standard
LLM Logic
Prompt
LLM API
Raw Output

Application Logic
Guardrails AI
LLM Logic
Prompt
LLM API
Raw Output
Reconstruct prompt
Fail Validation
Verification Logic
Contains PII
Contains profanity
Mentions competitors
Code Executable
Summary similar
Pass Validation

[illegible]ardrails AI

### Guardrails AI under-the-hood

A guard combines an output specification, an LLM callable, and instructions. It validates the LLM response, logs the process, and applies a configured action such as reasking or fixing when validation fails.

[9:08](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=548s) [Diagram of guard creation, LLM invocation, validation, logging and actions for invalid output.](https://assets.ai.engineer/sites/aiecode2025/sha256/c62e76103eaca35b6c6fbdf8cf2e53cadff428696f677ca3ef0b962523e9e6fa.webp)

Guardrails AI under-the-hood

Creating a Guard
Select type of output to validate
RAIL Spec
Pydantic Model
String
LLM Callable
Add prompt & instructions
Initialize Guard from spec
Invoke Guard

Calling a Guard
Guard invokes LLM API
LLM API Returns
LLM Output is validated
Logs
Invalid
Valid
Check validator on-fail action
reask
filter
fix
refrain
no-op
Return output

[illegible]ardrails AI

### Guardrails AI under-the-hood

A guard invokes an LLM and validates its response: valid output is returned, while invalid output triggers a configured action such as reasking, filtering or fixing. The process is logged.

[9:13](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=553s) [A guard is initialized from output specifications and optional inputs, then invokes an LLM, validates its output and applies the configured failure action.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5a77f442c6a52a935cb3c39e091e28c5fc1d8aa713c141550957daa1530a6c2.webp)

Guardrails AI under-the-hood

Creating a Guard
Select type of output to validate
RAIL Spec
Pydantic Model
String
LLM Callable
Add prompt & instructions
Initialize Guard from spec
Invoke Guard

Calling a Guard
Guard invokes LLM API
LLM API Returns
LLM Output is validated
Invalid
Valid
Check validator on-fail action
reask
filter
fix
refrain
no-op
Logs
Return output

Guardrails AI

[9:47](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=587s) [Diagram showing guard creation, LLM output validation, failure actions, logging, and output.](https://assets.ai.engineer/sites/aiecode2025/sha256/80bc17c119174a202705b3cec89dcd867c21034e23091e2c8533af02655a646e.webp)

Guardrails AI under-the-hood

Creating a Guard
Select type of output to validate
RAIL Spec
Pydantic Model
String
LLM Callable
Add prompt & instructions
Initialize Guard from spec
Invoke Guard

Calling a Guard
Guard invokes LLM API
LLM API Returns
LLM Output is validated
Invalid
Valid
Check validator on-fail action
reask
filter
fix
refrain
no-op
Logs
Return output

[illegible]ardrails AI

### What Guardrails AI does

Guardrails AI combines custom and reusable validators with prompting, verification, and re-prompting. Its specification language turns validation requirements into prompts for the LLM.

[10:55](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=655s) [Slide listing custom validators, orchestration, a validator library, and a specification language.](https://assets.ai.engineer/sites/aiecode2025/sha256/faaa7809f094b02d4a5c6e465453590bf2ae3b4c19aa9ac8f17b098802acf786.webp)

What Guardrails AI does

Guardrails AI is a fully open source library that offers

✓ Framework for creating custom validators
✓ Orchestration of prompting → verification → re-prompting
✓ Library of commonly used validators for multiple use cases
✓ Specification language for communicating requirements to LLM

Guardrails AI

### Why not use prompt engineering or a better model?

Prompts cannot guarantee instruction following, and LLM outputs can vary. Training or fine-tuning a model takes time and money; commercial APIs also typically limit control over model updates.

[11:53](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=713s) [Slide stating that LLM outputs can vary and prompts do not guarantee instruction following.](https://assets.ai.engineer/sites/aiecode2025/sha256/4c44bc87162dcab99561876cff5654ddb2a6680746c3acc9f35edba2be9d567c.webp)

Why not use prompt engineering or a better model?

Controlling with prompts (including

○ LLMs are stochastic: Same inputs ≠ Same outputs
○ Prompts don’t offer guarantees: LLMs don’t always follow instructions

Guardrails AI

[12:49](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=769s) [Slide lists stochastic outputs and imperfect instruction following, plus the cost of model training and limited control over API model updates.](https://assets.ai.engineer/sites/aiecode2025/sha256/b59062e040a88ed0e8925cc5031d9b00c62c9a7c5ef6c53238bf4e52a9e90ab5.webp)

Why not use prompt engineering or a better model?

Controlling with prompts (including

○ LLMs are stochastic: Same inputs ≠ Same outputs
○ Prompts don’t offer guarantees: LLMs don’t always follow instructions

Controlling with models

○ Expensive and time consuming to train/finetune a model on custom data
○ LLMs are hidden behind APIs – no control over model version updates, etc.

Guardrails AI

### Implementing Guardrails

Guardrails can use grounding, rules, ML classifiers, or LLM self-reflection. For code generation, the speaker illustrates grounding by connecting LLM output to a runtime containing application-specific data.

[13:55](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=835s) [Diagram branches from guardrails to external-system grounding, rules-based heuristics, traditional machine learning, deep-learning classifiers, and LLM self-reflection.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8e95ae8dfa8219df1293ceaf5507d98fa4afd5ae3b0658cc5af30c32358e31b.webp)

Implementing Guardrails

Guardrails

Grounding via external systems
Rules-based heuristics
Traditional ML methods
High precision DL classifiers
LLM self reflection

Guardrails AI

### Internal chatbot correctness criteria

A help-center chatbot should answer questions without hallucinating, using foul language, or mentioning competitors.

[16:14](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=974s) [A help center chatbot example with criteria to avoid hallucinations, foul language, and competitor mentions.](https://assets.ai.engineer/sites/aiecode2025/sha256/5929b86693286c4714525667a62fa33a29f33153de213d61b04ba34d5228a371.webp)

Example: Internal chatbot with “correct” responses

Problem
Build a chatbot over the help center articles of your mobile application

Correctness Criteria
Don’t hallucinate
Don’t use foul language
Don’t mention my competitors

[illegible]ardrails AI
18

### How do I prevent LLM hallucinations?

Provenance guardrails aim to trace each LLM response back to a source of truth. For retrieval-augmented generation, that means pinpointing where the response came from in the supplied context.

[16:32](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=992s) [A meme illustrates provenance guardrails alongside the claim that every LLM utterance should have a source of truth.](https://assets.ai.engineer/sites/aiecode2025/sha256/7eb63e35b6a94d495f50aebaa3c168faca3b9f3db20fb0cc96e5a49b2b13284b.webp)

How do I prevent LLM hallucinations?

Provenance Guardrails

Every LLM utterance should have a source of truth.

Meme text:
LLM
HALLUCINATIONS
PROVENANCE GUARDRAILS
LLM
HALLUCINATIONS
imgflip.com

[16:36](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=996s) [A meme illustrates provenance guardrails grounding LLM responses in a source of truth.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c6f3b4cf8bdad79459302e678c52a22d04cd8b4d6a4104e01fff195268e89a2.webp)

How do I prevent LLM hallucinations?

Provenance Guardrails

Every LLM utterance should have a source of truth.

https://docs.guardrailsai.com/api_reference/validators/#guardrails.validators.ProvenanceV1

Meme text:
LLM
HALLUCINATIONS
PROVENANCE GUARDRAILS
LLM
HALLUCINATIONS
imgflip.com

## Full transcript

[0:00](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=0s) [on-hold music] Hey, everyone.

[0:15](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=15s) Thank you for coming. I am Shreya Rajpal. I am the, uh-- I'm one of the co-founders and the CEO of Guardrails AI. Uh, and today, we are going to be talking about trust but verify, which is a new programming paradigms that we need as we're entering GenAI native, uh, application development.

[0:35](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=35s) Uh, before we get started, a little bit about me. Uh, uh, as I mentioned, I'm currently, uh, [REDACTED:username] Guardrails AI. In the past, I've spent about a decade or so working in machine learning.

[0:45](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=45s) Uh, previously, I was the machine learning infrastructure lead [REDACTED:username] Predibase, which is, uh, an infrastructure, uh, machine learning infrastructure company. I spent, uh, a number of years in the self-driving car space working across the stack of self-driving.

[0:58](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=58s) Uh, and before that, did research in classical AI and deep learning.

[1:05](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=65s) Awesome. So we're seeing this massive explosion o-- in, uh, AI applications over the last year. Uh, there's a lot of excitement and, you know, that is also why so many of you guys are here attending this.

[1:16](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=76s) Um, we have folks from AutoGPT, uh, which, you know, really took the world by storm and opened up the possibility in all of our minds with, like, what AI can do.

[1:26](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=86s) Uh, we've seen, like, a lot of really awesome applications in mental illness, uh, sales, uh, even like software engineering.

[1:34](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=94s) Uh, this is a relevant graph. This is basically search interest for artificial intelligence over time. Uh, and you can really see that peak, uh around where, uh, ChatGPT came out.

[1:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=108s) Uh, but if you think about, like, where a lot of the reality is or a lot of where the value lies today, uh, even though generative AI applications have seen fastest adoption compared to a lot of these other consumer applications, uh, their retention right now tends to be lower.

[2:05](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=125s) Um, so these, these are some graphs I, you know, uh, borrowed from this really fantastic article by Sequoia. And you can really see that retention for AI first companies versus the one-month retention, you know, for, uh, for non-AI first traditional software companies.

[2:22](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=142s) So why is this the case? Um, a common symptom, uh, that a lot of people experience as they're working with generative AI applications is, uh, my app worked while prototyping, but it failed, you know, the moment I tried shipping it out or even the moment like someone else tried te-testing this.

[2:39](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=159s) It just behaved very unreliably. Um, but the root cause of this symptom is that machine learning is fundamentally non-deterministic.

[2:50](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=170s) Um, for those of you, um, you know, we're gonna, like, dig deeper into what that really means. So I'm guessing that a lot of you here have worked with traditional software systems before.

[3:00](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=180s) So if you think about, like, a database and querying a database, uh, to get a question about how much was, you know, the spend of X user over the last month.

[3:10](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=190s) Every single time you hit that database API, you are going to get what is the correct response, right? And correct really means like representative of whatever your true data actually is.

[3:23](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=203s) So this is completely irrespective of like uptime, um, you know, um, uh, and availability, et cetera.

[3:30](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=210s) This fundamental property allows you to really build these, like, really complex software systems which, like, power our world today. Um, but if you think about, like, machine learning model APIs, this is not really the case.

[3:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=224s) Because of, you know, fundamental like, um, stochasticity that is, like, part of machine learning systems. Um, for a lot of you that have worked with generative AI systems and LLMs in the past, you'll see that even if you ask the same question across, like, multiple times in a row, you're going to end up seeing, like, different responses.

[4:04](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=244s) And the-- because of this, being able to build these, like, really complex systems, uh, that talk to each other, that rely on previous outputs, et cetera, becomes harder because you have this issue of, you know, like, compounding errors that really kind of explodes.

[4:22](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=262s) Um, this is just, you know, like, diving deeper into the problem a little bit. A lot of, like, common issues, uh, as you work with these problems. Hallucinations, that's a very buzzwordy, uh, thing that a lot of us here are familiar with.

[4:33](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=273s) But there's a lot of other issues like incorrect structure, uh, you know, their vulnerability to prompt injections. Um, and all of this is exacerbated by the fact that unlike all other previous generations of programming, the only tool that is really available to you is English, right?

[4:51](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=291s) It's just the prompt that you can really work with.

[4:55](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=295s) So, um, we end up in this scenario right now and in the current, like, time that we're in, where use of LLMs is limited wherever correctness is really critical, right?

[5:07](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=307s) Um, I love GitHub Copilot. It's on my badge as my favorite tool. Uh, but if GitHub Copilot is wrong, you just kind of like ignore it and move on.

[5:16](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=316s) Uh, same as like ChatGPT. The chat interface is really, really great because it's iterative and you can give it feedback and, you know. Uh, if it's incorrect, you can tell it, like, why it's incorrect, and it can, you know, maybe give you something that's more, uh-- that's more appropriate.

[5:29](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=329s) Um, but this is not the use case for a lot of, like, really high value critical applications. And so how do we add correctness guarantees to LLMs, uh, while still retaining their, like, flexible nature, uh, you know, that really, uh, uh, allows them to adapt so well to so many tasks?

[5:50](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=350s) Um, so I'm gonna add this, like, quick quote here by Alex Graveley, who is, uh, the creator of GitHub Copilot. It's a very simple idea, which is that, uh, "Add a constraint checker to check for valid generation.

[6:03](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=363s) Um, on in-- on violation, inject what was generate-- what was generated and the rule violation and regenerate." Um, so once again, we're trying to think about, like, how programming paradigms change as we are working with this fundamentally non-deterministic technology.

[6:17](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=377s) So this is something that, you know, wasn't, uh, needed for the longest time because we were working with, like, deterministic systems, but becomes very relevant now

[6:26](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=386s) Um, so interestingly, this tweet was actually pretty recent. Uh, but Guardrails AI, the open source framework that implements this and kind of like f- builds a framework around this strategy has existed, um, for a little while longer, uh, from the beginning of this month-- uh, from the beginning of this year.

[6:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=404s) Um, so Guardrails acts as a safety firewall around your LLMs, and this kind of fundamentally introduces, uh, a novel paradigm that once again wasn't as necessary in the previous generations of software development.

[6:59](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=419s) So this is what a lot of the software development like architectures for applications that you might build, you know, uh, may look like wherein you have like some application, and then in that application, you have a prompt that gets sent to an LLM, and then you end up getting like some output or some response back.

[7:17](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=437s) Um, this is the new paradigm that we propose, um, and that Guardrails kind of like, uh, implements as a framework wherein every output that you get back passes through a verification suite.

[7:30](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=450s) And that verification suite looks [REDACTED:username] all of the functional areas of, uh, you know, inconsistencies or risks that you are really sensitive to as an application builder, which may be very, very different from, you know, um, if you're building a code generation application whereas if you're building like a healthcare chatbot, right?

[7:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=468s) Uh, so maybe like containing PII or PHI, like sensitive information might be something you wanna check against or profanity, uh, filtering that out. If you're building a commercial application, you might really care about the fact that there's no mention of any competitors.

[8:03](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=483s) Like if you're, uh, building a McDonald's chatbot, like nobody should be able to get your chatbot to say that Burger King's the best burger in town. Um, making sure that any code that you generate is executable within your environment, uh, as well as, you know, summarization or free-form text generation is true and grounded in the source that

[8:22](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=502s) you know to be, you know, correct and not just hallucinated from the model. So each of these ends up being an independent check that runs as part of this like comprehensive verification suite that allows you to build trust in the models and the, uh, ML applications that you're building.

[8:39](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=519s) Uh, so the paradigm that we propose is that only use, um, large language model outputs if your verification suite passes. On failure, you can really hook into this very powerful, uh, capability that LLMs unleash, which is, you know, their ability to like self-heal.

[8:56](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=536s) Uh, which is that if you tell them why they're wrong, they can often correct themselves, and you can kind of go through this loop again if you have the, you know, latency budget or the-- even the dollar budget or the token budget to implement this.

[9:10](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=550s) Um, I'm gonna like go over this very briefly, but under the hood how Guardrails does this, um, is that it allows you to create what we call guards, uh, from, you know, different inputs.

[9:21](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=561s) So you can use like either a declarative model spec, uh, such as like, um, uh, you know, like XML or RAIL. You can use Pydantic models that implement like specific validation criteria and structure, or you can use string implementation.

[9:36](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=576s) Uh, you can create a guard from all of these components. If you want, you can add information about, you know, your prompt as well as the LLMs you wanna use.

[9:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=584s) Um, and then you create this [REDACTED:username] initialization, but [REDACTED:username] runtime, this guard will basically surround your LLM callable and then make sure that everything that you're sending in or getting out of the LLM is valid and correct for you, right?

[9:59](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=599s) Um, so for example, uh, if your output is valid, you end up sending the output back to your application. But if it's invalid, uh, you go through this loop of, uh, looking [REDACTED:username] which constraint is violated or which check is violated.

[10:13](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=613s) And then if on violation, uh, you have a set of these policies including like re-asking, which we touched on earlier, uh, filtering or fixing, which is programmatically trying to correct outputs, uh, falling back on some other system, uh, so refraining from answering or, you know, just no-op where you don't actively take an o- action, but you

[10:35](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=635s) log and store what the outputs of those checks or verification was and like why, uh, that particular check failed. And then you only do this like on-- uh, you only return the output once you know you can trust whatever came out of the LLM.

[10:52](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=652s) Um, so within this framework, what Guardrails AI does is it's a fully open source library, um, that allows you to, A, create custom validators. Uh, it orchestrates the whole va-validation and verification process for you, uh, to make sure that, you know, you're not taking on this like, uh, really kind of like often latency-intensive task of doing validation

[11:13](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=673s) and make sure that it's done as efficiently as possible. Um, it's a library and a catalog of many, many commonly used validators across a bunch of use cases. Uh, and it's a specification language that allows you to compile your requirements into a prompt so that like whatever specific, uh, validators you wanna use are automatically turned into a

[11:34](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=694s) prompt so that you know that, you know, those requirements are also being compu-- uh, communicated to the LLM.

[11:43](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=703s) All right. So a common question, why do I need this? Why can't I just use prompt engineering or, you know, a better fine-tuned model?

[11:53](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=713s) Um, so okay. So for some reason my, um, rendering here is weird. Um, but controlling the outputs with prompts, uh, including using retrieval augmented generation, which basically injects specific context into your prompt, uh, doesn't act as a guarantee, right?

[12:10](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=730s) Um, LLMs are stochastic. Even if you do all the prompt engineering in the world, there's nothing guaranteeing that those instructions will be followed. Um, we actually did this as an experiment for an unrelated thing where we used LLMs as evaluators.

[12:24](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=744s) Um, we ran the exact same experiment five different times, changing like absolutely zero parameters with zero temperature and saw like different numbers across our benchmark, which is, you know, really fascinating and wouldn't really fly in like previous generations of machine learning.

[12:40](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=760s) Um, and then second, prompts don't offer any guarantees. LLMs don't, you know, uh, always follow instructions.

[12:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=768s) Uh, the alternative is also like controlling, uh, the outputs with models. Uh, so first of all, it is very expensive and time-consuming to train a model. Uh, in my past life, this was basically what I've done my whole life.

[13:01](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=781s) Uh, and I was so frustrated with this whole process that I joined a startup, uh, where my job was to make this, you know, this process easier like as a function.

[13:10](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=790s) Uh, but it still requires like, you know, compiling a lot of dataset which is expensive, uh, training a model over a bunch of hyperparameters, um, and then serving it.

[13:19](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=799s) Um, and then if you are, if you aren't doing that and you're using like an LLM that's hidden behind a commercial API, uh, you typically don't have any control over model version updates.

[13:30](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=810s) Um, so I've kind of seen this where, you know, I, I mentioned like validations get compiled into prompts. So I've kind of like observed where commercial models will get updated under the hood.

[13:41](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=821s) Uh, and so prompts that might have worked for you in the past will stop working, uh, just over time.

[13:49](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=829s) Uh, so how do these guardrails work under the hood, right? Uh, there's no like one-stop sh- uh, one-stop shop solution for h- for a guardrail here. It really depends on the type of problem that you're, you're solving.

[14:03](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=843s) Um, so a very reliable way, if possible, uh, for, for implementing a guardrail is to ground it in an external system. So let's say you're working in a code generation app.

[14:14](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=854s) Uh, a really good way to generate more reliable code is to actually hook up the output of the LLM into a runtime that basically contains application-specific data. So we, um, tried it for a lot of text-to-SQL applications, which is something that is supported as a first-class citizen in, in Guardrails.

[14:32](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=872s) Uh, and we found that this re-asking framework where you hook it up to, you know, a sandbox that contains your ta- database and your schema, um, really substantially improved the correctness of the SQL queries that you got.

[14:45](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=885s) You can also use, uh, rule-based heuristics. Uh, so really looking into like, okay, if I'm, let's say, trying to extract, uh, an interest rate from a really long document, I, I always must know that interest rates, you know, end with like, uh, percentage signs, and so that can be a clue that I must always be retrieving.

[15:03](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=903s) Uh, you can try to use like traditional machine learning methods or high-precision deep learning classifiers. Uh, so really you don't need the full power of an LLM to solve, you know, really basic constraints.

[15:14](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=914s) So, uh, trying to find like, is there, uh, some type of toxicity in this output? Uh, does some type of output contain, you know, uh, advice that is harmful for my users or is misleading my users in some way?

[15:27](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=927s) Uh, you don't need, um... My favorite analogy to use is you don't need like a jackhammer to crack open a walnut. So if possible, you know, some of the guardrails should use like smaller classifiers that are much more reliable and deterministic, um, uh, a- instead of, you know, using LLMs.

[15:43](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=943s) And then finally, you can also use LLM self-reflection.

[15:48](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=948s) Um, all right, so we're gonna walk through this example of how this works in practice, uh, for building a chatbot, uh, where you wanna generate correct responses always. Um, so let's say you're an organization that has certain help center articles, and you wanna make sure that, um, you always generate, you know, y- you-- your users can ask

[16:09](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=969s) questions over those help center articles in a chatbot, and you always generate like correct responses, where correctness means no hallucinations, uh, not using any foul language, so don't swear [REDACTED:username] your customers, um, and never mention any competitors.

[16:27](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=987s) Now, how do you really prevent hallucinations? Like that's a very fundamental question, right? Um, provenance guardrails. Uh, provenance guardrails essentially mean that every LLM utterance should have some, some grounding in a source of truth, right?

[16:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1004s) Especially if you're building like retrieval-augmented generation applications. Uh, you make the assumption that, okay, I gave it this context. I hope it's using the context. What you wanna make sure is that every output that is generated, you're able to pinpoint to where in the context, uh, you know, your response kind of came from.

[17:02](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1022s) So this is one of the guardrails that, you know, is-- exists in our catalog of guardrails. Um, under the hood, there's a few different techniques that we employ. Uh, we use embedding similarity.

[17:12](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1032s) Uh, we also have like classifiers that are built on traditional NLI, like Natural language inference models, uh, and we use LLM self-reflection.

[17:22](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1042s) Um, this is a very brief, uh, um, you know, snippet of like how to configure a guard, uh, when you can essentially like select from this catalog which guardrails you wanna use.

[17:32](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1052s) So we've, we've used provenance, profanity, no references to peer or competitor institutions. Uh, and then you essentially wrap your, uh, LLM call with, you know, the guard that you've created.

[17:46](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1066s) Uh, so very briefly, let's say you get some question which is like, "How do I change my password on your application?" Um, you have like some prompt that you know is constructed from your retrieval-augmented generation application.

[17:58](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1078s) Um, but because LLMs are very, very prone to hallucinating, there's like-- it hallucinates where the setting exists for you in your, uh, you know, uh, in, in the response.

[18:08](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1088s) Uh, when this passes through your verification suite, the provenance guardrail will essentially spike and will cause, uh, the LLM to, you know, like go through this like re-asking loop, where a re-ask prompt will automatically be constructed for you via Guardrails, which will like pinpoint which part is hallucinated, uh, give it the context again and ask it to

[18:28](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1108s) correct itself. Uh, and then finally, the re-ask output, uh, you know, it tends to be more correct. And so we can kind of see here in this toy example, uh, that the output is, you know, corrected for you.

[18:39](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1119s) And finally, verification passes, and you can send this back to the output.

[18:44](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1124s) Uh, very briefly, more examples of validators that you can, uh, create or that exist. Uh, never giving any financial or healthcare advice, making sure that any code that you generate is usable, never asking any private questions from your customers or mentioning competitors, um, no profanity, prompt injection, et cetera.

[19:02](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1142s) Um, and then just to summarize what Guardrails does for you, custom validations, uh, orchestration of verification, uh, a catalog of commonly used guardrails, as well as automatic prompt compilation from your verification checks.

[19:16](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1156s) Uh, to follow along, you can look [REDACTED:username] the GitHub project, which is [REDACTED:username] [REDACTED:username]. Uh, our website with our documentation is guardrailsai.com. Uh, or you can follow me or the p- project on Twitter, uh, and that's for my LinkedIn.

[19:30](https://www.youtube.com/watch?v=9-vGxMoUM9Y&t=1170s) Awesome. Thank you so much, everyone. [audience applauding] [upbeat music]
