BDD, ADR, PRD, WTF: Capturing Decisions for Humans and AI Alike — Michal Cichra, Safe Intelligence

[upbeat music] Hi, I'm Michal.

Uh, welcome to Capturing Decisions for Humans and AI Alike.

Yesterday, with a team from Safe Intelligence, we have released Pact.27, a new product to test agents. Before that, I was in Microsoft, Red Hat, and spent ten years working on a single product.

The consistency problems we face with AI and the story of capturing decisions show up in every product I have seen, and these notes are distilled from that experience. And you can find me at the booth.

Um, so BDD, PRD, ADR, like this lot of acronyms, uh, why does any of it matter? So let's unpack it from the end.

You probably know this story. Uh, I hope it's not an urban legend, uh, but scientists put five monkeys in a cage with bananas on a ladder,

then gave them a cold shower every time a monkey tried to get a banana.

Other monkeys beaten up the poor fella. Then they replaced the monkeys one by one, and none of the originals remained, and yet they have beaten up every monkey that tried to climb the ladder, not knowing why.

So humans and LLMs, they suffer from the same trait: limited context. People forget. LLMs, context compacts. Humans leave. LLMs have no memory. After a while of operating a product, the team starts asking, "Why do we have this flow?

Why is this goal of this feature? Why is this code shaped like that? Why-- where does this belong?" And you might not have the founding engineer available to answer.

And these problems show in every org, uh, maybe with AI much sooner than they used to.

So ADR is Architecture Decision Record. It records why you do something and how you enforce it or how you wanna do that. And you can cover examples by reference docs and code snippets.

For example, we split code in layers to prevent N plus one queries. We enforce that split by linting imports in modules. And we also enforce reading from database, returns, um, plain shapes instead of ORM objects, so we cannot, um, cannot make these, uh, these queries and to prevent duplication.

And also linting it by module imports and another like fifty ADRs that define the architecture of the product.

There is not a single format that, that you need to use. It's just a concept. Um, it's a text, so there is no specific, uh, way how to enforce it.

You still need a tool to enforce it. But the tool will tell you that this is the rule, why are you doing this, and how are you supposed to fix it.

Then the agent will go and try to find this document, why this reason exists, and more information about how to fix it. Also, you can define, like, which files it actually concerns to, like, is it some Python files or some folders, and how you actually enforce it.

PRD is a Product Requirements Document, uh, that's something lighter. When you're building a feature, you describe why that thing exists and what problems it solves and how user goes through the app to actually interact with it.

What's the journey through the application. It can be very light. It doesn't need to be really long and exhaustive, like a massive document. Uh, you can just capture why the problem and the goal and the journey that connects them.

And it's not just for the agents, but also for you six weeks from now when you forget why you did that.

Now BDD, um, it's a Behavior-Driven Development. You have probably seen Spec-Driven Development lately, uh, but if you practiced it, uh, you might have suffered the same thing as me.

How do you validate that the product actually adheres to the spec? It's a markdown document. You describe how it's supposed to work, but how do you know it actually works like that?

One thing harder than reading an AI code is reading AI tests. Um, so what if you had an intermediate layer that actually describes how the product behaves in a human language?

And BDD is not new and shiny, but it's, it can be executable and readable,

so enter Cucumber. It's almost forgotten, suddenly useful again. It's definitely easier to review than your average test.

You can connect scenarios directly to your PRDs and critical user journeys. It can be readable, executable, and it closes the loop that a Spec-Driven Development leaves open.

These rules, uh, these specs are later parsed by steps, and they are executed as code. But what you can do is that you can actually write and read these, and you can review these, and you can understand these.

The language is on you. It doesn't need to be, um, enforced. Like, you-- There are multiple ways how to write these, uh, these features. And

that's it. But they describe how you're supposed to go through the application, why this thing exists, and how it runs. And similarly, they can refer back to all the documents that you have about why things exist.

So, and as a bonus, making consistent UIs with agents is just another level of hard. Like design system and pattern library are the way to build consistent UIs. Like that was the way before AI, and it is the way now.

So you document your language. You say, for example, "A primary button is this and that. It is blue, it has this shape, it has this color, and it's this size."

And you say your rules. You say, "We will have only one primary button visible on a site at any, on a, on, on a page at any point in time."

And then you can enforce these rules. Similarly, you define components and patterns. So for example, if you have multiple colors of these buttons and multiple states, you define components and you define previews, and you demonstrate how they work, and you create snippets of previews so you can actually see them, and the agents can see them.

And then you can go and review and like, "Do these actually adhere to the principles that I have? Do they adhere to the visuals?" And then you reuse them.

As with code, you build these from the ground up, from small pieces into bigger ones. You compose them, and you reuse them. Otherwise, it's, uh, chaos, like with the code.

So cool. These are cool ideas, but how to actually enforce this?

So my team and agents stick with it. How do I keep it consistent? Well, with the loop.

So you probably have heard about closing the loop, reinforcement loop, the harness, how to remind the agent of their rules and how to follow them.

So our loop is simple. It is Git hooks, skills, CI, and linters, and bunch of other checks. Agent's goal is to deliver a pull request, and they, to do that, they need to use Git.

So we use Git hooks, Git hooks to run predefined tasks, and these tasks are later executed on a CI. They are the same tasks that they are executed as, as hooks.

If, for example, agents would get lazy and not wanna execute them or skip them, then they get caught.

And we include linting, formatting, type checking, code duplication, architecture checks, document li- linting, everything that's, that's possible.

So there was a time where code reviews were about style and tabs and spaces, and there is no space for that anymore. All these things are not for discussion.

They are rules, and they are enforced, and they are automated because there is no space for discussion about these anymore. It's more about the high level concepts.

What you cannot find, you cannot enforce. So for example, we enforce architecture of the product and of the code. We separate modules, um, and their imports, so what you can use from where.

For example, our end-to-end BDD test suite cannot access database, so we forbid from accessing any module that could access database and basically force the module to, the modules to iterate without database and really use only the browser features of the application.

Similarly, in the product itself, we enforce we cannot talk to database from rendering templates, so we know that there are no N+1 queries ever. We just define ways to prevent these problems from happening ever.

You cannot keep finding them. You need to prevent them entirely.

Then the com-- the agent tries to commit it and push it, and they get feedback on the commit and get rejected, and they get linked back to the document, and they go read it and fix it and iterate.

So there are some drawbacks. Um, it is... Oh, sorry, it's not drawbacks. Um, so this loop is generic, uh, this loop where they do some work, they, they push it, and they get feedback, and they iterate.

But the loop can be multiple things, right? Like sometimes you're working on a product feature, sometimes you're working on a UI, sometimes you're working on more backendish sh- like backendish things.

So that loop is the same, but what changes is the focus of the loop. So we have different skills. Uh, there is ADR that whenever there is an ADR mentioned, the agent will look up ADRs, how to operate with them, uh, how to find code that affect, that's affected by these ADRs.

For PRD the same. For w- for UI loop, we actually skip bunch of checks and rather force it to iterate in a browser quickly.

And test skill that actually identifies tests to run based on code coverage and file changes, so we run just the focus part of the suite and not the entire suite.

And some goal execution to actually keep decisions that the model made so we can review them later. But all of these provide focus in the loop, but the loop stays, stays the same.

There are drawbacks. Uh, it is very context heavy. Like you can run out of half of the context, uh, in, um, starting the research. Um, but I have no fear of context compacts like this actually for like last half year actually works, I think.

So in my sessions, there are twenty, fifty context compacts, and it's, it's okay because the important things survive, and the agent will always look them up again. So and that's the goal anyway, right?

Like you want to have multiple hour sessions with a clear goal that agent can operate autonomously with the rules that you define. So that's the goal anyway, like.

So there are decisions that you can record. There are parts of the product that you can describe why these exist. There is, um, Cucumber or BDD that can have executable specifications that you can actually rev-- read and review and understand.

Design systems can help you to build consistent UI from components and, um, again, enforce it. That, for example, there are no inline styles anywhere else. And you employ harness to loop it all together.

So may the spec be with you. That's it. [applause] [outro jingle]