How to Build Quality Gates into Agentic Coding Workflows — Nnenna Ndukwe, Qodo AI

Um, so my name is Nnenna Ndukwe, and I am a developer relations lead at Qodo. We're-- We used to be just, like, an AI code, uh, code review platform, but now we're kind of expanding beyond that. It's all about code quality and governance across the software development life cycle. And I've spent a lot of time, um, thinking about and working on, like, what does a structured AI coding, uh, process look like? There are many different ways that we're all doing it. I

would love to hear from y'all, especially towards the end of this. But I'm just gonna have... What I want is to walk through what that can look like. There's a repo I have that everyone can fork, and if you do nothing else from this, ask your AI to analyze and grab some tips from the structure of this, the repo, and try to see what you can apply to your own workflow. All right? All right. So we're gonna walk through just a little bit of the beginning, who I am. Nnenna, um,

definitely passionate about, uh, AI for developers. This is my thing as a DevRel and as a former, uh, full-time software engineer. And I live in Boston, and this is my cat, Pippin. He's named after, uh, Lord of the Rings hobbit, not the basketball player which people ask me a lot about. And, you know, that's what-- this is what I'm here for, implementing structured AI coding workflow. Um, we're gonna be using, uh, Qodo. Um, you'll be able to sign into the platform. If you don't wanna use it for, um,

some of the earlier stages of the SDLC we'll walk through, you can just use it for, uh, PR code reviews.

And, uh, first things first, uh, if you wanna, you wa- you'll wanna have access to the slides, and I think the best way to do that or the easiest way to do that is to go to github.com/ my name, and then, uh, it should be... The repo should be at the top of the repository or, um, those are the slides, um, that will link to my GitHub or my, um, my Google Slides. So hopefully everybody can...

I'll give everyone a minute to do that.

So yeah, slides are here. The repository should be at the top of that. And somebody-- anybody raise their hand if they don't see it or can't get access.

Thumbs up if I'm good to move on or, or I'll wait. Okay.

Um, so in general, where and how code quality can break down. Uh, who, who here is, you know, using AI for coding? I'm assuming everybody. We have-- Do we have Claude Code fans here?

And Codex? Wow! Wait, less hands for Codex. Okay. I thought they, like, blew up in the past couple months. I've been using Codex for, uh, a lot of stuff. But, um, where can code quality break down in the agentic development life cycle? Uh, everywhere. Planning and design, development, code review, testing, deployment. There are so many potential failure modes for, um, for using AI for coding, just to ship high-quality software.

And that is where we come into this process where we kinda have to overdo it with steering AI coding agents to get what we need out of it, just so that we can get behind or stand behind the quality of the code and be like, "Okay, you know what? I can put my name on that. This is good to go. We can move this forward through the pipeline." Or, "I'm, I feel safe to have another developer review my code at this point." And so I really believe that we should be building quality-driven systems. I think that there needs to be a

verification, um, aspect to agentic engineering, and it really starts with, number one, defining what code quality means to you, uh, and your team if you're working on a team of developers. That needs to be defined, and then that needs to be... That needs to live somewhere. It needs to be codified so that our AI, no matter what interface you're using, can extract that information and do its best to adhere to the code standards that you have

in place. And then once you've decided where those standards live, whether they're locally in your specific individual repos or if they're in a context management layer, which we'll get into, then you can decide and pull that into the different interfaces. If that's your CLI, if that's your IDE, um, your, your CI workflows, it should be, uh, distributable.

And, uh, we do not have to go through all of that. Um, I already explained that. We might just be able to... Okay, well, here's what we're gonna do. We are going to operationalize our context. Um, I have a repo that has rules, it has skills, it has a, a, it has, you know, a bunch of different things in it, and we wanna leverage that for planning, for code generation, and code review. And the whole point is that we want to verify the quality

early and often. And ideally, it's the same standards that we are asking the agents to check against across each of those phases. That's, that's really the whole point.

Oh, these are my skills. I have so many. Um, and yeah. So I think, uh, I'm gonna go ahead and just jump into what we need to do. Um- So first, let's go to the Qodo Quality Workshop SignalPay. If y'all can see that. Oh, I can't increase the, uh...

Let's see.

Okay, so this is the repo that we're going to. It is Qodo Quality Workshop SignalPay. This is a payments app, and what I want you to do is fork this repo and let me know when you have done that, and then what you're gonna do is clone it. If you want to work through this on your own time, all of the instructions are there in the README, and I'm gonna be leaning heavily on my coding agents to actually help me implement this, you know, like, at a meta level.

So Qodo Quality Workshop-SignalPay. And in this repo, this is where we're just gonna walk through the process of planning for a coding change. We are going to read the repo rules to determine, okay, which rules are relevant for this particular coding change? We're gonna use skills and then make sure that we're writing proper tests using some TDD and BDD skills, test-driven, behavior-driven development.

Then we're gonna ask our coding agent to implement, um, the code change and then run local gates. We have linters, we have, um, you know, static analysis that we're running because we want some deterministic checks, and we want AI as well, like a good combination. And then we'll open a pull request and have Qodo, uh, review our PR in, uh, directly in GitHub. And after that, we'll actually review what the coding, uh, the issues were that Qodo

found, and we're gonna resolve that automatically with an agent skill. And what are we building? This is a self-contained FastAPI payments API. And, um, there's no front end or anything, so we can just, uh, go ahead and focus on it'll be like a, uh, it'll be so-- like an API that you can view in the UI.

Okay. Some things to look at with the repo structure. We have an AGENTS.md fi-file. We have a hidden plan directory, hidden, um, and rules, skills, docs. We have the actual API itself. We have tests, and we're using Semgrep as well for static analysis, and we have our CI workflow, like verification, um, process.

So first things first, we wanna run this app locally. So I'm-- We're gonna use the instructions make setup once you have cloned this repo, and like I said, you can walk through this on your own and just copy and paste the, uh, instructions. Let's see, where do I have...

Okay. Once we've done that, it should be make run, I believe.

All right, now it's running, and

locally it should be available, and this is the payments API that we have. Nothing crazy. Very simple. Um, not really the point of this, but, uh, this is what you should see when the app is running locally.

Okay. There's also some other endpoints available if you wanna take a look at that. Now, you can, uh, choose your lane if you want to just do this hands-on or if you just wanna observe. Here are the quality gate checkpoints that we'll be walking through and, uh, there's a whole setup gate process, right? We have these different commands that we're running, and we have a verification command that is, uh, pretty important. So you have this, uh, working already in your local repo, and if

you need help, ask your agent for, um, changing anything locally to make sure that this can run very smoothly. The next step we're doing is we are going to use Qodo, and, uh, what's great about Qodo here is that we're not just gonna run this in a pull request. We wanna set up our account with Qodo, so it can actually ingest the skills that are in this repo already and to ingest the rules that are in this repo. And this is going to be critical for us to be able to then apply that

as context to the code review phase as well, not just with your, your coding agents or when you're planning. Um, and like I said, you don't have to do this, but we can, um, we'll get to that part in a second. So some standards gates that are important to this are the rules themselves. We have different payments API-focused rules that are contained in this repo, and it is the agent's job to be able to determine when I have to implement this code change that you're requesting for me,

uh, I need to determine which of these rules are going to be relevant for this, so I can adhere to them. They're all here in this, um, in this repo, as well as the skills for implementation. Um, and then we have the planning and test-driven development gate. There is a high-level plan in, uh, planning template in this repo, and then we have an execution-focused plan. And this is very detailed templates because from what I have found is the more detailed that you

are, uh, relatively speaking, uh, the, the more detailed that you are or consistent and structured with your planning, uh, phase and the document that your agent has to go back to and read, then it's going to adhere a bit better and also correct itself when it realizes it's missing some, uh, critical points of your, uh, process.

And you wanna be able to identify what is the expected behavior, what are the skills that I'm going to use for implementation. All of that is here as a part of the planning template. And then we have the local verification gates. I use Ruff, Pyright, Bandit, pytest, Semgrep, and, uh, conventional commits, deterministic evidence that the changes made locally preserve the local contract. I am a huge believer in

combining deterministic tools with, uh, AI. And then the last gate is the pull request review and the remediation gate. When we open a PR, Qodo's findings are inspected, fixes and deferrals are documented, and then we can rerun our verification command after we fix those, um, coding changes. And that is going to be a process of completing the code review loop, um, on your own and as opposed to just stopping right there at that

local, uh, passing that local run. Here are the prerequisites. We need Git, GitHub account, uh, GitHub CLI, and Python, uh, uv, Node.js, and npm, and any coding agent that you use, preferably Co- Codex or Claude Code, and then we will get our Qodo account.

Okay. So here are the setup, uh, instructions. I'll let y'all do that, and we're going to sign in to the Qodo app. This is app.qodo.ai, and that's it. Um, if you can sign in here, you'll be able to get set up to connect the GitHub application to the forked repository. So app.qodo.ai. Um, use, uh... Who here has a Qodo account already? Does anyone? Okay,

great. Um, so yeah, sign in and sign in using your GitHub email. I think that's the easiest way. And, uh, and you will be able to, uh, you'll land, let's see, here at the integrations page or the homepage, and you'll need to add a connection, which will be the GitHub Cloud, uh, connection, and it'll take you through the installation guide. I recommend not connecting, like, all of your repositories, just the forked repo. It

should be listed under there, but let me know if it's not. So if I click on GitHub and the installation guide, connect your organization. Um, I'm already connected, but it should be listed right here for you to add. Um, this is just an example. Like, we're using Qodo as an example of whether you use Qodo or not, you'll want skills and rules to live somewhere. Ideally, if you're working on a team, it's gonna

be outside of your local repo, unless you have a good use case for containing them in that repo and it's, like, hidden or something. If not, there are many... Uh, there are a few context management layers that are out there. I think Unblocked is one of them, and that's... It's just a separate place for you to manage that context at, at all times that you should be able to pull down wherever you are. Does that make sense? Mm-hmm.

And co- u- us using Qodo is just an example.

Okay. When you've signed into the Qodo portal, you've connected GitHub to your fork, then you'll need to generate an API key from the Qodo portal, and you can do that down here by clicking on API keys on your profile and create an API key and name it.

You'll probably wanna save this API key somewhere, um, locally so that you can then add it to qodo... .qodo, uh, /config.json. So generate the API key first, and then you'll wanna save it, save that through, um, this, uh, JSON structure. And again, if you have access to the repo, you can have your coding agent do all of this for you.

We also have Qodo skills. You can use npx, npx skills add qodo-ai/qodo-skills/skills. If you don't wanna use it, the skills are already local in, in the, uh, repo.

Okay. This is the task that we're gonna work on, a payment workflow safety process. The task is to add one small payment workflow. It's going to be either a refund workflow or a capture retry handling workflow. You can choose which one you want, but this is an intentionally small change, right, with, um, some potential risks kind of built into, um, the code or, or it's a sensitive aspect of this, uh, this payments API. And, uh,

payment mutations are where a lot of AI-generated code can look, like, plausibly correct but can introduce some issues. So I think it's a perfect example for us to walk through. Um, and yeah, we're just going to use this, uh, all of this text really. We're gonna have our coding agent use all of this to do either one of these tasks, refund workflow or capture retry handling.

And here is where you can copy-paste the agent prompts to get started. First, it's like help me configure this repo for the workshop. I have a Qodo API key from the portal if you're using that, and then here are the requirements for getting set up locally. Um, I am going to... Since I already have this part, I'm just going to copy and paste install or verify Qodo skills, read AGENTS.md and rules, and select the relevant repo local rule IDs for this

hands-on task. And I'm doing that in, uh, Codex. I wish I could... Let's see.

So I'm in my, uh, I'm in the project, and I'm just copy and pasting, getting my agent really ready to start working on this project, giving it that context that it needs.

And raise your hand if you have any questions of, of where we're at at this point, or if you want me to slow down or go back.

Can I see your initial prompt? Right here? The initial prompt. "Install or verify Qodo skills." We're reading the AGENTS.md file and the rules, and then we are asking it to take a look at the repo rules for the hands-on tasks that we're about to implement and explain why each of these rules will be relevant for the upcoming task. And then I have optionally compare those rules with Qodo get rules if those are--

those skills are available, if you want to download the skills.

Okay, let's see. What is it saying? "I'll ground this in the repo's own instructions first, then verify the local Qodo setup and dependency health before selecting the rules." Okay, it found, uh, it's aware that we're inside the workshop project.

And it is-- has also identified that I have the Qodo skills.

Oh, the repo local workshop skills are not installed yet, so I'm gonna run the repo's installer, which will only copies committed local skill folders into the agent skills directory. Okay, great. So it's gonna process the local skills that we have.

And I'm gonna let that run. I'll go back up to where we are at. It's the setup prompt.

Let's see if it's made progress. After we are clear with the agents and they're good to go, then we're gonna switch into plan mode. And so for this planning, uh, mode, we're saying, "Use the local-- the repo local workshop planning skill to turn the hands-on task in the README into a high-level implementation plan and a build session execution plan." And then we want it to read agents, read the rules, and, uh, use

any relevant skills to determine what are the constraints. And this plan must include an implementation skill handoff, and that names the first, uh, uh, that names the first implementation skill and the test-driven development prompt that it's going to be chained to and run next. Do not write the code yet. I'm a big fan of plan mode, but if you don't wanna use plan mode, it still has this prompt to, uh, to make sure that you're not gonna implement anything.

It is taking a while. Well, that's because I have all the verification stuff set up.

All right.

Let's see.

All right. Okay. It's taking a while

to do this. Where do you tell it which half you want to take? The refund or the capture retry? I'm probably gonna do the capture retry. Uh, I've... Well, what about you? Well, I was thinking the refund, but where do you tell it that you're doing that? Uh, it should prompt you and ask you. Okay. Um, and if not, then you can make that clear as a part of the prompt.

And what point are you at now? About to run the plan. Okay. Awesome. Okay. Finally, it's done, verified the repo instructions. It ran the install skills, um, instructions or, uh, command. All of the local skills are now pulled down, and here are the selected repo local rules for this payment workflow task.

So these are all the rules that are going to be considered as a part of this, uh, process.

You know, maybe I'll do the refund, uh, workflow with you as well. I'm gonna copy and paste the planning prompt and switch into plan mode.

Um,

it should ask me, but I will say I will work on the refund. What was it? What is it called? The refund.

Uh, here we go. The refund workflow

plan.

Okay, let's see. So I'm gonna use the repo local workshop plan from task skill and the payment idempotency skill, and then ground the plan in AGENTS.md, README, and the rules. I'll only read and inspect files. No code or plan files will be written in this particular mode.

So now Codex is assessing the current behavior of the app. We obviously-- we don't have this implemented, um, but it's determining, okay, what needs to be done, what would take-- what would be the shape of the code change that needs to take place. This is, uh... I should probably dig into the templates actually right now as Qodo is working so that you can see what the planning templates look like.

So here are the planning templates. I truly believe that this is a really important artifact to be able to take with you if you're not already using a structured template. Um, I think that a plan should answer what behavior are we changing, which rules apply, what skills would guide the agent, which tests prove the expected behavior, and what kind of quality gates should pass before we review.

And it starts with a template for a high-level plan and then a build session execution plan. And for this workshop, this is what we would expect it to s-look somewhat like.

And here are the templates. We have a summary,

the problem that we're solving, any particular constraints that we need to worry about, and then we have skill routing. This can be helpful if you need to plan ahead of all the skills that you would expect your coding agent during the implementation to start kicking off one by one, especially if you wanna use some type of loops, um, or, like, goals i-in Codex or Claude Code to work on your behalf without you stepping in. And we have a scope, behavior scenarios, and verification gates,

failure and recovery rules, and a commit plan. This allows for a very structured process for, uh, your coding agents to work on very specific tasks, organize the code in a nice way that is presentable, make those commits in an organized way using conventional commits, and then m- get to a point where you can actually submit a pull request and everything is organized the same way that one would working on a team back when we didn't have AI.

And then the build execution template. What I've noticed is super helpful is having a documentation of the starting state. What branch are we in? Um, what is the current local verification process? Are there any commands that are expected to run to determine that, uh, this code, uh, passes or fails, um, any of the expectations with linters? And, uh, and then of course, this is the implementation skill handoff and

execution steps. Um, I want the-- I wanna know that the agent knows every single step in this process of making the code change, and I want it to determine what the test plan is, um, as well. And then it's definition of done or completion notes. You need to make sure you've run all of the commands, uh, for this, uh, for this branch. You need to make sure that you've identified what was deferred, if you've changed your mind at some point. Everything is documented in this

plan.

And now we've got one, the refund workflow workshop plan. We've got our summary. We have the repo rules that are associated, the high-level implementation plan, and the build session execution plan.

And also, there are some assumptions here. Um, who here has arrived at this, at this point?

Okay, cool. So I'm gonna go ahead and implement this.

And while this is happening,

we can go to some of the slides.

So what I was saying about operationalizing context. Planning codegen and code review, this is what I found to be incredibly helpful for consistency across the SDLC. I use a combination of different agent skills, and it depends on the repos that I'm working in. It depends on the programming languages and frameworks that I'm utilizing. But at a high level-- And so you'll see some Python skills here at the top, like Python Pydantic Pyright or Python typing, uh, contracts. Um, a lot

of these skills are available, uh, all over the internet, or you can have your coding agent build it themselves. And there are some ones that I've gotten from, from some popular, um, developers out there. Um, and those are one f-ones from Aaron Francis are the audit skills. There's audit boundaries, drift, audit names, audit abstractions, dead code, and auditing errors. I found that these are incredibly powerful for really simplifying and improving the quality of

code during the implementation, um, phase. I don't know if he has these publicly available, but if you can, um, look these up or find them or have your coding agent create them yourselves, this is definitely a step forward in, like, the right direction for quality.

Um, as I said, enforcing quality early and often. So one of the values of using skills and the Qodo get rules here, this is about pulling down those rules from your context management layer and then determining, um, which, uh, which rules are gonna be relevant for the task. And then I believe in then running local code review workflows. I don't care what- You know, tool that you use, if it's, uh, Codex, if it's Claude Code, their built-in code review,

if you're using Qodo, whatever it is, always run a local code review because there are always these small changes that you could probably make earlier before another developer has to see your code, and it's worth it to be able to address that sooner rather than later. We're trying to mitigate, um, issues, um, popping up. And then we're running automated code review in CI or Git provider, which is by default. And when I did this, um, I walked through this entire workflow for an

open source tool that I'm building, and the proof was in the amount of lines that I changed in a large PR. It was intentionally large because I wanna test, wanted to test the, the quality of this actual workflow itself. And only one bug was uncovered in over nineteen hundred lines of code, um, using, uh, Qodo for that, like, automated code review. And it was one reliability bug, which made a lot of sense. Um, but this was a really good exercise for me to see that this-- there

was value in this process.

Yeah. So I, I, I wish I had, like, a graph I could, um, uh, show you of how I have, like, higher level, uh... I have, like, planning templates that I'll use for, like, no matter what repo I'm in, here is the planned structure that I expect. And then there's overall code quality, like, at the highest level, like, engineering standards, um, that are specific or that are-- that can apply to any programming language. And then, like, below that, it's, okay, all the repos

that use Python, here are all the skills for that. And, like, you, you kinda have to decide what would be most relevant depending on your use case. But the more general a skill is, then the more access it-- you need to have, it-- other-- many repos need to have to it. Does that, does that make sense? Um, take a look at this, uh, for example. Uh, let me make sure I'm

scoping this just to my,

uh, repos. Oh, okay. Okay. Well, take a look at this. We're in the Qodo portal, and this is the rules section. So across multiple repos, there are rules that were inferred and ingested and saved here in the portal, and they are divided by-- They, they are-- There's, like, a, a rule name in general and, like, all the details that are generated here based off of the

context from your repository. And then they're divided by categories. There's accessibility, correctness, maintainability, architecture, security. Um, and then they're also, uh, organized by level of severity. Not every, uh, not every rule should be treated with the same level of severity. There are some rules that are strong enough to where you wanna block a merge. There are others that are just, like, it's good practice, but this isn't something that it would be detrimental if it were to get through.

Um, and yeah. So this is-- these are some of the ways in which, uh, tho-these rules can be categorized, and I believe they can also be scoped to specific repos if you want to get granular or if you want Qodo to infer that, uh, itself or whichever tool that you use. You can actually design that process yourself to scope those rules so that they-- rules or skills so that they only apply to very specific

repositories.

Can I do, like, code owners of the repository?

Can you, can you-- What was that?

Can I do, like, code owners of the repository?

Like code owner and then the repos-

I don't think so. I think what-- the only thing you can do is, like, filter on that in this dashboard. Um, but that's a really good question. I'll probably bring that back to the product team to, to verify. Um, but what brought up that idea?

I just-- We have multiple teams in the organization, and there's a lot of diversity across the teams who run the projects end to end. And so I'm like, "Oh, if the team has to have code owner of the repository, that would be easier than scoping repositories-

The repositories

... repositories team.

Right. They're, they're divided right now by org level and by, um, repo level, um, naturally or natively.

That wa-- Those were good questions. Yes.

Yeah. So I'm actually really interested in Qodo rules and some of that feature.

Yeah.

Um, could you talk a little bit about, um, how Qodo decides to create a rule, um, and kind of categorize it? Because, um... So I guess what's in my head is, like, you know, depending on how autonomous you are with your agentic coding, you can be hands-on and you know every single line of code top to end. Maybe you're really hands-off, right? But I would imagine engineering would probably want to create a unique set of rules for this product, but it seems to be kind of automated. Is that right?

So I'm just curious how it decides, like, if it's gonna be able to apply more rules than-

It can be-- And so great question. How do rules become rules? Like, how does Qodo determine, um, what is a, a rule or what isn't because it seems very automated. So you can... I-it depends on how you've organized things already. If you have coding rules that are literally in a rules, uh, you know, directory, it's gonna be very easy for Qodo to be like, "Okay, every single one of the rules that I see here, I'm just going to copy and paste, like, most of those contents, um, in and, uh, ingest it here

into the portal." But there are some rules that will show up as, uh, suggestions. And this is from, specifically from reading or analyzing the actual code base. Um, so you might not have explicitly written these rules into your code, but these are the conventions that Qodo has noticed based on analyzing the entire code base. It will not automatically create a rule for it, but it will come up as a suggestion for you to be able

to review, and this is what will be generated. You can edit all of this. There are examples of good code examples and bad code, um, just to make it very clear to AI agents, uh, what good and bad is. And then you can define the category and the, the warning levels. And then once you're happy with the way that this looks, you can then accept it, uh, and activate that rule, and it'll automatically be applied.

Is it okay to ask a question?

Yes, keep going.

Okay. So on the other side of that, um, how does, I guess, the skill you're running in Codex decide which rules are relevant for that task? Like, what kind of a, kind of a search, I guess, for the various skills?

Um, for Qodo? It's... Or-

Well, I-- So I saw, like, in your Codex session, there was a point where it was like, "Okay, these rules are relevant from the Qodo get rules. I'm going to use these in this session." Right? Exactly that. So, like, how did it select those rules? Did it say, like, "Oh, I'm just pulling all of them"?

Mm-hmm. Um, I think for, for this, it's just pulling in the local, uh, repo rules that are m-most of them, if not all of them, are associated with the different paths to take in this workshop. Um, but in general, like, the Qodo get rule skill is talking to the back end of the Qodo platform, and it's sort of like this agentic, uh, RAG process of determining, um, what, which ones are actually relevant, and it filters

it down.

Okay. So it, so it is like a RAG-based-

Yeah. Mm-hmm.

Thank you.

That was a really good question. Uh, okay. So the plan was implemented, and I don't know why it created a draft. I like real PRs, but fine, whatever. Um, copy the link.

All right. Add refund workflow with quality gates. Did you get there? Uh, okay. Awesome. And here we've got the summary. I'm just gonna go ahead and make this a real PR because what I want is for Qodo to review this

without merging the pull request.

If Qodo is connected to your repo, this is what you should see automatically pop up. It's going to start the analysis process. And of course, I'm not looking at all of the code because we don't have all day to do that. Um, but the-- you understand the point of, uh, implementing these processes. So in a few minutes, um, Qodo will be done analyzing this, uh, code and will bring up any potential issues, and we're gonna take a look at what those are.

Um, as a result of any potential coding issues that come up, we're gonna run an agent skill to automatically resolve it.

Are there any other questions? Those were really good. Yes.

Yeah, I have a question, and a, and a question that follows from it is, something I've seen with skills is they can be, like, super general, like, "Write DRY code," and, like, "This is what good code looks like." And I have arguments with engineers where I'm like, "This is super general. It should be in the agent's weights by now," and, like, "I think you should wait until something pops up." But to settle these arguments, it would be really useful to have data of, like, this skill in the metric shows that it's actually useful. Does, does Qodo offer anything like that? Or how do you know these skills are actually valuable versus just, like, taking up space?

Um, Qodo is just ingesting the skills. It's not evaluating, uh, the quality of them. But there's a tool that does do that, Tessl. Have you heard of Tessl before? T-E-S-S-L is a, um, company that is all about skills evals, and you can determine-- It will tell you, are there any security concerns with this skill, or is this actually as effective as you expect it to be? That's a tool that I would recommend for, like, integrating that to, uh, to determine, to determine

that.

Thank you.

Yeah.

Okay. Um, a PR summary was generated automatically. We've got a description, and we've got a diagram of the changes. And it's showing the legend API in-memory store in the function. We've got the API client and the FastAPI app and the different, uh, let's see, the endpoint of the payment refund endpoint that's pointing to four different, uh,

fields.

Um, so one of the things that I would recommend, but I, I realize I just skipped over, was the verification prompt. So we can, after doing a, you know, a, a process of creating the, the-- making the code change and actually doing a local review, we run a verification prompt to ensure that everything passes. But Codex already did that automatically on its own, um, just because of the way that the repo is, is actually set up. But this is-- would be a

best practice for walking through that process.

And any day now, Qodo would be great.

Well, we can take a look at, uh, a p- PR that I've already created, and this was a PR where I didn't run any of the processes that we were just talking about. But I do have the context, uh, pulled in through the Qodo portal, and this is something that, uh, highly consider you looking into when it comes to that context management layer that I was talking about. Here in the code review, Qodo

identified that there were rules in the Qodo- Qodo portal that are relevant for this actual code review. We're not talking about planning or implementation anymore. We're talking about code review specifically. And then it pulled in also the relevant skills, um, for this, uh, this code change.

And here, there were some action required, uh, results that came up. There's a rule inside of this, um, inside of my org where Python functions must all have a docstring attached to it. And so this was shown as a rule violation and an issue around maintainability. The only reason why it's action required is because that was determined by our engineering org that this is of high-- as high of a severity. Because you can customize the level of severity of your

rules, it doesn't have to be something that is block-worthy. It just needs to be something that is addressed. That is up to you. And here is the evidence that's associated with it. Require docstrings for all Python functions, or I would say methods. Yes.

So, uh, I think you said at the beginning of the talk that the agent obviously like fixing, uh, uh, workflows that are deterministic.

Yes.

And so when we're pulling rules like this, it's a core level rule. Is that still like a RAG type retrieval, where it's looking in the agent to evaluate this rule applies? Or is it saying, is it deterministic saying that this rule applies to all the code or experience?

Yeah. Um, so this is being determined right during the code review phase of, okay, which rules are going to be relevant? The part that can be deterministic is that you can decide based on the level of severity of the rule, this, uh, code, this, uh, branch or PR will be, will be blocked. And so that's something you can configure in the Qodo portal is in the case of any action required or high severity issue, uh, is identified,

we block the merge until you have fixed it and Qodo will run again and verify that you have.

But in terms of making sure that an organizational rule is applied, that's still there's a potential for this error to be introduced.

Uh, th-there's a potential that it won't determine that the relevant rule will actually... Yes. Um, and I, I'm really glad that you brought that up because it is something that we have talked about with the product team, um, and the more, uh, enterprise customers that we speak to, the especially in highly regulated industries, they really want determinism in exactly that fashion. Um, so that's something I'm hoping we can push for, for making a real thing, 'cause I think it's extremely important.

I agree with you.

Sorry, I have a question.

Yes.

After, uh, it says, do we run verification also because the code is, is changed?

Yeah.

Do we run again, uh, rule violation-

The-

And find out, uh, there is-

You mean this?

Yes.

Um, after I've-- you've resolved issues?

Resolved issues, yes.

Yes. I think it's best practice to, just in case. Um, you, you never, you never know, so.

Is it done automatically?

Um-

So what-- how many iterations you have, you don't care for it?

You're-- I think that's more on your, your agent might be the one that will determine like, "Oh, I'm gonna run verification or not," and you can prompt your agent to do that anyway, um, regardless. Um, but yeah. So it depends on how your, your agent kind of responds.

Thank you.

Mm-hmm. And finally, here are the results. Oh, okay. Five skills were invoked. Uh, all right, so we've got one bug, three rule violations, and two skill insights. There is an issue with the refund idempotency stored in memory. So refund payment uses the process local, uh, dictionary to store and replay refund results, which is not durable

persistent storage. And we have the piece of code that is related to that, and we have the evidence as well, the rules and the following files. Um, the important part here that y'all should definitely play around with is copy and pasting this agent prompt that is meant to fix the suggestions that, uh, pop up. Um, you can do that manually for each of the issues that present themselves, or you can run the Qodo PR resolver

to automatically resolve. Did you already run that, that skill?

Uh, run that?

Mm-hmm.

Yeah, but we have implemented in our workspace-

Mm-hmm

... something really similar, so, uh, it doesn't want to decide. I want an error fix to be again rerun and, and, you know, whole code or PR be reject because it can happen that if you fix something, you can break something else.

Absolutely.

So it's important if we have any kind of automation to include verification sec- section again.

Mm-hmm. Yep. I think that's best practice and that's, um, that's what I would do or I guess pre-gen AI era, um, just to be careful, and that's what you want your agents to do as well.

So that was a really, uh, good point. Um, after seeing all of the results in this, uh, PR, then that's when I ran the Qodo PR resolver skill, and I ran it against the pull request number four, and I saw that it's pulled down all of the issues that Qodo has already identified from critical to high. And where did it go?

Oh, okay. So it pulled down all of the results, and now I can decide: Do I wanna review each issue individually, or do I want to auto-fix all? Um, for the sake of this, uh, workshop, I'm just going to auto-fix all, but, uh, this is something if you wanna be a bit more hands-on and make decisions with your agent to actually plan how you're going to make the fix, you can do that yourself with the review each issue, uh, um, option.

Auto-fix all.

Now, let's see how it reacts. So I'm gonna auto-fix all six Qodo issues, and what should happen is... Okay, and you can see here it says then it's gonna run verification, and it's gonna make the, uh, commits, and it won't be-- the commit will be blocked if it's not structured in the correct way automatically. Um, but this is that point where it'll do that after. And it'll take that next step further by

actually posting back to the PR and, uh, with a summary of all the changes that were fixed. And from there, that's how the PR will be updated, and all of the issues that were resolved will be, uh, crossed out.

How are you handling parallel workflows when you have, like, touching the same

files?

I, I mean, I've done-- I've used worktrees for, for that. Uh, have you-- is that something that you've used? Yeah.

That's kind of how I do it. Um, but I will ensure to do that a separate way, especially for, um, new Qodo.

Mm-hmm. Um, yeah, that's not something that Qodo really handles. Like, that's something I f- I found the coding agents kind of handle themselves. That's a good point, though. A good question.

Any other, uh, questions before I wrap this up?

Yes.

So right now, it's going to the last stage and then doing the commit. Should it be able to run it before you push it to GitHub?

Yes, you can run a code review, and I, I didn't run that for this because time was flying. But, um, it-- you mean for Qodo or for your coding, your coding agent?

Qodo.

So we are coming out with a Qodo review local workflow, and it will talk directly to the Qodo back end. So all of the agentic capabilities that you would have in a pull request, you would also have locally and be able to address. Mm-hmm.

So that pull request, does it stay private or does it push to GitHub?

W-what was that?

So when it needs to review it, can it-- can it run it? Does it push to GitHub? How is it reviewing the complete second code in relation to the first one?

How does it review all of the code? It clones the repo.

Okay.

Mm-hmm. Yeah, that's the process that we do on the back end.

It's like a temporary cloning just so it has an understanding of, yeah, the, the broader scope of the changes beyond the code diff. Really good question.

Okay, I think we are about, just about out of time, so I wasn't able to go through, um, everything in detail. But like I said, uh, if nothing else that you take from this, take this repo and ask your AI some tips on what kind of verification steps that you would want to implement or you would wanna practice implementing locally. Um, and I think that that would be a, a really good major step to just put, add-- automatically add more structure into

your workflows and consider any tools. There are some, uh, open source tools that have similar context management layers to what Qodo does. I think, um, uh, there's, there's one of them out there. But if you look it up, you should try practicing having rules and skills and other, um, engineering standards, whatever it is stored elsewhere that you're, um, across so that you can pull down that information, um, when relevant to your-- the stages of your workflow.

Oh, yes. It's called Packmind, I think. packmind.com. So definitely look into this if you want an open source solution for, um, uh, governing your AI code, um, through different rules and things like that, uh, consistently across, um, your teams and your repos.

Okay, I wish the internet was faster, but it's not. So I, I, I don't-- I can't, uh... This never happens, but here we are.

Uh, let me know if y'all have any questions. Um, I'm, I'm here, I guess, for the next couple minutes.

Thank you.

Thank you.

Let's...

Yeah. Question.

I did

Do you typically do that or do you try to, like, clear and, like, start a fresh session?

I'm at a point now where I don't clear... Like, I let the, the, the context, like, compact, and I haven't really had issues like I would've maybe three, four months ago with the, I guess, the advancements that they have made. I just don't experience much degradation.

Okay. Like compact, you just kind of let it do its thing?

Yeah, I do. Um, but if I really feel like it, I'll just create a new, uh, a new session and maybe copy and paste some of the stuff from the previous session into it.

So how are you here?

Yes, we have a, a massive booth here, so-

Thank you

... yeah, check us out.

I, I assume all this runs in the cloud. Do you have, like when it does the, the, the code checking and everything-

Um, Qodo?

Qodo is what-

Yes, yes

... you were talking about. Do you have any local options, uh, for security reasons if you don't want your code running maybe in Amazon?

Yes. We have, like, uh, on-prem deployments and p- pretty much all deployment options that you would want, um, because we do work with, like, banks and, you know, things like that, and, um, companies like, uh, some teams at NVIDIA. There's, like, no other option but to do that. So yeah, we do that integration.

Okay. Thank you.

Mm-hmm.

Thank you.

Let's see. Where did... Oh, there it is.

And here is the fix summary.

It did finish in time.

Yeah, it did. I'm, like, not even, I don't even know what time it actually, uh, it is right now. And how many things were resolved? All of them. All issues were resolved. Awesome. Okay. Thank y'all. Uh, if you have any questions, I'll be out there.

Yeah, girl. You crushed it.