Catch the Bugs Your Code Review Misses - Killian Carlsen-Phelan & Taylor Luttrell-Williams, Sonar

My name is Taylor Luttrell, and I'm joined here by Killian Carlsen. Together, we make developer content at Sonar, the maker of SonarQube. But today, in particular, we are jazzed, dare I say, to get y'all up and running with Gitar so that you can both find and critically fix issues in your agent-generated code before they ever merge. Now, before we dive into the hands-on demo itself,

if you would please direct your attention to one of the screens on either side of this room, we've provided for you a QR code and also a link there below, which once followed, you can use to set up a free, I should note, free fourteen-day pro trial for Gitar. So this is gonna be really important if you want to follow along step by step and play by play with this workshop. You're gonna need Gitar, so we'll give y'all a minute to get that set

up should you like. And just in the meantime, while you do configure that, if I could get a quick show of hands from the audience, who amongst you is already using an AI code reviewer as part of your CI process?

Okay, so that's a fair number of y'all, and that's what we like to see. And whether you are or you're not, we hope that you will be able to derive some utility from today's session. We'll give y'all just another minute as you sign into Gitar and set up that trial.

Next, I have for you another QR code, and I promise this presentation is not exclusively QR codes, but this one is in fact very important, as this will take you to the dedicated GitHub workshop repo that we have created so that you can get up and running with Gitar. This project is going to help exercise some of Gitar's incredible capabilities, and Mr. Killian here is going to walk you all through that in a minute. But as you get set up, I do wanna note that

everything we do in this session is going to occur in the browser. So you're not gonna need to set up any local tools, install any local environments. All you need is a GitHub account and access to Gitar to follow along play by play. Uh, just a couple of housekeeping notes as y'all are getting signed in and set up. Uh, in this workshop, we're gonna guide you through a few different things, namely installing Gitar as a GitHub App. Gonna need access to GitHub for that. And then critically utilizing Gitar

across a few different pull request scenarios. We have three prepared for you in this demonstration that are gonna demonstrate how verification with Gitar goes beyond simply code review. And don't worry, this is very beginner friendly if you're not especially technical. We're gonna start slow with Gitar's out-of-the-box verification, and then we're slowly going to ramp and build into some of the more advanced features. We're actually going to create a project-specific

configuration that teaches Gitar your project's conventions. And then lastly, we're gonna round things out with a CI failure diagnosis, and you're gonna watch some automated fixes occur directly on your PRs. So if you follow along with Gitar and with this repository, then by the end of this workshop, you're gonna walk away with Gitar installed, three verified pull requests, as well as a practical understanding of a number

of things, such as, but not limited to, verification instructions, rules for your repositories, as well as CI failure analysis.

I hope by now you're all set up with Gitar. You've logged into GitHub. Go ahead and refrain from forking the repository right now, as Killian's gonna walk you through the steps, uh, specific for our configuration. I wanna start instead with a scenario that I'm sure many, if not most of us, are intimately and also frustratingly familiar with, which is we all know that AI coding agents are wildly effective

at writing code, both at speed and also at volume. But while these agents are supercharging our teams to write more code and much more quickly, we ought to pause and ask ourselves who or what is actually verifying the code that our agents ship. And whether consequently, the code that our agents ship is not only secure, of course, and maintainable and reliable,

but also whether it is practically and not just functionally or technically correct. As we know, agents, as they're generating more code, and they're doing so faster than anyone can realistically inspect, a couple serious issues have begun to arise. The first is an obvious review bottleneck. We have teams with a fixed number of members, but which are now facing

an increasing number of changes, uh, via their AI agents. They're being inundated with code to review, and as a result, queues are growing, merges are sitting, waiting, and unattended to, and reviewing is essentially becoming, uh, a job, a chore, if you will, that nobody wants to handle and simply doesn't have the time or the resources to tackle. Number two is we're noticing an increase in the number of production incidents. As incidents scale with the

rate of change, given that coding agents Do not always respect the project architectures that you spend so much time devising, uh, your team or your company or organization's coding standards or any of the guidelines that you've established. They're unaware of those things. They lack the proper context. And with these issues threatening to undo the very progress that coding agents promise, the answer has to be faster than,

um, anything that humans can perform, and also better than human review, in order to keep up with the rate of changes that are being introduced by these agents. And so what we have for you is Gitar, which is the code change verification agent, and it stands out apart from and above the rest in a number of ways. Namely, it functions with a high signal-to-noise ratio, in effect minimizing

the false positives, the number of false positives, in order to, uh, improve the fix rate and reduce token expenses as well. It diagnoses CI failures, it's really unique, uh, and it separates real breakages from, uh, flaky noise or infrastructure noise, and it also, crucially, fixes these issues accordingly, and it fixes autonomously also, so that you can move forward as quickly and as freely

as possible. Effectively, you choose your level of automation, and you also choose your level of involvement. We're giving you the reins. Gitar also iterates until all of your changes pass CI to be absolutely certain that you can ship the code that your agents are generating. And lastly, it provides a high degree of transparency into the review process and your CI through a number of dedicated analytics dashboards so that you can

continuously improve the speed, quality, and effectiveness of your code delivery. So in essence, and, and just to recap, Gitar is designed for platform and engineering teams, uh, who need to ship code with confidence at agentic volume and also agentic velocity, and it complements SonarQube with an understanding of each code change's context and intent.

And Gitar leverages a variety and a, a wide body of context sources to accomplish this, suggesting educated and informed fixes from such things as PR diffs and details and comments and whatnot, um, from the CI, uh, CI logs as well, the, the very logs that I personally hate to, to have to sift through, um, from project-specific instructions as well, um, configurations that you can write

yourself in a repository rules, and, and much, much more, and we're gonna dive into some of those during the session, uh, today. And I should note that for anybody who is worried about noise, as I mentioned, uh, Gitar is tuned for signal, uh, because we understand that every false positive is a token spent and a fix attempt that risks an unded- unintended change. Um, and rather than adding more noise and burdensome decisions to an already complicated review process, Gitar

does do the work of getting the change through, and in the process, verifying and fixing everything so that you don't have to yourself. At a high level, Gitar's review engine is triggered when a PR is opened, and it calls upon that wide body of context that we just spoke about to produce findings and also fixes posted directly to your pull requests as actionable guidance that you can

interact with with comments, or if enabled, automatically fixed and merged. And to demonstrate just that, I'm gonna pass things off to Killian, who's going to walk you through the hands-on component of this workshop. Should you have any questions, if you get blocked, feel free to raise your hand or call on me. I'm gonna be walking around the room, and I'm happy to assist should y'all need any help. Thank you.

Cool. Thank you very much, Taylor. Um, hey, everybody. I am Killian. I am a developer content engineer at Sonar. And, um, we're gonna hop into the hands-on section of this workshop. And before we do that, uh, the first thing, like Taylor mentioned earlier, is we're gonna wanna have the Gitar app already installed. So if you haven't done that yet, please go to app.gitar.ai and sign in from there, and then start the GitHub

install. And the second thing is right here I have the workshop repo. It's at github.com/sonarsamples/AIENewYorkWorkshopGitar, but you can find it under the Repos tab. And the first thing that we're w- gonna wanna do is just go ahead and fork the repo. And make sure that the owner is the organization that you've installed the Gitar

app to. And you can name it anything. You can keep the name, you can name, you know, test, run, whatever works for you. But the most important thing is that this right here, copy the main branch only, is unchecked. There are multiple branches for different exercises I'm gonna walk through, and if you only copy over the main branch, you're not gonna get those different exercises. So I'm gonna go ahead and click Create

fork.

And once that's done, the next thing that we're gonna wanna do is hop into Actions. Right. By default, GitHub disables workflows from forks, and in order to set up some of the things that we're gonna do, like automated merging and approvals, you're gonna wanna enable that. And then the last thing is that if you wanna make sure that Gitar is able to read this fork, you can go into the Gitar dashboard, um,

go to Settings, go to Configurations and Repositories, and you should see your repo pop up there. But okay, just to hop into this exercise. I have here Box Office, which is a fake sample ticketing API for a small concert venue. And it's, it's a pretty simple Python p- uh, project. We have orders.py here, where somebody is able

to purchase a ticket or cancel it and get refunded. We also, um, have this, uh, check-in.py, and what this does, it just handles the logic for somebody to hypothetically be able to scan a QR code and get someone into a concert. And this is a pretty basic, uh, example application, but there is a test suite here, um, that cover all of the existing, uh, functionality, and every time a PR

is opened, uh, the test will run. And so what we're gonna do right now is actually open pull requests for each of the three branches that are in there. And the reason for that is because Gitar can take a few minutes to run, and instead of having to wait for each review to happen sequentially, we're gonna open them all up and then go through them step by step. One thing to note is make sure that you set your base repository

to, um, what you are trying to do. Uh, set your, set your base repository to the fork that you have made. So I accidentally hopped back to the base repository, but yep, right here. So I'm gonna open new pull request.

Oops, I kept unchecking it.

Yep. So you're gonna set that to your, um, run right here, and then click sequentially Stage 1, Stage 2, Stage 3. So yep, create pull request here.

Gonna set up another one.

Pick Stage 2.

Create pull request.

And then open up the third one.

Again, make sure that this is opening against your fork and not the upstream repository.

All right, so all three should be running. Gitar should be running its review, um, while we wait. And so let's hop into the first PR right here. Um, this is just going to be Gitar's out-of-the-box review. We haven't done anything. We haven't configured anything. This is just when it looks at a PR, what it suggests. And what this change does is we are adding the functionality for someone to order a ticket and then transfer it to

a friend. So if I buy two tickets and I send one to my friend, we should both be able to go to a concert, hypothetically. Now, I have included in some pretty basic validation here that just makes sure, like, hey, the ticket's not found. If you don't have a valid email address or it's a voided ticket, it's not gonna let you transfer those tickets. And, uh, some tests for that as well. And the test for this, uh, PR pass, the workflow is green, the

CI is green. Um, and one thing that I wanna point out here as well while we wait for the review to land is that while Gitar is, is reading this, it's analyzing in a different way than more traditional static analysis tools or other tools might be doing. Where static analysis like SonarQube is incredibly important, and it catches security issues like SQL injections to where, you know,

traveling through 20 different files and functions, it's able to identify, um, you know, SQL injections before they land in your database. What it can't do is infer intent, right? By looking at your descriptions, your titles, at your existing code base compared to the features that you're adding, it cannot infer that, hey, the business logic that is being attempted, uh, to create here, um, doesn't add up in

the way that it should. That's why we need both layers on top of each other. They cover different capabilities, and both of them are complementary towards having a fully verified application. We need to be making sure that our code is completely verified as it scales in volume and speed, otherwise, um, outages and, and breaks are almost guaranteed. So, um, I'm gonna hop back into the main conversations

here, and we see that Gitar has added a finding. So anytime Gitar finds an issue- It'll add a comment, um, at the line that it finds an issue, and it also posts a code review that has all of its findings as well as descriptions and suggested fixes and, um, provides a nice summary at a high level. If we read this here, it says, "Buyer can refund and void a ticket after giving it away." So

when adding this PR, right, when you buy a ticket and then send it to someone else, if you go ahead and cancel your original order, it gets canceled for your friend, right? This is an edge case that wasn't accounted for, and there was an even a test that almost caught this. There was a test that said, "If you have a voided ticket, it won't let you send it," but not the reverse. Um, and it also identifies right here that the issue is in the

original ordering file, orders.py. So the, the-- if you were just a reviewer looking at the diff, you wouldn't necessarily see in the code that you're adding that there's anything wrong. You would have to have working knowledge of how cancellations already work in your application in order to understand what's being broken. Now, this issue only multiplies as, as we do more and more

agentic coding. The surface area of applications gets more and more complex, um, and in order to maintain an understanding of how every last function works is almost impossible at the rate at which we're coding. That's why reviewers like Gitar are absolutely necessary, because it compares all of your features against existing functionality, even when it's not readily of, um, apparent in the PR diff. And so

awesome. Gitar's posted its review. Now what do we do? It's very simple. You can either click this Apply fix right here, or you can just comment Gitar fix, right? Comment that. What Gitar is then gonna do is add a new commit to the PR. It will continue to iterate and re-verify the new commit blindly. So if any side effects in the workflows happen to occur,

Gitar will keep working, will keep iterating at no extra cost per PR to make sure that it gets fully green by the end.

So Gitar is working on that extra fix, and what it will end up doing is just fixing in that orders.py, um, file and adding extra validation for this. Just takes a couple minutes, and it automates this entire process of diagnosing a break, re-pushing, and then fixing and verifying your commits every time. I am, I am really passionate about the fact that we not--

cannot be shipping unverified code. This is absolutely essential, and we need to be verifying every PR with context, with verification, with both deterministic and, um, a logical understanding of what's being done here. So while this fix lands, it'll just update, uh, the commit here. We'll see it pop up in the files

change diff. Um, it will analyze and make sure that the workflow goes green, and, um, it should be ready to merge. But I'm gonna move on real quick into the second pull request, where we're gonna cover context.

So every repor-- or every team has extra rules and, uh, business logic that doesn't necessarily get represented in the code. So that could be something that lives in your senior engineer's head or in a wiki or in a Confluence page or something like that. And these business rules don't necessarily have anything to do with code correctness. So for this PR right here, we are adding in a three dollar and fifty cent service fee for every

ticket sub that's purchased. So if you buy a fifty dollar ticket to a concert, at the end, you're gonna be paying fifty-three fifty. But for this hypothetical company right here, they have a rule that says every price we show a customer is the all-in price. I don't know about you, but it is the biggest pet peeve in the world for me when I go and buy a ticket for something and it's like thirty dollars, and then with fees and everything like that, it's like seventy dollars by the end of it.

Um, this hypothetical company cares about user experience. They care about you. So they have a business rule that says every price we show is the all-in price. But how do we enforce it? How do we enforce that this is gonna get reviewed by a code review agent? Well, there's a couple different ways. One is this, uh, review directory, .gitar/review/ directory, and another one is this .gitar/rules/ directory. So what a review directory

does is you can add all the context you want, as many MD files, as much of your business logic as you want to add, and Gitar will steer its own reviews according to this.

And this is super powerful, by the way. Like, I encourage you on your own, just throw as much context as you want at it. Uh, Gitar is incredibly efficient and powerful at ingesting this and applying it. Uh, for the rule section right here, rules are awesome. I set these up all the time. Rules are if a certain Uh, criteria are met by a PR, then a rule will fire and take an action. So for this one, is every time and some of the

pricing files are edited or updated, it will check and present to a reviewer a list of, "Hey, these are all the customer-facing fields that are edited," and it'll let you know these are all in or otherwise. But you can do a ton with rules. Um, one that you can do is if, let's say, you and another engineer are talking about, uh, a new capability or feature in a Slack thread. You can set a rule that says, "Hey, every time I paste a Slack link in my

PR description, go into Slack, look at the conversation, and check my code against the discussion that we were having." And it'll just fire every time that you do that. It's super cool. I love this capability. Um, and if we scroll down, oops, a little sneak peek of what Gitar found there. But we can see that, um, there's this, you know, service fee that gets added and some tests. If I hop back into the main conversation,

we can see that Gitar posted its code review. It posted the rule, right? It sees all of these pricing fields, and it lets you know, ooh, this is, is or is not an all-in price. It finds the bug. Now, again, guys, this is not an engineering bug per se, right? If you're just the world's best programmer in a vacuum, it's, it's not like this is incorrect, but according to your business and your use case,

this may or may not be what you wanna do. So Gitar is able to check against this. It is so powerful. We're not gonna go ahead and merge this one in. Um, we're just gonna leave it before hopping into the third exercise.

Oh, and just, um, the Gitar fix, uh, ended up going through, and it's been fully verified from the previous exercise that we did. Um, Gitar, uh, approved it, fixed it, and we can see that, um... Let me look at the diff. That it went ahead and fixed this orders.py file and added in this extra check. But moving on to the third exercise, and

what we're gonna do here is we're going to enable full automation and merging, complete pipeline verification from the moment you open, uh, your PR to the moment that it gets merged in. And there's gonna be a little bit of settings that we have to adjust here, so if you wanna follow along, I highly encourage you to. If you just wanna watch and set this up later, the README for this repo has all of the instructions available for you. Um, so the first thing that we're gonna wanna do is hop into

Rules, scroll down, and click Allow auto merge. That sets up the GitHub setting that allows for Gitar's auto merging. Next thing is to go to Rulesets. Hit New ruleset, New branch ruleset. And for ruleset name, you can just say, you know, workshop-rule or whatever you want it to be. Make sure it's active, and make sure that

you target default branch. And the rule that we wanna, uh, set here, the check we wanna set, is Require status checks to pass. And what that will do is if you type in test here, is that we're going to require our CI to go green, our test to pass before we say, "Hey, this is good to go ahead and auto merge." Again, if, you know, you missed any of those steps, uh, this is in the README,

um, with all of those different requirements, uh, listed there for you. So go ahead and create that, and if your company has secure auth set up like mine does, it may enforce, uh, a code which I have right here.

Awesome. Cool. So we've created a new ruleset.

So you may see already for this PR, it has been blocked. A test has failed. But what this is doing is it is essentially refactoring the way that tickets are scanned. Um, what it's saying is that, "Hey, we're gonna update tickets now to point to the ticket page URL and refactoring a little bit of the logic that already existed within the application." Um, but the, the, the test failed, and if you go into that and you look at what

failed, it says, "Test ticket cannot be scanned twice." So it caught a check that was missed in the refactor. Um, if I scan my ticket, that should be one and done. I don't want to continue scanning over and over and let all my friends use my same ticket behind me. It's pretty basic, but it's easy to miss logically.

In Gitar's, uh, review, it is able to inspect CI. It says, "CI failed." It says, "One test fails at the gate check-in. Scanning an already used ticket a second time returns two hundred." So if you have any failures in your pipeline, Gitar is able to inspect the logs and fix. One of the, I guess, biggest annoyances in my engineering career is, like, the test log inspection loop, where you have to go

in and see what failed and then try and diagnose and then, you know, fix it locally and then push it and then wait for the pipeline to rerun, and some of these pipelines can get really long. And then you're just waiting, and then it fails again, and it's- Just this time-consuming, money-burning process, and you may even need to, like, keep pinging another rev, uh, engineer about reviewing, and I'm getting a headache just thinking about it. Gitar just solves all of that for you. It is able to automate that process and keep

going. So it's very simple. All you have to do is type Gitar... Sorry, I'm doing this with one hand, is Gitar auto apply on, and then Gitar auto merge on. And what auto apply is gonna do is just allow Gitar to keep pushing fixes without needing you to, to click yes on it every time. And then what auto merge is

gonna do is whenever it doesn't detect any more issues and whenever your pipeline finally passes fully, it's just gonna go ahead and merge that in for you. Um, cool. I'll add that.

Gitar auto apply on, Gitar auto merge on. If I refresh it, Gitar... Oh, it already did it. Hit it with a nice thumbs up. Thanks, Gitar. Um, and so Gitar will iterate, push a fix, validate it, verify it, and then eventually merge it in, and this will happen within a couple minutes. This is beyond powerful, right? To be, to be able to set up your application. And by the way,

adding the comment like we did is just for the sake of this workshop because I didn't wanna auto merge in the other PRs. You are able to go into the Gitar dashboard and turn on auto-approve and auto-merge for any of your repos that you want. You can be selective. You can choose the level of automation and approval that you set up for each repository. If you have an incredibly agentic workflow with tons of guardrails and tests and a sophisticated pipeline, um, and then you add Gitar on top

of that, you're just able to let these rock and roll right off the bat. So I'll refresh this.

Gitar's still working, um, but it will continue to go and push until everything is fully merged.

The repo as well, uh, I encourage you to continue working on it on your own at home. There are some extra things at the end, um, that, that are more advanced capabilities for you to check out. Uh, Taylor's gonna talk about a few of them, but you have a 14-day pro trial that I encourage you to use on this repo, that I encourage you to use on your own repositories. Um, I think you're gonna get a lot out of it. Um, so instead of, uh, holding y'all

hostage to watch Gitar add its, uh, auto merge in as it's iterating and applying those fixes, I'm gonna pass it over to Taylor to talk about a few more features.

Well, thank you, Killian, for the insightful demo. I hope that we were able to help y'all get rocking and rolling with Gitar. Just wanna pop back up to discuss a couple more advanced features that Killian has just mentioned, uh, which he touched on but whose walkthrough would go beyond the scope of this short and simple workshop. So first, note that at the bottom of the README in the repo is a Take It Further section, as he just mentioned. I would encourage y'all to follow

that after this workshop. Follow it at your own pace. Take it home. Um, but to quickly highlight, Gitar does enable some additional features including but not limited to, number one, functional validation. As Killian mentioned, it checks each PR against the ticket, in the case of Jira, that it implements. And this is really powerful because it ensures that the change is correct and complete and not just clean, right?

So it works with Jira currently, Linear. It works with Slack as well, which you can connect through the Gitar dashboard at Settings and Integrations. For example, you can reference a Slack message or a Jira ticket to better inform and structure your reviews with Gitar. Gitar also takes advantage of a feature that we're calling Knowledge, which learns conventions and also references false positives from the feedback

that it receives and the feedback that you provide to it, and it's weighted by reviewer seniority actually, which is pretty neat. Um, and this shows essentially the rules behind every review, where they come from, and why they're coming into play. There's two that I would like to show y'all today. The first is cross-repo, and the other is cross-PR analysis. Cross-repo analysis finds PRs that may very well merge cleanly

independently, but, um, but break code in another repo, right? And there's actually no limit on how many repos it analyzes. Just food for thought. Um, and then there's cross-PR analysis as well, which checks in-flight PRs against each other. As you probably know or may very well have experienced, I certainly have, two PRs can each pass alone and yet still break production together, and that's

just a risk that grows with every agent you add into the stack. So Killian's walking you through the dashboard, um, just to quickly touch on each of those features. Um, if you would, perhaps we could click into a single one of these findings. Note that it points you to the fix, shows you where this is occurring, and importantly, why it actually matters, right? Why you want to defer to, uh, to Gitar to go about fixing it.

And that is Gitar in a nutshell. We hope that y'all have found some utility from this. Um, again, feel free to take this home should you like to better familiarize yourself with the tool. Play around with all of the context that you can throw at it to get better and more informed reviews. Um, there are a couple that come with the, um, enterprise feature. All of this is going to, to ship with, with Pro. And so we'll open the floor now to any

questions. Should y'all have any questions for us, we'd be happy to answer those. Thank you very much.

Yeah. So Gitar has MCP connectivity. It's able to ingest, um, context like that in a variety of ways. Um-

Configure your own integrations, yeah

... I-- That, that sounds a little bit different from the knowledge capability that we have. That sounds more like the context capabilities, where it'll review, you know, code against, you know, like architecture guidelines and things like that. The Knowledge tab that we have is, um, steering Gitar-specific reviews. So if Gitar posts a review and suggests a fix in a certain way, you can say, "Hey, we actually tend to fix stuff in, you know, this type of way or, or XYZ manner," and Gitar

can save that for its own fixes in the future. That's a great question. I'd have to follow up on you about how exactly knowledge ingestion works. Um, but I, I, I believe, actually, actually I know for a fact that there are ways to get that connected. Whether that's in the Knowledge feature itself, I would have to follow up with you about that.