Redesigning How Software Gets Built With AI Agents — Sonar & McKinsey Panel

Um, we'll get started. I'll introduce my fellow panelist in a second. But just to set the context of what we are going to talk about, um, you know, in the last six months or a year or so, almost every company has been trying to deploy AI or agents in their workflow. Um, we did some research on this, and I'm sure it's the same for a lot of organizations around the world that they have seen this. Basically, eighty, ninety percent of the companies that you ask, they say, "We have deployed AI or agents in some shape or format." But when you ask them how many of them have actually

scaled or how many of them are seeing business impact from it, it's less than a third. Um, right? And there are a lot of reasons behind that. We have seen a lot of companies doing it in POCs and small environments, but not actually across end-to-end, um, organizations. So what gives, right? Like, what was the reason why that doesn't happen? Um, we worked with Sonar and Tariq to figure out what are the key factors that actually helps our organization scale if you're deploying AI in your software development lifecycle. Um, and we saw some really

good results. Not surprising, right? Like, anybody who is doing this at scale is actually seeing pretty good results in terms of PR throughput, PR cycle time, um, self-reported productivity gains and tasks in the SDLC. Um, and we put these organizations on a scale of maturity, essentially. So if you think about where companies are today, most of them are in what we call Horizon two, where you have human-guided and validated agents that are executing tasks across SDLC, but very few companies are on the

Horizon three of the right-hand side, where it's fully end-to-end automated workflow, right, with human oversight. Um, and the things that we found when we were doing this work with them was there are three factors that actually help this, not surprisingly. One is redesigning your process workflow, which is you are not thinking about a traditional SDLC. You are thinking about redesigning the entire workflow end to end. You are thinking about how agents are changing that step. Second is tooling, obviously, whether you have good agents, whether you have good harnesses, whether you have good context layers, et cetera.

But the more critical thing that people have often overlooked, um, is the people part, right? So when we were doing this work, we have recognized this across many, many, many, um, pilots or scaling that we have done, is that boundaries of how you traditionally define your roles, what a software engineer was, what a product manager was, what a designer was, they have started to blur. And organizations that have recognized that those boundaries are changing and have actually remodeled their entire operating model around that are the ones that are

seeing the promised land of 10X productivity from AI and agents. Um, and as we were doing this work with Tariq, and we'll talk about it a little bit more, we saw that, you know, people who were skeptical about this, like, "Hey, agents will do the task, but we don't need to touch anything else," um, they gradually then were converts into, "Okay, this makes sense. We need to rethink and reshape how our orgs, um, are going to be organized." So with that, I will invite our panelist, Tariq, um, CEO of Sonar, and Ali, CEO of Gitar, to the stage. We'll have a Q&A. We'll

leave some time at the end for questions if we can. So you guys should also ask, but please.

Awesome. Um, I think to start with, Tariq, I'll probably ask this question to you first. You know, for companies that are going through this transition, going to Horizon two and beyond, um, what are some guidelines or learnings that you have that you can share with them on what things they should watch out for as they are transitioning to a full-scale, end-to-end orchestrated agentic pipeline?

Okay. Hi, everyone. Um, thank you, Prakhar, for having us here today as well. You can all hear me, I assume. You can hear okay? Yeah, okay. You need it higher? Okay. A little bit higher? All right, perfect. Um, so you know, we, we have, as Prakhar said, been going through this journey of how do you use AI responsibly in the organization? How do you actually, uh, transform an organization? If you don't know us, we started off as an open source project eighteen years ago. Uh, so while we're very proud of everything we're doing in the AI world, it

has been a bit of a transformation for our engineering team to get used to the new tools to really figure out how to adopt it in a native way. And, um, a part of this has really been about being super clear about what the objective is, and the objective was never for us token maxing or anything like this. We don't really like that, uh, type of idea. The idea is how do you really generate real value? Um, and so su- being super clear about what you're measuring was really important for us. I think the second piece

is, um, being very deliberate about what the process is that we're putting in place, right? And if you heard my talk I just gave a little bit ago, we've orchestrated everything around how do you actually do this guide, verify, solve model inside of our company as just like we recommend doing it inside of everyone else's company. And so this notion of how do you construct the tooling is really important. How do you construct the systems that we're talking about is really quite critical a- as well. Um,

getting the team to trust the agents, right? As working with the McKinsey team, we actually looked at what we're doing, and our team had built a bunch of agents themselves. But there are things like PRD agents and how do we really capture what's in the PRD and use that to generate the evals that we're using to generate the, the, the, um, uh, output verification. These things, um, were not natural to the team. And part of what we needed to do was set up these lighthouse

teams to really try and highlight the best practice ways of doing this. Frankly, the part that has been most surprising to me is that I thought that once you have these lighthouse teams who are operating in this way, the other teams would be able to follow super quickly-

Yeah

... almost through the role modeling. And what we found is that's not actually the case. That, that role modeling gives you part of the way there, and you can set up the tooling, and you can do all of this stuff. But the next piece of- Of actually getting them to think differently, to understand why they're using these agents, et cetera, is one of the critical missing links here.

Makes sense. Um, Ali, maybe a question for you. We have seen that smaller teams actually have a much better chance of scaling agents because there are less dependencies, less, um, challenges to overcome. What can organizations that are at much larger scale learn from those, uh, teams, right?

Yeah. So first of all, uh, it's great to be here. Uh, I'm Ali. I was previously the CEO of Gitar, and now we're actually part of Sonar. Um, and, uh, before joining Sonar I was the CEO of Gitar. And Gitar, we were a small startup.

Yeah.

Um, only about five, six people.

Yeah.

And, uh, we were building an agentic solution for validating PRs, uh, so the entire workflow for PR validation. And then before that, I was at Uber, where I was responsible for the entire developer platform organization. So it's an organization I built responsible for, you know, making sure that all developers are productive and happy, and that the company's moving at a very fast pace in terms of velocity with high quality. So I've kind of seen both-

Yeah

... um, large company and running developer organizations in a large company, and then operating in a startup world. And, uh, when we started Gitar, uh, we actually weren't AI first, and then we quickly transitioned to become AI first. And once we made that transition in terms of adopting AI internally, as well as building AI products-

Yeah

... we saw both an acceleration and demand.

Yeah.

Um, but also we were able to get a tremendous amount done with a much fewer smaller sized, uh, team. So I've kind of seen both sides of things. And I think as you think about how do you take the learnings from a small company and scale it up into a large company, actually large companies have a structural advantage. Uh, because typically as you scale your engineering org and you get to a large enough size, you have a platform team.

Yeah.

Which was exactly my responsibility when I was at, at Uber. And, you know, the roles I had in the past, like at Google and Facebook, were all about platform organizations. So there's already an organization there that's responsible for driving standards and making sure that developers are productive and, and happy. And I think there's a couple of principles that, um, are important to follow as a platform team when you're looking at how do we, uh, adopt these tools. One is, uh, to have really good metrics in

place. I think you, you mentioned it, but, uh, this is not always an obvious thing-

Yeah

... uh, for large organizations.

Yeah.

It's actually very hard to drive enough standards across the organization so that then you have, uh, telemetry that you can actually get to understand how fast am I able to ship code into production? What is the productivity of engineers in terms of throughput? Uh, what does the quality look like? How often am I erroring and going back to the beginning of my workflow? And what's the sentiment of engineers? Uh, as we are speeding things up, making things more productive, are engineers actually feeling it?

Yeah.

So I think it's very, very important to have that kind of developer productivity and metrics in place as a baseline, so that as you add, uh, AI and agents into your tool chain and into your PDLC, you can actually measure and, and iterate and quickly improve and adjust. So I think that's number one. Number two, uh, I think again you mentioned it's very important also to look at it in terms of the entire life cycle and not in terms of introducing tools into one specific, uh, part of the life cycle.

So think in terms of the entire life cycle and in terms of entire workflows. Uh, so how can you automate entire workflows, um, and take-- make developers more efficient by offloading entire workloads to agents that can autonomously act and deliver outcomes that are the outcomes of those workflows. And I think the third thing, um, I, I, you know, in my experience as part of a platform, having run platform teams in the past, is to make sure you're actually paying attention to developers' time.

Mm-hmm.

And optimizing for keeping developers in the flow-

Yeah

... um, and automating away as much grunt work as possible.

Yeah.

Um, and-

No developer likes to do grunt work.

No. But agents are perfect for that, right?

Yeah. Yeah.

Uh, so i- if you can really deploy agents to taking entire workflows that are grunt work off the plate of, of folks, that's where you get, I think, uh, big, not only productivity gains and velocity increases, but also developers feel it, and you get the high sentiment. Which is equally important, uh, if not the most important thing here, I think. So that's how I would kind of approach the problem.

Yeah. Makes sense. Um, Tariq, maybe a question for you. So as agents get more embedded into the workflows, the LLMs get better at their job, we assume that there will be a point when you have a fully automated end-to-end orchestrated SDLC or PDLC. What implications does that have on people, on org, on operating model of the future for any software organization?

Yeah. I think, um, there, there's, there's a, you know, the models are clearly getting better. Uh, Tarik from Anthropic talked about the Fable models and-

Yeah

... and I'm looking forward to getting my hands on it to start-

Everyone

... start using it, right? Um, I, I think you can, you can see a world in which this, this so- software factory idea really starts to come to life. Um, I think you used the key word in my opinion, which is orchestrated, right? Um, because I don't think that there's going to be a world in which you can be completely hands-off.

Yeah.

I think the question is how much do you leave to the machine, how much do you leave to the agent-

Yeah

... and how much do you, um, how much do you control on the outset? So what is the specification, what is important to you, what does success look like, are all really important elements. You have to, I think, continue to have people who are involved with this. So from a skill set standpoint, there's a lot of talk about how software developers need to become architects and, and orchestrators and managers of agents, things like that, and I think there's a lot of truth to this. One thing that I think is an incredibly important skill set- Is there's a lot that we

keep in our head, right? If you think about how you design software, how you build software today, there are certain things that are in your PRD, and there's a bunch of things that you just know, right? My company likes to do things-

Yeah

... this way. My company likes this sort of code. We like this dependency. We like th- whatever it is. And there's gonna be a lot more that needs to be made explicit as opposed to implicit, right? And that's actually ... there's a skill set associated with this.

Yeah.

So I think in addition to being an architect and all of that, you have to start thinking in a much more explicit way.

Yeah.

This is thing number one. I think s- thing number two, you know, my kid, my, my oldest son is about to go to college, and he's, like, trying to figure out what to study and all of this, and the piece of advice that I keep giving him is, the most important piece is to be critical, a- amazing at critical thinking, right? Because the, the point, and I read on Twitter and I'm, I'm shamelessly stealing it- ... is that AI always gives you very high conviction plausible output, right? Um, it always sounds correct. It never admits to any doubt

whatsoever. It never says, "I think the answer is this." It always just says, "The answer is this." And I think the job of the people in the system is going to be to say, "Where do I need to question it?"

Yeah.

Where do I need to, um, well, where do I need to poke? What are the things ... Kind of the same way that you do this with your team, right? If you have somebody joining your team, you have an intern joining your team, you're going to kind of know where to poke.

Ask questions. Yeah.

You're gonna know where to look, what questions to ask, what to look for. I think that skill set is also super important. Last point, to the question you and Ali were talking about, you know, um, there's a lot of resistance that I see in the organizations we work with to this idea of smaller teams.

Mm-hmm.

Right? Um, and this idea of, of working in a smaller atomic unit.

Yeah.

I think the, the, the, the pushback when you get underneath it, in my experience, is almost always, "But I'm a front end person, and I'm not a back end person," or, "I'm a this..." You know what I ... Um, and we have historically gotten to a world where we segregate roles quite a bit.

Silo in a way.

We silo people.

Yeah.

And I think this is the era of the generalists, right? And I'm not saying that somebody who knows nothing about software generalists, but, like, your software engineering generalist is, I think, going to be a much more important, uh, thing than do you know, you know, are you super deep in front end or server side or whatever it happens to be.

Yeah. Um, I think following up on that, and Ali, I know you have talked about this at other forums as well, a lot of the coding part of the SDLC is quote unquote "solved," and so focus rightfully is shifting now towards other parts. Um, the requirement generation, the validation, the testing, et cetera. Do you think we are at a point or we will get to a point in the next three months, six months, where the trust in the agents to automate all of this will go high? Because right now I think everyone is skeptical of whether you can automate your validation, your testing, your integration testing,

et cetera. How do you see that? Has it shifted already, or is it on a curve to shift?

I think, uh, trust is the, the key word here. And, uh, trust I think is gonna come with time. And so, uh, you know, what we're seeing is our users, uh, over time, as they work with the tools, uh, and they see the precision of the AI, it's that precision that gives them the trust that then, um, allows them to unlock more and more levels of automation-

Yeah

... uh, to the point where, you know, changes can completely autonomously kind of flow through into your code base and into production.

Yeah.

Um, for example, in, in, with our users in Gitar and the automation that we have in Gitar, the typical pattern we see is at first they just use reviews-

Mm-hmm

... and look at the CI failure analyses that we produce.

Yeah.

Uh, and over time when they see, okay, actually these are very precise actual bugs in the code that are being uncovered-

Yeah

... they say, "Okay. Uh, I'm actually now gonna give the agent some teeth. I'm gonna now set up some rules to block if there are any issues that are uncovered by the agent."

Yeah.

So now you've kind of moved one step in the automation to-

Onto trusting it

... not only do you have reviews that you trust, but now you actually trust them enough that they're high signal enough that you're blocking. Then typically what happens is they start playing around with automated fix, uh, features. "Okay. Let me, let me ask the agent to go ahead and fix some of these issues."

Create a PR or CI.

Yeah.

Yeah.

Fix the CI failures. Fix all the issues that you're, uh, uncovering. And then when we see the precision of that, that, okay, actually the fixes are working as well. Uh, it's producing fixes that don't introduce new problems. They pass build, um, and they can iterate to actually pass the build until it's green. Then they unlock auto fix. So now they've taken one more step-

Yeah

... towards, hey, I'm gonna ... When a PR is created, a review happens. If there are critical issues, it gets blocked. Then the system start iterating and makes everything green, fixes all the CI failures, all the, um, you know, code review issues.

Yeah.

And now you've delivered green PRs that are ready for somebody to approve. And then here's where the big shift is beginning to happen now, which is, you know, uh, there are some parts of the code base and some kind of PRs that I'm comfortable having the system approve.

Sure. Yeah.

Right? And so now we're beginning to see users put in rules to say, "You know what? If you're touching this part of the code base, it's not critical. Or if the PR is making this type of change, go ahead and automatically approve it." So now you've gone from PR creation all the way through approval, and now only thing that's left is somebody to come in and, and merge, and that's the next step as well that we're seeing happen too, which is go ahead and merge, fix any kind of merge conflicts, and now what you've got is fully autonomous end-to-end. PR creation through merge is fully

auton- uh, automated. But it doesn't come immediately.

Yeah.

Right? It comes with using the system, building trust that the agent that you're working with is actually, uh, precise. Um, there's enough context that's flowing in. You know, I've set up enough rules and guardrails that are now comfortable with the system being fully autonomous, right? So I think to answer your question, we're rapidly approaching that-

Yeah

... phase. I think within the next year, uh, we'll see more and more teams-

Yeah

... you know, along this journey. And some of these, uh, teams that I'm seeing are actually in large companies-

Yeah

... which is, uh, pretty interesting.

Nice. Um, I think that's our time. Thank you so much for being here and answering questions.

Thanks.

Thank you.

Thank you all.