Building AI Agents with Real ROI in the Enterprise SDLC

[on hold music] Uh, so my name is Beyang.

Uh, I'm the CTO and co-founder of a company called Sourcegraph. We build dev tools for, uh, big, messy code bases.

Yeah. And I'm Bruno, Bruno Passos, and I lead, uh, the product side of developer experience at Booking.com. And, um, yeah, over the past year, I've been overseeing the, um, the GenAI innovation side of Booking as well.

Cool. And today we're here to talk about, uh, how we're partnering to build software development agents, uh, that automate a bunch of toil inside Booking that are actually having real ROI and impact.

So how many people have heard this before? You know, you're, you're the... You, you're working inside a large company, the CEO comes in and says like, "Hey, we need to adopt AI."

Uh, and then folks are like, "Okay. Uh, what does that mean?" You know, how do we measure it? You know, maybe you like FOMO purchase Copilot or something like that.

Uh, and then six months later, uh, someone else, maybe the CFO, is asking you, "Hey, so what's the ROI of, of that AI tool we just adopted?" Or, you know, "What's the measurable impact of the agents that, that we're building?"

Um, this is a question that I think a lot of people aren't quite sure how to answer right now, but Bruno and Booking have been sort of on the leading edge of answering this question, uh, and very proactive at, uh, acquiring and building the best tools, and also following through to demonstrate, uh, how they're actually impacting their

org.

It's very kind of you to say we are, we are leading this. I think we are, we are right at the beginning, and, uh, I feel couldn't be f- couldn't feel farther from, from actually the forefront of it.

Uh, but let me, let me start by talking a little bit about Booking. Um, I am sure, uh, that most of you would have heard about this company. Uh, our goal is to make easier for everyone to experience the world, and my team's goal is to make sure that our developers have their path cleared so that they

can do their best work. Now, are we close to that? In some parts of the company, yes. Other parts, we couldn't be, uh, farther away from it.

To get-- to set a little bit of context, uh, we are one of the largest, uh, online travel agencies in the planet. Um, and we serve about 1.5 million room, uh, nights, uh, um, with more than 3,000 developers.

Uh, can you raise your hands, uh, who work in a company that has more than 1,000 developers? Quick show of hands.

Good number of people.

Okay.

Wow.

Uh, on the-- more on the, on the dev side or on the technical side, um, we serve over 250 merge requests, uh, at a given year, uh, with 2.5 million CI jobs running at a given, uh, year as well.

And we are extremely data-driven. Um, our company has gotten to where it got to over experimentation and being obsessed about data. And the reason I'm going into this is because as we experiment and in the form of primarily AB tests, we start adding those experiments and ex- and feature flags into the code base.

And as we push forward to bring new features to our users, uh, most likely those experiment flags or dead code will stay in the code base. And now fast-forward decades, our code base became extremely bloated.

Uh, fun fact, I was-- uh, my kids were looking at me, uh, editing this slide, and they said, "What are feature flags?" And I said, "Well, um, you know, they stay in the code base, and they start po- po- polluting the code base."

And they were like, "Like code farts?" And I'm like- [laughs]

... now you're going into code smells. It's a different topic. [laughs] But let's, uh, let, let's move forward. Um, and so as the, the code base starts to bloat up and become bigger and bigger, cycle times also become, uh, larger and longer, and they s- the time that developers spend to debug and to work on that code base

just becomes over 90% toil, right? Who here is familiar with this?

That's even more hands than a, than, than-

Yeah

... than 1,000 developers. And so we survey our developers at least a quarter, uh, on how they're feeling, how they're, how they're, they're feeling about working on that particular code base, and it's, it just becomes harder and harder for them to do anything.

And so we had to do something about it.

So I've seen the best developer minds of my generation destroyed by decade-long dead feature flag migrations. [laughs] It's crazy.

Claude Shannon actually say that or? [laughs] But I mean, if-- seriously, though, like, there are probably, like, geniuses out there. Like, I was talking to someone from PWC, uh, the other night and described the system that they're building to, like, update, uh, all, all the kinda, like, legacy code in their system, and it was amazing.

Like, the guy was really smart, really brilliant, uh, really, like, interesting tech. But wouldn't it be great if, you know, those sorts of minds were unlocked to actually work on, like, you know, new features and thinking about, like, user problems rather than all this kinda, like, legacy cruft?

And so in a nutshell, that's why Sourcegraph exists, uh, as a company. So our mission is to make s- building software at scale tractable. Uh, and so you might be familiar with a couple of the, the products and tools we built, uh, along the years.

Uh, Code Search, it's kinda like a Google for your code. Allows any human developer to find in and, and build a working understanding of what's going on. We have a tool for large scale refactoring and code migrations.

Uh, you might have heard of our AI coding assistant, Cody. It's a context-aware, uh, code generator that's tuned to work well in large, messy code bases. Uh, and the topic of this talk is really about the agents that we're building to automate toil out of the software development life cycle.

So a bunch of different products that we've built over the years. The unifying theme really is to, uh, accelerate things in the developer inner loop, augment hu- human creativity there, and then to automate as much of the BS, uh, out of the outer loop as possible.

All right. So, um, as Beyang talked about, uh, Sourcegraph Search, just over two years ago, we started using their product, and it was a big success within our, uh, our community because they were able to search that bloated code base much, much easier and find small pieces of context lying here and there.

I, I totally encourage you to have a look at this, uh, particular product. It's awesome. Um, and so about a year ago, uh, January last year, we started experimenting with Cody.

Why? Because Cody also has Sou- uh, uh, has Sourcegraph Search as context, and so it became extremely useful for us to use a, uh, tool that had that context to be able to experiment with the, the GenAI topic.

And now we are hoping to reach the path of, uh, uh, building agents with Cody and, and Sourcegraph Search, uh, uh, built in.

All right. So, um, if I summarize very quickly, and hopefully this illustrates how fast things are moving, uh, uh, forward. In January, we started, um, with Cody. We gave everyone the ability to start using the tool in the company.

So all our three thousand developers, uh, um, had the, the opportunity to use it. Some started using it, some, uh, uh, uh, used it, didn't see any value with it and then stopped using it.

And that started intriguing us. And so back then, right in the beginning of the year, we had the choice of one LLM to use across the entire company and some token limits, uh, uh, um, uh, limiting what we could do with it.

And so the first thing that, that we, uh, started pairing with Sourcegraph, and we appreciate the partnership on that, was to remove all the, the, the guardrails that we had in order to be able to really, uh, give it a go.

And so Sourcegraph was very quickly to be able to give us multiple LLMs, pair developers. We could choose that. And why that was important is because we found that LLMs had expertise, right?

And so if we were going to excavate our code base, our bloated code base, a particular LLM would do better than someone that was working on a completely new, uh, uh, piece of service and, and developing features there.

And so fast-forward to July, um, we started training developers, and that became incredibly important because the people that started using and didn't see the value, when they started getting trained, they started using it and falling in love and becoming what we call that now daily users.

And I'll explain how-- why that's important. Um, and then we started g- looking into more metrics. Back in January, the main metric was hours saved. And, um, I mentioned that we are a data-driven company, and hours saved wasn't the most statistically relevant metric that we could use.

It was based on research only over a couple of developers, a few developers, and, uh, that wasn't cut. Um, raise your hand here if you, uh, heard folks out there in the beginning of the hype talking about thousands or, or eighty, a hundred thousand hours they saved with GenAI.

Has anybody ever heard that? And then you go back to your company and say, "Why are we not doing this?" I call that semi-BS. Uh, uh, uh, and so we had to start going into other metrics, something that were more statistically relevant.

And so we started brainstorming with that. Come October, uh, October, we defined new KPIs, which I'll go deeper into it, and metrics to measure, uh, to measure GenAI. And fast-forward to November, end of last year, we then started finding traces that developers were thirty percent plus faster if they were using Cody on a daily basis, and that's

twelve plus day in a month to take a- away weekends and the times that they're, they were not coding. And most importantly, we were able to partner with Sourcegraph to be able to create an API layer in front of Cody, so we could be creative in using with some of the tooling that we use, like Slack, Jira,

and, and being able to extract some of that away from the IDE.

All right. So as we, as we, we finish around October, we started looking into so those KPIs. And what was important for me is that we defined something that we could measure within a year.

Why? Because things are moving so fast, and if we-- Uh, uh, it was really helpful to ground us to what can we measure within the next year. And so we defined four KPIs.

The lead time for change, quality, code base insights that would then go into how we could modernize some of our, uh, bloated code base. And so some of the metrics, uh, uh, when I say short, mid, and long term, these were metrics that we could see results in the short term, in the mid term, and in the

long term, and that long term is precisely a year. And so we started seeing results with time to review MRs. Developers that were using Cody on a daily basis would ship thirty percent more MRs than the ones that, that didn't.

And one very interesting piece is that their MRs, their MRs were lighter. They had less code in it, which I still don't know what to make out of it, but we are, we are working on it.

And then on the quality side of things, we are hoping to go into the vulnerability. Can we show some of the vulnerabilities we've had in the past? Give the context, the code base's context, and try to see where we can predict where the new vulnerabilities will, uh, will appear, or if they're still lingering in our code base.

And then we started, uh, the obvious one is test coverage. Can we increase test coverage? Can we create test coverage on the legacy so the new stuff, when we re-platform, passes that, that particular set of tests?

And then we went into coding sites, which is more related to, like, can we track what parts of our code base are not being used, some feature flags that are still lingering but shouldn't be there, and the code that is not performant enough.

And all of this is to feed into our ultimate goal, which is can we bring the time to re-platform our code base from, from, yeah, years to months, right?

Okay. So while all this is going on, one of the things we noticed is that the same engineers that were using the, the, like, coding assistant to generate code were also playing around with the underlying APIs.

And so what we realized is that, like, asking people to customize prompts leads to them wanting to build and compose those calls into longer chain automations that we now call agents.

Um, there are a lot of pitfalls that we encountered, uh, uh, you know, in, in the early stages of this, like helping u- people understand what the expectations were with respect to what the, the LLM can do and what it can't do.

But the long story short is, at some point, we basically said, "All

F this, it's not really working. Let's just, like, put our brains together, you know, fly out to Amsterdam. We'll do, like, a week-long joint hackathon and build some agents together.

And so the first thing to come out of that hackathon was this thing that, uh, generates GraphQL. So Booking has a, a huge GraphQL API. We'll play the video.

It, uh, it's seriously, like, more than a million tokens long. [smacks lips]

Uh, so it does not fit into the context window of any of the existing, uh, LLMs. Even if you could shove it inside context, it's not gonna do a good job of, of integrating that context into something that's coherent.

There are a ton of hallucinations. And so what we did is we built this system that basically searches this very, very long GraphQL schema, finds the relevant, like, nodes wherever they are in this, like, schema tree, uh, and then, uh, agentically figures out which ones are relevant, and then walks up that tree to pull in the relevant

parent nodes. And so on the right-hand side, you can kinda see its, like, inner dialogue. This is, like, its thought process for, uh, reasoning about which nodes of the schema to pull in.

And then, uh, after it's done that reasoning, it generates a response. And so if you do this naively, you know, the UI looks very similar, but you just end up getting garbage, which is what we were seeing, you know, before we ran this hackathon.

After we sat down and, and, and actually worked through, like, the specific prompts and stuff, uh, to make this work well, we saw far better results.

All right. So, um, a pretty interesting one that, uh, uh, that we started, uh, uh, working through in terms of agents were the automated code migration. Could we go into that legacy piece,

functions with over 10,000 lines to give you context, and, uh, uh, and speed up that replatforming effort? And so, uh, code search, structure, structured meta- meta prompts, uh, and then the, the concept of dividing that particular code base to conquer the small bits were, were really, uh, really interesting. [smacks lips]

Um, one of the things that I totally recommend if you started to embark on, on, on a journey like this is pairing with some experts to bring that expertise into, into your offices was incredibly valuable to us.

And we started seeing, uh, back to when I mentioned that the developers were using coding and stopping and feeding back, doesn't, doesn't add any value, was pure lack of knowledge.

Uh, folks didn't know how to work LLMs out. They didn't know how to pass the right prompt in the right context, and this was, uh, a, a pretty important piece for us to be able to, uh, to work on this particular, uh, agent.

And so when we go into this, we had developers working for months at this point to try to figure out the size of the problem that we had to then be able to divide and conquer.

And then we came within two days within a hackathon, we were able to really define, uh, and understand where the, the call sites were coming from. And then being able to define how big the problem is was important for us to be able to have a start point and then collect the low-hung-- low-hanging fruits that were a-available

for us. So, um, all of this is still in experimentation, uh, mode, uh, but we've seen a lot of values and a lot of, uh, uh, sort of like firing that smoke in t- going from months, uh, uh, of, of understanding the code base to days.

Cool. And so the, the last agent that really came out of this joint effort, uh, was targeted at code review. So this is something that we found is, is pretty universal across many different enterprises.

Like, everyone, who does not do code review here? [chuckles] One hand, okay. Uh, I'll talk to you later, sir. [laughs]

Um, so, like, everyone does code review. And what we found-- Like, originally, we didn't think this was a very interesting space 'cause there's, like, two dozen startups now that are popping up that do AI code review.

But when we talked to Booking, we talked to other enterprises, what we found is that, like, code review is kinda, like, very specific to your organization. There's a long tail of, like, rules and guidelines and other things that, uh, you wanna bake into your review process.

And a lot of the tools that are off the shelf there aren't super customizable. And so what we built is this interface, uh, where we're going through and productizing the, the process of building a review agent that's tailored to your team and your organization.

So the, the basic idea is that you define a set of rules that you wanna hold in the code, and then those are defined in kind of like a simple flat file format.

And then the agent will go and consume those rules, apply the relevant ones to the specific files that are modified in any given, uh, PR, and then, uh, very selectively post up comments, uh, that are tuned to those rules.

So it's very not noisy. Uh, we're trying to optimize for, you know, precision, uh, over recall here in, in the feedback that we're, we're giving, uh, the, the de- the developer. [smacks lips]

All right. So knowing what we know, in the beginning of this year, we've been working on this for a year together, uh, uh, with Sourcegraph. Then a few ideas started p- uh, popping into our minds of how we could go forward here.

And, uh, one of the things that I'd love to leave you with is the concept of declaring what are the rules of your service, right? So think, think of your CI pipelines today.

Uh, when they give you errors, could we anticipate and shift this left to the IDE so those errors appear there, and they appear in the form of, "Here is an error, and here is a fix," so hopefully the service gets to a point where it's self-healing, right?

And we started seeing that we could do that, that there are, there are, there are areas that we can start using, giving the-- all the context, all the prompts that the developers started creating via the prompt, uh, library that we created, and, uh, asking those questions, autom- uh, automate those questions to the server to see what comes

out in terms of knowledge, uh, um, uh, out of that code base. And so we think this is ultimately what we, we are trying to achieve within, I would say, a short, as short as the end of this year in terms of, uh, agents.

Um, but lots, um- Lots here to, to go

Can I... Sorry, can I just say-

Yeah, of course

... one more thing about that last, uh, slide? Um, I think that we have the potential here to solve one of the problems that has plagued software development since its inception.

So, you know, i- who here has read The Mythical Man-Month before? So, yeah, oh, basically everyone. So, like, it's this problem of, like, any software that becomes successful eventually becomes a victim of its own success.

Because if you have revenue, if you have users, that's gonna generate feature requests, bug reports. Any business that's prioritizing that is gonna take on tech debt to, in order to compete, quite frankly.

And over time, as you add contributors to the code base, you lose this cohesion of vision. You lose the set of standards that you wanna maintain and hold. Uh, with declarative coding, now you can have, like, the senior engineers, the architects, the, the, the people in charge of the organization define constraints and rules that must hold through

the code base, and enforce those rules both at review time, as well as inside the editor for, you know, the code that's written by human or AI.

Yeah, for bigger organization, all your compliance rules, all the things that, that the developers need to work on, but it's not necessarily feeding new features to your, to your end users, I think those could...

Are, are perfect examples of, um, um, yeah, being declared into your service. But anyway, the main important thing so far in this past year that we've been, uh, um, you know, pairing to be able to figure this out has been education.

The more we educated the developer and handholding entire business units to be able to show them the value, but then have them experiment within two days of, like, workshops and hackathons, have them experiment with the tool, they were coming out the other side incredibly passionate about what it can do, but also becoming that daily users that we

are trying to transform them, so hopefully to defend that 30%-plus increase on speed. So educate your folks. If you take one thing from this, is education.

And if you wanna dive deeply into any of this, we got a booth downstairs. Feel free to stop by. We'll talk shop. Or, uh, also tomorrow, I'm giving an expo talk that covers some of the more nitty-gritty details of how some of those agents were implemented, so.

Thank you.

Thank you all. [audience applauds] [upbeat music]