Agentic SDLC at Uber - Building Blocks for Uber’s Software Factory

Talk transcript

1. 0:00

   [upbeat music] Hey, uh, let's get started. Uh, good morning, everyone.

2. 0:15

   I'm Uday. I'm here with my colleague, Adam. We'll talk about our journey towards managed software factory. And in the beginning, in the first part of the talk, I'll talk about the key building blocks that we are investing in, and later, Adam's gonna to-talk about how we take all of these blocks to build an end-to-end cohesive solution for

3. 0:32

   our engineers. To set some context, we have a few thousand engineers across twelve global tech sites. Over the last year, all of the investments we made in agentic AI have led to more than seventy percent of our PRs now either by local or cloud agents.

4. 0:49

   And all of this led to twice the number of lines of code per engineer year over year. And this extends way beyond coding, and we see it in every aspect of the engineering life cycle today.

5. 1:00

   And we are also accelerating toil at a-- toil reduction at a massive pace. Uh, we handled more than two hundred and fifty automated migrations, cumulatively nine million lines of code automatically for our engineers.

6. 1:12

   And before even the building blocks, um, you know, one-- all the investments we made over the last six years on moving to monorepos, moving to Bazel, all of that also laid a really solid foundation for us to accelerate this.

7. 1:27

   So the first, I'll cover all of these six building blocks, and Adam's gonna talk about a specific example and show how that feature can be built end-to-end, uh, with all of these.

8. 1:38

   And all of these are in various stages of maturity and rollout within Uber, but we want to give everyone a sneak peek of what we are up to.

9. 1:46

   So let's go to the building blocks, the six building blocks one by one. The first one is Model Gateway. This is one of our earlier investments. The three things that we wanted to mensure-- ensure was no PII ever leaves our perimeter to any of the vendor by default.

10. 2:02

    And any guardrail that we add here, the latency of that is strictly bounded. And every request that goes through this, whether it's, uh-- and we need to be able to attribute per user, per project, and per team.

11. 2:15

    So we have a Model Gateway, and we made sure all of our internal use cases, our coding harnesses, our external use cases, they all go through one single OpenAI Anthropic-compatible endpoint.

12. 2:25

    It goes through a series of middlewares. The first one is iden-identity and authentication using Spire. Uh, we have a data anonymizer that redacts twenty plus PII types. We have a AI guard that has five specialized models that handles various parts of safety and policy that we want to ensure, and all of that runs under a hundred milliseconds.

13. 2:46

    We also are investing in all kinds of caching and token optimization strategies at this layer. And every request that goes through this, we are able to attribute to a specific project in our catalog, and we can attribute per caller, per user, per team, both in real-time but also in our data lake.

14. 3:05

    This enables us to create all kinds of spend tiers and guardrails in a holistic way across our portfolio. We also use this layer for capturing audit log session traces, which are then plugged into our benchmarking and all kinds of sim-- self-improvement loop efforts.

15. 3:23

    And for an engineer at Uber, you take the vanilla client, you set the project ID, and we, and we take care of everything else. Today, we have eight hundred plus projects internally going through this, cumulatively handling more than a hundred million model requests per day.

16. 3:38

    This includes both the frontier models and also open source models, whether that is hosted in our infrastructure or some of our vendors.

17. 3:46

    The next is how do we provide tools to all of these models? Last year when we started on this journey, we had thousands of internal APIs, but none of them are agent accessible out of the box.

18. 3:57

    And we had so many other SaaS tools, and each one of them have different way to authenticate, different way to set up, which is a lot of hassle for, uh, for everyone.

19. 4:06

    And once you end up with enough MCPs, they'll all add up to and c-- they-- and have a massive token tax.

20. 4:14

    Similar to Model Gateway, we have an MCP Gateway that handles whole bunch of middlewares for, for engineers, and we have an automated crawler that looks at our internal APIs and projects all of these into MCPs with one single config change.

21. 4:29

    And we do the same thing even for our SaaS MCPs. Whether it's Google, Slack, Jira, all of this, they go to the MCP gateway. We ha-- we host them, we do the token exchange.

22. 4:39

    So for all the engineers, they go through one single entry point, one common way to install any MCPs. This simplified a lot for all of our engineers and employees.

23. 4:49

    And then the whole bunch of token optimization strategies. We initially had direct MCP pattern. Earlier this year, we created OmniMCP, which is one single MCP that you install, which can discover and invoke any MCPs within the gateway.

24. 5:05

    And couple of months ago, we projected all of these MCPs into CLI pattern so that even the response doesn't eat up in your context. And of late, we also have a, a codemod skill which is auto-installed, which on the fly creates Python scripts to hyper-optimize some of the top MCP token consumer-- consuming use cases.

25. 5:27

    And all of this led to like now we have thousand plus MCP tools and, uh, just with these optimization efforts, we've saved more than forty percent fleetwide savings.

26. 5:41

    So once we have the models and the tools, we need a place to run all of this. For many years we had DevPod, which is our cloud, uh, remote environments.

27. 5:51

    We, we, we had this because we had like large monorepos with millions of lines of code, and this is how engineers work at Uber. And now we took what we had with DevPods and we a-agentified that.

28. 6:02

    Now we need An environment for agents to run for a longer period of time. They need to be quick, they need to be isolated. We can install any number of them and they need to be globally available across all of our sites.

29. 6:15

    So we have a pre-provisioned Kubernetes balloon ports. When an agent requires a new environment to run, it can take one of that, which is already pre-provisioned. It has all of the repositories already snapshotted.

30. 6:26

    The search index is already built, so the agents can start working within a matter of seconds.

31. 6:32

    The next thing we noticed is the, the, the roles of engineers are getting blurred. We used to offer a DevPod per language flavor for Go, Java, Android and so on.

32. 6:44

    Now, we need agents to work across repositories and engineers also to work across repositories. So we have a mega DevPod that has all of the repositories in one, one common place, and this is what we use for our autonomous coding agents now.

33. 6:57

    And even for our non-engineer employees, we are providing a simple way for them to get started with any of the agent harnesses in matter of seconds.

34. 7:09

    Then now we get to knowledge part of it, and we jumped on this bandwagon earlier this year. Uh, we started noticing engineers building tons of skills across many repositories.

35. 7:19

    Um, and three problems we noticed was there's a lot of duplication, same skill being built by different engineers in different repos, and discovery and configuration was a huge hassle, and a lot of skills were of subpar quality.

36. 7:33

    So what-- We built entire lifecycle around skills. So we have core se- core skills and domain-specific skills. All of that go into a managed skills marketplace. We have two thousand five hundred skills there right now.

37. 7:45

    Um, and it goes through a whole bunch of lint checks, automated reviews, which ensures a baseline skill quality for any skills that we have.

38. 7:54

    And we also simplified the installation and discovery. So there is one single command to discover and install any plugin in our ecosystem. And based on the engineer personas, we even auto-install some of the default skills so the agents automatically can pick up the right skill.

39. 8:09

    You don't even have to even install them.

40. 8:13

    And of late, we started working on collecting traces and comments and capturing continuous evals so that we can go give feedback back to the skill authors for skill improvements.

41. 8:24

    And this is an area of big investment for us right now. And we have two thousand five hundred skills and cumulatively more than twenty thousand skill executions per day a-across our fleet.

42. 8:37

    The next piece of knowledge is context graphs. Uh, we st- we started noticing in our execution traces, agents spending a lot of time even trying to find basic context, especially in our large monorepos.

43. 8:48

    You need to identify where the service is located, what are the dependencies, um, who owns it, what kind of patterns I need to follow. And all of this context is gathered across scattered sy-systems across Uber.

44. 9:01

    There's twenty to thirty different systems. Each needs its own skill-skills, its own MCPs to in- to gather the context. And this burns tokens, this adds a lot of latency, and it creates more unpredictable outcomes.

45. 9:15

    So we have one context graph. We took all of the information of how Uber runs into one context graph. This has one hundred and fifty unique node and edge types.

46. 9:24

    We have forty million entries there right now. It captures all the way from how our mobile apps are built to our back-end, to our data lake, all the design docs, Jira, incident bugs, everything is connected.

47. 9:35

    And this enables agents to quickly find the right context within our ecosystem. We are now plugging all of our skills and use cases into the graph, whether it's our on-call RCAs, whether it's our planning or data analysis or security scans, and we see across all of this that we are improving, uh, the skills by a lot.

48. 9:55

    And I'm just showing a very simple example of asking a simple question of how many mobility trips in India are, are cash. This needs to understand the concepts of each of these, which tables, what kind of, uh, CTAs you need to create for the SQL.

49. 10:08

    With and without graph, we see massive improvement in tokens, turns, and latency. And we see that across any early eval that we did, uh, within our infrastructure.

50. 10:19

    And the last thing is, how do we package all of this for everyone in the company to use? So we have, uh, our AI assistant called Cortana. All of the things that I mentioned so far, whether it's skills, MCPs and context graph, they're all plugged into that in every surface possible, whether it's on Slack, CLI, web.

51. 10:37

    So anyone in the company, they can ask a simple question. It can look up the context graph, invoke any skill, check any code, check any code in any code base, and give an answer across any of these surfaces.

52. 10:49

    And now we started allowing employees to even personalize that. You can hook up your custom skills, custom prompt, and hook it up into your team Slack channel so that it, it knows all of the things about that team and works like a-- the, the, that teammate.

53. 11:07

    And this, this is a simple example of how you can invoke the same question before in Slack, um, and ma-- all of the employ-- m, m, more, like one or more people can even collaborate on the same Slack channel.

54. 11:19

    And we have just in the last one month, three hundred unique personas created and more than twenty thousand sessions per day. I'll now pass on to Adam, who will talk about how we take all of this and build, uh, take, uh, and ship a feature end-to-end.

55. 11:33

    All right. Thank you, Uday. All right, as Uday said, we've got those building blocks. We're gonna use those to power our software factory. So we're gonna take a feature here and show it going end-to-end through this.

56. 11:46

    All right, first up, right, we need to have an idea, right? A good idea probably for this moment would be something around the World Cup, right? Would it be awesome if you were a rider and you were leaving a busy stadium if there was a better pickup location to get you away from the crowd?

57. 11:59

    So that's the idea, right? We can have our idea. We're jamming on it in Slack here. Let's tag in Cortana, right? That's our AI assistant to help us with that idea.

58. 12:08

    Cortana with that context graph can help us determine whether this is a good business opportunity to go after. So we can go here from Slack and now open Cortana into a web interface,

59. 12:20

    and you'll see an example here of what that business research could look like, right? What other large-scale venue events have happened before? What are some stadiums that would make sense here?

60. 12:31

    From there, we start to think about the product requirements, right? This should be probably just a North America rollout since that's where the stadiums are. We can even then bring in Cortana to help us think about, uh, the Figma designs.

61. 12:42

    We can create some initial mock-ups, right? Do two variants here. We wanna run an experiment, A and B. So the, the button strings here are different between the two.

62. 12:50

    So let's test those two variants and see which one performs better.

63. 12:54

    Now we'll start to think a little bit about the design, and Cortana too can help us think about what code changes we need to happen, right? What can we leverage that's in the app already, what screens, and what can we leverage on the back end?

64. 13:05

    Right. So this process before could take a long time. It could take weeks to get everyone aligned. Now we can compress this into a very short amount of time now, right?

65. 13:14

    And get to a prototype here very quickly.

66. 13:17

    All right, so now we gotta go build this. So we, we hand off from that Cortana agent to what we have at Uber. We have a Minion agent. It's, uh, Uber's cloud coding agent solution.

67. 13:28

    All right, so you can use Minion in an interactive mode, or you can run it in an autonomous mode as well. So I'm gonna show you what this looks like.

68. 13:35

    Um, Uday mentioned the DevPod building block, so this is powered by that DevPod, so it's got a full build environment and it can work across repos. So we're doing back-end changes and the front-end changes here too as well.

69. 13:46

    We're gonna see Minion kind of progress here, and it's gonna stop at just creating a draft PR, and it's not gonna push it to CI yet. The reason being is that we were seeing, um, uh, uh, that this is great for doing like toil sort of workloads, but to build more advanced like end-to-end features, we really need

70. 14:02

    to be able to validate, uh, the feature first, and we want to prevent, uh, a lot of extra load coming onto CI. So if we can validate, um, sooner before we push to CI, um, that would be a big benefit.

71. 14:14

    So that's what we're gonna see here next on validation. Right, in the SDLC, we have an inner loop. We have the outer loop, of course. We can have these be agentified, where we're shifting more checks now to happen in this inner loop.

72. 14:27

    All right, so some of the checks that initially happened that we've had there pre-previously is like the, the static analysis sort of checks. When those are detected, now we fix those.

73. 14:35

    But we can shift things to happen in inner loop, things like visual validation. So we can launch, uh, in a simulator with a skill, grab a screenshot from the simulator, compare it to the Figma specs.

74. 14:45

    We can also bring up the service in our back-end staging environment and compare the front end and the back-end integration together.

75. 14:53

    So now that we've moved a, um... Now that we've done, uh, that part, uh, we move to the outer loop where CI, uh, typically happens. Errors can still happen on CI, right?

76. 15:05

    Uh, so self-healing CI is something that we've implemented here where we can fix a lot of the issues that you hit on CI. Um, code review is another thing that happens in the outer loop, but this is another thing that we've shifted.

77. 15:16

    We've moved parts of code review to happen in the inner loop, right? The outer loop code review can have a powerful model, use reasoning, a skill to do a deeper review, and in the inner loop, we can have a smaller model that runs, uh, faster with the, with the, with the medium model.

78. 15:30

    Now, another key thing here, right, is this, if this is an autonomous diff coming from Minion, we want to give a human reviewer some confidence that this diff has gone through a lot of self-improvement already, right?

79. 15:40

    That does-- not just touching that initial generation that happened, but all these other steps have happened. And so on the PR, you will have a table attached that says all these different checks that it went through, including the screenshots.

80. 15:53

    All right, so we've got a lot more code coming through the software factory now, right? Let's talk about maintenance, right? Maintenance is even more important. Um, what we have set up now is we can actually enroll our feature or skill into, um, our feature or serve, uh, service into maintenance, uh, skills.

81. 16:10

    So these are, uh, some examples of those skills that we have. Uh, feature flag cleanup, right? We had two variants of that World Cup, uh, modal. Now that the, the B, the variant is no longer needed, we can have that scheduled on a loop.

82. 16:24

    So the key thing here is that this is actually a managed loop that you go to, right? We don't want thousands of loops being set, uh, set up across the company without any bounds.

83. 16:33

    You have a managed surface that you go to to set up the loop, so it runs on Sunday when we know we have better, uh, CI capacity available. Um, we also don't want to overwhelm engineers that Monday morning with a bunch of extra diffs.

84. 16:45

    We want to control how many diffs they're seeing on Monday as well.

85. 16:49

    Another cool key thing here is that when that ski, uh, skill runs and makes those diffs, those diffs will get comments and either get landed or not landed. That's all good labeled data that we can use to improve the skill itself.

86. 17:00

    And then at a kind of monthly cadence, we're looking to see what skills can we now learn, um, from in our incident reviews and turn those into new maintenance skills that we can apply to all of our services.

87. 17:14

    All right, so you've seen, uh, these parts of the SDLC that we have agentified. There's other parts too, like monitoring. You've seen the, uh, building blocks that you can use to power those and the architecture underneath them.

88. 17:27

    One of the other things that we're really thinking about now is bottlenecks, right? Where now we're putting more strain on our infrastructure, so we're trying to anticipate where our CI capacity needs to be and make the right foundational investments there.

89. 17:39

    There's only so many experiments that we can feasibly run as well, so that's another bottleneck. And then lastly too, right, decision-making, right? It's not about, you know, can we build?

90. 17:47

    We know we can probably build it now. It's more of a question of should we build it?

91. 17:52

    All right, so that's what we have for you today. And the next talk is gonna be actually from Uber as well. And if you want to learn more about our agentic code review, uh, Amaya and Will will be presenting that next, uh, in this room.

92. 18:04

    Thank you. [audience applauding] [upbeat music]