What We Learned Deploying AI within Bloomberg’s Engineering Organization

Talk transcript

1. 0:00

   [upbeat electronic music] I don't have a joke about the dog.

2. 0:22

   I only have a joke about the, uh, hot dog data, so I would just jump to the topic right away. Um, so my name is Lei. Um, I lead the, uh, Department of Technology Infrastructure in Bloomberg.

3. 0:33

   So we're basically a group of technologists focused on global infrastructure. Think data centers, connectivities, um, developer productivities, uh, think SRE tooling, and also our reliability solutions.

4. 0:48

   Think telemetry and incident responses, right? So, um, depends on the audience. Sometimes, uh, you know, you're familiar with what Bloomberg is, sometimes you don't. So I thought it might be a good idea to talk a little bit about, about our company.

5. 1:03

   Um, so there's no better way to talk about our company by sharing some numbers. I wanna highlight a few numbers. We have more than nine thousand engineers, and most of them are software engineers.

6. 1:14

   Uh, we handle a lot of market ticks, uh, which in the billions and six hundred billions, I believe. And, um, we also have tons of fo-folks, uh, focused on AI research and engineering.

7. 1:27

   So we have, uh, more than, uh, really today's five hundred plus em-employees focused on AI products, uh, for, um, sort of our customers. So takeaway here is we are, I guess, you know, building a lot of software and use a lot of data to empower our flagship product, which is called the Bloomberg Terminal, and to really support

8. 1:50

   our users to make the most important fi- decisions for them to do their job, uh, the best.

9. 1:59

   Um, in the technical lens, um, a lot of time kind of need to explain that we actually have one of the largest private network, uh, in the whole world.

10. 2:09

    We also have one of the largest JavaScript code base, um, in the world. Um, we-- Because the domain we're in, uh, so the Bloomberg Terminal is really, you can think of a, um, software that supports thousands of different applications.

11. 2:26

    Uh, we call them functions, right? Um, email is a function. Uh, news is a group of functions. Um, let's say fixed income, price to yield calculation to spread calculation is another function.

12. 2:41

    Um, trading workflows is another group of functions. So there's many, many, many different type of functions. As you can imagine, we kind of have to utilize different technologies to really support those, uh, functionalities.

13. 2:54

    Uh, we also been s- increasingly more than use, but also contribute to open source communities. Um, for this audience, I guess I wanna call out, you know, we've kind of helped creation of the KServe, En-Envoy AI Gateways, and among many, many other things that, that we deploy in-house and support the communities.

14. 3:14

    Again, in summary, there's a lot of software, there's a lot of data. Uh, we kind of have to, um, figure out how to make the best of AI tooling to support us to do our engineering work.

15. 3:26

    All right, so get to what is AI for coding. Um, we start about two years ago, maybe a little bit more than that, um, and as I guess the rest of the world, we look at the toolings provided and, you know, I apologize if your, if your logos are not here.

16. 3:46

    Um, but I-- as you can imagine, it's kind of, like, overwhelming, right? There's so many things, and every day there's news about this is great, this is great. Um, so at the time, we actually didn't know what

17. 3:58

    all the AI solutions can help us to, uh, boost our productivities as well as stability. But one thing we knew at the time is, um, unless we deploy and try, we wouldn't know what's the best way to benefit from all the awesome work and, and, you know, a lot of folks are contributing to.

18. 4:21

    So at the time, uh, we quickly formed a team, people start

19. 4:26

    kind of like release, um, kind... a, a set of capabilities so that people start iterating on, um, utilizing the toolings. And then, of course, you know, we are a data company, so kind of wanna get a sense of how we measure the impact and, um, what we can do from the capability we provide, right?

20. 4:46

    So we look at the typical developer productivity measurements. We ran a few survey. Uh, it was very obvious that people felt like there's much quicker, uh, proof of concept.

21. 4:59

    People roll out tests. Um, there's a lot of one-time use scripts being generated. And then the measurements dropped actually pretty quickly when you

22. 5:09

    go beyond all the greenfield type of thing, right? And then, then we start thinking like, okay, so what are the things that we should really be doing using all these wonderful things so that we can really make a dent, um, in the, in, in the space?

23. 5:27

    And then at the same time, we also kind of like also be thoughtful of, um, unleash a very powerful tooling, right? Uh, the, the benefits is, it's very fast.

24. 5:40

    The challenge is also it's very fast, right? Um, for any of you who actually dealt with hundreds of millions of lines code, you probably understand the system complexity is a at least, um, exponential or at least polynomial, I guess, function of your line of code or software assets, right?

25. 6:04

    So at some point, you kind of want to be very careful, uh, what you do with your software assets.

26. 6:10

    And what we thought, so maybe we should look at some of the basics. One idea we had is, um, all right, so AI for coding, there's a narrow definition of what coding is, but there's also a broader definition of what software engineering, right?

27. 6:24

    And then maybe we can also look into some of the work our developers don't really prefer to do. For instance, um, some maintenance work, some of the migration work, some of the, I don't know, maintenance work and stuff like that.

28. 6:41

    So I want to give some examples of the things that we've been trying and we think there's pretty good return on investment.

29. 6:48

    So the question we ask ourselves is how do we evolve our code base, right? Uh, the first one is, all right, wouldn't it be cool, uh, the day you get a ticket saying, "Hey, you know what?

30. 6:58

    This piece of software needs to be patched," and at the same time you have a pull request with the fix, with a patch, and also with the thinking of why the patch happened that way, right?

31. 7:09

    So it's kind of like we're trying to, uh, broadly deploy something called uplift agents. Um, broadly scan through our code base and figure out what patch would be applicable and be able to apply this patch.

32. 7:23

    Step back a little bit, we did have a regex-based refactoring tool. Um, it works to some extent, but it's limited. Right now with, um, our LLMs and the other tooling, so we are able to, uh, see very much better results from the, um, uplift agents.

33. 7:39

    So there are a few challenges in case you also plan to deploy such capabilities. The first one is,

34. 7:46

    I guess, any AI or ML, it would be really nice if there's some deterministic verification capability. Uh, oftentimes it's not so easy, especially if you don't have test cases, you don't have good linter.

35. 7:56

    If you don't have good verification, the, the, the patch can sometimes be, uh, uh, difficult to, to, to be applied.

36. 8:05

    And, uh, one thing we also realized when we deploy AI tooling is the average open pull requests increased and time to merge also increased, uh, because you're spinning a lot of new code and then still we have to review the code and merge the code, right?

37. 8:21

    So time to merge become a challenge sometimes. And the last one is, um, I think it applies to any gen AI thing is the shift becomes what do we want to achieve rather than how we want to achieve, right?

38. 8:33

    So, the second example that I, I wanna share is, uh, the other area that people kind of like sometimes im- really impact our productivity in a negative way or impact our stability in a negative way is how we handle instance.

39. 8:49

    So we're trying to develop and then deploy, um, instance response agents. Um, now

40. 8:58

    the importance of this is if you really think about gen AI tools, it's really, really fast and it's also unbiased, right? In my instance, it can go through your code base really quickly.

41. 9:11

    It can go through your telemetry system very quickly. It can go through your feature flags very quickly. It can go through your, um, I don't know, call traces very quickly and in an unbiased lens.

42. 9:22

    When we do troubleshooting, sometimes we have this biased view. It's like, "Okay, it must be this." It turns out to be not the case. So there's many, many interesting benefits, um, by, uh, deploying agents from this perspective.

43. 9:37

    And then the second question is, become interesting is imagine you have organization of ten thousand pe- um, let's say nine thousand people as I described. A lot of people are trying to fix those problems, right?

44. 9:49

    And you have ten teams who wants to build a pull request review bots. You have twenty teams who wants to build a incident response agents, right? They become very quickly chaotic and sometimes can have duplications.

45. 10:05

    So before I talk about the paved paths, I'm gonna give the example of the, uh, incident response agent. So basically this is what, you know, a incident response agent will look like.

46. 10:17

    Um, the key part is we're gonna need to build a lot of MCP servers to connect to the, um, the metrics and logs dashboards you have, connect to the topology you have, whether it's network topology or it's the, um, your service dependency topology, uh, your alarms, your triggers, right, your SLOs.

47. 10:37

    And then we kind of don't want people just start building MCP servers, uh, without a paved path. So we created a paved path in partnership with our AI organization, and I will talk a little bit what that means.

48. 10:52

    Before that, um, I do want to explain a little bit some of the platform principles.

49. 10:59

    Some company allow teams to be, have a lot of freedom as at, at the same time responsibility. In a sense, a business unit can build whatever infrastructure, whatever platform.

50. 11:10

    Um, some organization have a very, very strong, tight abstraction of the service infrastructure and typically kind of have to use their platforms, right? So Bloomberg is kind of in the middle.

51. 11:22

    If you look at the golden ones, we kind of believe in provide a golden path, um, with enablement teams. So, so my team is really a enbal- enabling team.

52. 11:35

    And one of the guiding principle for us is we want to make easy things extremely easy to do. Uh, sorry, the right things extremely easy to do, and we wanna make sure the wrong thing is ridiculous hard to do.

53. 11:47

    So that's the guiding principle here. Now, move on. So what is the paved path here? So the paved path is, uh, we have a gateway so that teams can easily figure out which model works the best.

54. 12:00

    They can do quick experiments. They can, um, we can have visibility of what kind of models are being used, and we can also guide through teams which model should, is a better fit for the s- for the problem they want to solve.

55. 12:12

    Uh, we have a tool discovery, uh, basically MCP directory via hub so that y- let's say team A wants to do something, they will go to the hub, they will, okay, someone's building an MCP server already, maybe I should partner with them to build it together, right?

56. 12:26

    Uh, tool creation and deployment is via a pass. Uh, it's basically a, um, you know, a, a standard platform of service where you can do your SDLC and, and we provide runtime environment for you as well, taking care of all auth and cyber things as well.

57. 12:41

    So it really reduce the friction of, for, for teams to, to deploy, um, their MCP, MCP servers.

58. 12:48

    And then the, this is kind of in- interesting is we want to make demo very easy so that, or actually we say proof of concept very easy so that people can try have idea generation, uh, because we believe in creativity come from some freedom of try different new things.

59. 13:05

    But we also want to make sure the production requires some quality of the control, um,

60. 13:12

    because at the end of the day, stability and system reliability is, is at the core of our business. This is sort of the pay path that we deployed, um, and enabled the rest of engineering, really the nine thousand software engineers, to do their job.

61. 13:27

    Okay. And, um, with all this, and then we start maybe, okay, yes, we've got pa- uh, pay path, we have some good ideas of how to evolve our code base.

62. 13:39

    How about our people? Right. Um, now this is where I find that

63. 13:47

    any new things, any adoption of new things provide opportunity to leverage the strengths you have and also identify the, some of the weakness that you may have. So, um, in Bloomberg, we have a well-established training program.

64. 14:01

    Uh, it's more than twenty years. So there's onboarding training, depends on entry level, it depends on senior level. Um, so we have this whole training program to prepare folks to, uh, before they join a team.

65. 14:12

    And what we did is we just incorporate AI coding in onboarding training program and also show them how to best utilize them with our principles and our technologies, right?

66. 14:23

    There's a huge benefits here because, um, if any of you run into the challenge of adoption somehow run into a chasm, right, the rest of org is not, uh, adopt as quick as possible.

67. 14:34

    Whenever we have folks join a company, they learn how to do things in new way that when they go back to their team, they were like, "Hey, why don't we do that?"

68. 14:41

    Right? They're gonna challenge the, some of the senior folks as well to say, "Hey, there's a new way to do this type of things, so why don't we do that?"

69. 14:47

    So we actually find this program extremely effective, uh, to be a change agent for anything we want to push out.

70. 14:55

    And then bunch of results. There's a lot more famili- familiarity and comfort with the tooling. Um, and also the, the important part is there's a lot more nuanced insights of where it's add value, right?

71. 15:08

    The second one is, um, oftentimes we run organization to push, uh, new initiatives. So within Bloomberg, we have something called, um, a champ program and a guild program. That's basically across organizational tech communities where people have similar interests and similar passion.

72. 15:27

    They get together and get stuff done. So, um, we had this for more than ten years now. Uh, we sort of bootstrapped engineer AI productivity community two years back, leveraged the, the community we have already, and then have some few results.

73. 15:44

    Um, because we have this, pretty much everyone passionate about this and will be in that community, so

74. 15:50

    organically it deduplicates efforts and there's shared learning, uh, shared learning happening.

75. 15:57

    And it also helps to boost inner source contributions and the viz engineer idea, right? Oftentimes team A wants to do something, team B, let's say a platform team, have different prioritization, and the way we solve this is via inner source or via visit engineer, which is move someone over the team, work for six months, a year, get

76. 16:15

    it done, and then we can move on. Um, the last one is interesting. So our data shows individual contributors have a much better, stronger adoption than our leadership team.

77. 16:28

    Now, if you think about this, a lot of software TLs and managers,

78. 16:34

    in the age of AI, they kind of don't really have, um, enough experience to truly guide their teams to build software, right? So oftentimes the stuff that they learned before might not be exactly applicable.

79. 16:49

    It's still very valuable, but there's some missing piece there to make sure they can continue to guide the team to do the right thing. So we're rolling out leadership workshops to make sure our leaders are equipped with whatever knowledge they need to have to drive the technol- um, innovation.

80. 17:05

    So, um, I'm going to close my part and to share with you what, uh, the, the part I'm, I feel most excited about. The part I feel most exci- uh, most excited about is that with a lot of, um, creativity and innovation in the gen AI space, it actually changes the cost function of software engineering,

81. 17:29

    meaning the trade-off decision of whether we do something versus we don't do something actually changes because some of the work become a lot cheaper to do and some work become a lot more ex- expensive to do.

82. 17:41

    I tend to think it is a great opportunity for engineers and engineering leaders to get back to some of the, uh, basic principles and sort of ask a soul-searching question, "What is a high-quality software engineering and how can we use a tool for that purpose?"

83. 18:00

    So that's it. Thank you very much. [audience applauding] [upbeat music]