How Building with AI Can Double the Throughput of Your Engineering Team

Talk transcript

1. 0:00

   [upbeat music] Uh, hey, I'm Brian from Intercom.

2. 0:17

   Uh, and this has been such a great conference so far. I've learned so much alpha and inspiration from the talks and all the chats with people. Um, so Intercom is a 15-year-old privately held Irish American B2B SaaS startup that has pivoted to be an AI company the weekend that ChatGPT came out.

3. 0:33

   Uh, we've got about 1,400 people across Dublin, London, Berlin, SF, Chicago, Sydney. R&D is led from Dublin. Uh, engineering is almost entirely across Europe, four deployed engineers have kind of changed that up a bit.

4. 0:44

   Um, and this graph compares our revenue growth to the growth rate of publicly traded SaaS companies over the last few years. Um, and you can see publicly traded SaaS companies kind of on the down.

5. 0:55

   Intercom, this l- amazing U. Um, and like we're bucking this downward trend that SaaS companies, uh, have been suffering from recently. Uh, and now I'm gonna shut Intercom down live on stage. [laughs]

6. 1:08

   Uh, I, I once did a live deployment during a talk, and I thought that was impressive. Um, so like Intercom has become the poster child for companies redefining themselves in the age of AI.

7. 1:18

   New York Times recently did a article about SaaS companies reinventing themselves that prominently featured Intercom. Um, and being an AI company means a lot more just than s- slapping on, you know,

8. 1:30

   lightweight wrappers or, like, autocompletes in text field or whatever. Um, our agent, our AI agent for customer support, Fin, has over 8,000 con- customers, industry leading average resolution rates.

9. 1:44

   Uh, revenues like approaching 100 million. Launched the day GPT-4 came out, first product actually released on GPT-4. Uh, and we've been building AI features since about 2018 or so.

10. 1:55

    Um, and but, you know, the modern LLM models have unlocked huge capabilities for dealing with customer support questions, completely obvious. Um, and companies like Anthropic, Snowflake, Linear, Glean, LaunchDarkly use Fin for their customer support.

11. 2:09

    Um, so maybe SaaS isn't dead. Uh, and it works well for all size businesses. Um, also we recently announced that we have our own model serving 100% of Fin, uh, like English text-based conversations, um, outperforming frontier models like Sonnet.

12. 2:25

    Um, cheaper, faster, better, uh, and we're at like about two, 2 million resolutions a rate, uh, resolutions a week. Um, and we're also happy to sell direct access to our suite of models.

13. 2:37

    Uh, I'm not talking about any of this, though. Uh, so I'm a senior principal engineer at Intercom, been there for 12 years, and I'm on our platform group, and we take care of Intercom's uptime, performance, security, cost management, observability, uh, and our majestic monolith applications that we love, mostly Ruby on Rails, um, and all internal developer productivity.

14. 2:56

    Um, and another thing about Intercom is that we are obsessed with shipping. Uh, ship- sh- shipping fast and iteratively is the best way to build high-quality products that customers love to use.

15. 3:05

    Um, and so developer productivity is something we've always invested in. Shipping is the heartbeat of your company. It's great blog post that we po- did, like many, many years ago, and Honeycomb made cool stickers of it.

16. 3:16

    Um, and obviously for the last few years, I've been spending a lot of time, uh, on enabling use of AI in our software development life cycle. So I'm gonna kinda talk about that.

17. 3:25

    Um, and so unsurprisingly, we've been very excited, uh, about AI in general. You know, ch- changed the whole company to bu- to build customer support, uh, using AI agents.

18. 3:35

    Um, and we've been impatient about getting its adoption and changing how we build across Intercom. Uh, you know, we went down some kind of familiar routes. You know, we're all using GitHub Copilot, and, uh, then everyone started adopting Cursor, and, like, we looked at Augment and a few other things.

19. 3:52

    But ultimately, you know, say middle of last year, uh, we've been dissatisfied with the results. Uh, some good signs, some kind of tasks, some work, uh, made marginally better and kind of more fun.

20. 4:03

    Um, but you know, w- we- we're pretty aware of where the models are going and the harnesses, and we have a strong conviction that AI, uh, like f- for many years ago, is gonna change all knowledge work.

21. 4:15

    Uh, so last year, uh, middle of last year, we set a simple goal. Let's double the throughput of engineering in a year. Um, and you know, we measure a lot of things in Intercom.

22. 4:24

    We use like... We do a lot of developer surveys. Uh, we use tools like DX and, uh, but we picked code changes per R&D person as the primary way we're measuring productivity.

23. 4:36

    N- every measure is bad. Once you start measuring it, it's not a measure and all this. Um, but also, like, we're, like, impatient about, uh, or, like, expect the overall throughput to increase.

24. 4:47

    Like, if we're, uh, like actually adopting new ways of working, putting AI into all of the different places, uh, then we should expect a large throughput increase. Um, and so 2X, what we, we call this 2X.

25. 5:01

    It's the name of the project, and now a team and everything. Um, this is like wildly ambitious, like when we published this back last June or something like that, um, doubling productivity without doubling team size, but also kind of wildly unambitious as well if you, like, connect the dots and see where the models and coding harnesses are

26. 5:17

    going. So in this talk, I'm gonna talk about how we went about this, how we think about productivity, and a sneak peek at some of our internal data and skills and stuff.

27. 5:25

    Um, you know, this also coincided s- the work, uh, here with, like, the most noticeable shift in model capability and coding capability. Um, and so, you know, we've all seen it, and this was, like, one of our principal engineers, uh, posting just kind of like everyone else was, uh, in around the Christmas break last year, going like,

28. 5:44

    "Oh my God," like, "th- things have changed massively." And so, uh, that has contributed a lot to our success on 2X. Um, so this is the kind of engineering leadership-y part of the talk.

29. 5:54

    Uh, and so you need to be decisive and give clear executive guidance and, uh, you know-

30. 6:02

    Do organizational change. Uh, and, uh, we've done a lot of things. We updated job descriptions. If you're not adopting AI in Intercom, whether you're a designer, product manager, engineer, or whatever, you are not meeting expectations.

31. 6:14

    Binary. Um, and yeah, you have to say the same message over and over and over, 100 times, every different forum, whatever. You just gotta stay on message and constantly talk about the urgency of us doing this.

32. 6:28

    Um, you gotta reward us as well. Like, when people do good stuff, you gotta, like, all the Slack channels, um, uh, showing... Like, automating where people... Like automating, um, when people update skills or do this, that, and the other.

33. 6:40

    It's like, it gets put into these channels. We celebrate stuff. Uh, people are kind of showing each other different techniques and what's working for them and that kind of thing.

34. 6:47

    Uh, we've done hackathons, we've done AI immersion days, and, you know, all of these things are necessary to kind of bring people along. Like, um, also, we staffed this full time.

35. 6:57

    Uh, we have a team 2X, uh, that seems to be just keeps on growing and growing and growing. Um, and, you know, we're, we're not just saying, "Hey, you gotta AI everything.

36. 7:07

    Best of luck." Uh, we're, like, trying to bring everyone, like the hundreds of engineers, hundreds of people in R&D, along with us. So, you know, if you're in a medium or large organization, you absolutely need to have people, and, like, your best people, uh, on this full time.

37. 7:22

    Um, and so we chose Claude Clo- C- Claude Code as our platform. So, uh, prior to this, we were kind of omnivorous and, like, letting people choose their favorite editor and this, that, and the other.

38. 7:34

    And, uh, you know, there's, like, loads of people adopting co- Claude Code, loads of people using Cursor, loads of people using Augment. Um, but, uh, we, like, we're a believer in platforms in general.

39. 7:45

    Um, and it kind of doesn't matter what you choose. Uh, but choosing one is important. Uh, you know, to a certain extent, you need to get away from model anxiety.

40. 7:53

    It's like being multi-cloud. It's like you don't get the compounding benefits of a well-designed platform if you're sending all your different work across different cloud providers or whatever. Um, you're way better off being all in on one and optimizing it and proving that it works.

41. 8:08

    Um, and, like, unless there's, like, very specific or impactful reasons why you need to be spread across multiple agents or whatever. Um, and so our vision on this was, like, to treat Claude or to, like, work on, to get Claude to be able to act like a senior engineer on any technical task across of Intercom.

42. 8:24

    Um, and our vision here was, like, connect Claude to everything. So anything I do on my laptop, Claude should be able to do that. And that means everything. Like, uh, now of course, we're not reckless.

43. 8:33

    We're not, like, just trying to, uh, let the thing go off and delete all of our databases, but we're, like, a mature company. We've got plenty of controls and permissions and, uh, audits and everything like that.

44. 8:45

    Uh, that gives us a lot of confidence to be able to, like, unleash Claude in the same way that we unleash our engineers in our environments. Um, and you know, we gotta onboard it.

45. 8:53

    We gotta teach it all the stuff that we teach people when they join Intercom. Uh, all of our Rails conventions, like our architecture, React patterns. Like, we've built a lot of software in 15 years.

46. 9:03

    Um, but, like, stand- testing standards, security rules, all this, Claude h- absolutely has to know the Intercom-specific information to be able to do the job. Um, and, uh, and most importantly, start using the platform for all technical work, and it doesn't get things right first time, hits an issue, goes down the wrong path, update the guidance.

47. 9:21

    Like, this is a flywheel that we're all contributing to. Um, and so we've encapsulated a lot of this knowledge in context and engineering, captured the skills, guidance, hooks to force these things.

48. 9:29

    We spent a lot of time cajoling Claude Code to work well. Uh, we do things like, uh, push out our internal Claude plugins, um, to everyone's laptops, like bypassing the, all the Claude Code updates, uh, mechanisms.

49. 9:43

    Um, because just, it's, you know, it's, uh, you spend a lot of time debugging, uh, Claude Code installs on like hundreds of laptops. It's like trying to install Python or manage Python installs or something.

50. 9:53

    Um, and so ultimately though, like every single part of technical work, so it's not just code production. It's not like more advanced autocomplete. It's everything. Um, so debugging, testing, planning, all this kind of stuff, uh, it should just be you driving Claude, and ideally, like, driving it less and less and moving higher up the, the, the, the

51. 10:11

    food chain. Um, and it, you know, delivers real value, delivers the code, products, whatever, to, uh, customers. So everything's in scope. And, like, we think that even if the models and harnesses do not improve at all, uh, which is not, definitely not happening, um, if anything, like, this capability curve is, uh, is accelerating.

52. 10:31

    But, like, the building, we have the building blocks today to, um, to improve... Like, basically move vast amounts of work, uh, in our software development l- lifecycle to be agent-first.

53. 10:40

    Like, they could just pause everything, and we've just, uh, got this flywheel, and we're going through everything and looking at every single piece of work. And, like, the tools are good enough today to do this.

54. 10:52

    Um, so we wrote some principles to help guide us along the way. You know, when you have hun- you're trying to get hundreds of people to change how they work or understand what we're trying to achieve, you need to write things down and help them out.

55. 11:03

    Um, and you know, different principles should apply in different places. But, like, um, you know, we believe that all of engineering is changing. Everything that you can do, the agent must be able to do.

56. 11:13

    Uh, and that, that can feel weird as well, like, when you're first connecting it into production systems or whatever. Um, and, uh, yeah, like the, our job is moving up the stack as engineers, as product builders, whatever.

57. 11:25

    And, like, if I, like, a long time ago, I used to be a, a Unix sysadmin. Um, and, uh, you know, you'd, like, going into data centers, racking servers, uh, cabling things, configuring networks and all that.

58. 11:38

    Um, and then the cloud came along, and I moved up the stack. I, you know, and pe- people transitioned from being sysadmins to SREs. The work was more automation oriented, more impactful, higher paid.

59. 11:50

    Uh, and so I think this is like we're kind of speed running this 100 times faster on a full industry scale. Uh, but I kind of feel like I've been through this before.

60. 11:57

    Um, we at Intercom are technically conservative. We like using single tools and just using them extremely well. Um, so hence we end up with these Ruby on Rails monoliths and stuff.

61. 12:07

    Uh, and so we're kind of applying this thought process as well to like, uh, you know, what is the... Where should our focus be? Where is our attention? Do we want everyone writing their own multi-agent orchestrators or opinionated workflows?

62. 12:19

    And, you know, the, we want to build durable, testable, high-quality components and people to be considering like the lifetime value of what they produce. And like, you know, it, the, the tools, the specific implementations of these things will change over time, but, uh, I'm pretty sure that writing down how to do work in Intercom will be valuable

63. 12:36

    no matter what happens. Uh, maybe it might be easier to discover in the future. That's like, uh, a, a problem at the moment. Um, and so what this, what this really means in practice is that we spend our time focusing on small, high quality, durable, testable skills that do the job extremely well, that we can, you know,

64. 12:52

    use data, use backtesting. We've got like all of the work. We've got this huge body of work and changes in code and incidents and everything, and so we're using all of this to help form us and prove out that these skills are operating at extremely high quality.

65. 13:06

    Um, and, you know, we, uh, then we, and we also practice continuous improvement here, get these things to be self-updating, um, make sure that these things are very high quality.

66. 13:15

    Um, and, uh, yeah, we don't want to get stuck behind the curve, like getting stuck because we've implemented a lot of our own s- own, own things. We just want to use things as they become available, as in topic ship or whatever.

67. 13:28

    Uh, and maybe we mightn't stay on topic forever, but like, uh, we're very, we, we're eager to get the advantage of somebody else building and shipping great software and capabilities rather than us having to build everything ourselves.

68. 13:38

    So, uh, yeah, another thing we guide people to, to do is like you want to give problems agents, not tasks. You know, a lot of the time people even say in Intercom are saying like prompting agents, "Hey, run this skill to do a thing."

69. 13:49

    Uh, which is mostly fine and still kind of necessary. I still do it, um, a lot. But like, uh, we're more kind of having to like move ourselves to be kind of just describing the problem or de- de- des- describing the task, uh, and let, let the, uh, agent figure out what skills to invoke and what to

70. 14:05

    do here. Um, I've... Fun story. Recently, I was brought into a security incident. We had accidentally published some kind of

71. 14:13

    Snowflake table metadata to a public GitHub repository. Uh, and I just habitually, uh, opened Claude Code, told it to join a Slack channel, take a look. Um, and I didn't even know that a skill existed that, uh, actually perfectly encapsulated all of our like data breach policies and criteria and what to do, how to analyze this.

72. 14:33

    Claude just automatically downloaded, uh, the, the files, did full analysis, concluded it was innocuous, told me all next steps. Um, and I, like I didn't tell it to do this.

73. 14:41

    It just kind of figured it out. It was done in like two minutes. Um, and like that would've been a 20-minute task and kind of boring work. I'd have to go, "Oh, where's that policy?"

74. 14:48

    And take a look at this, that, and the other. Um, and like this just felt like a little... Like it was a small example, but it's like, again, I just like gave it the problem of like taking a look security incident, and it just figured out the intent, uh, and used a well-written internal skill that did this

75. 15:03

    job for me. Um, and yeah, it, I mentioned that it, even at Intercom, like AI adoption is unevenly distributed. I think we're ahead of the vast majority of companies.

76. 15:13

    Uh, but you still need to help people understand where they're at and grow towards being highly effective at using agents in their work. Uh, CV Aggie recently talked about like maturity rating for engineers, and like our internal one is kind of similar here.

77. 15:24

    You're kind of like trying to get through these different kind of levels, and like ultimately you kind of end up like mastering all the skills and like knowing the tool inside out.

78. 15:32

    And ultimate- like what we want people to do is like use Claude Code for everything, automate your work, then move that to a skill, then get really good at writing skills, and then writing s- write skills and improve the skills, uh, and then optimize the environment for agents.

79. 15:45

    That could be everything from software architecture, maybe just to documentation or other approaches or other ways of doing things, uh, that allows the agents to be even more effective and optimized for what they're great at today.

80. 15:55

    So here, here's where we're at. As you can see, yeah, wild inflection points, uh, after, uh, after going all in on one tool. That, that decision was made in December.

81. 16:04

    We started rolling it out in January. Um, and we've been just, like we have reached the doubling PR throughput in faster than one year. Um, here's more like data from our internal dashboards.

82. 16:17

    There's some interesting stuff in here. This is like, yeah, number of pull requests auto Claude Code. It's like in the 90-somethings. Um, you can see also, uh, we're starting to move into...

83. 16:28

    Like our current bottleneck is, uh, code review. And, uh, but you can see we have this like 17.6%, uh, approval rate, uh, of our automatic code approvals, and it's like a lot more, uh, in-depth than just like, "Hey, Claude, can you approve this?"

84. 16:44

    Um, we've gone through a lot of detailed work to figure out, again, using backtesting and previous data, uh, and then getting humans to kind of label the outputs and figure out, like get the confidence level of the automatic approvers and kind of shape the pull requests towards very safe and simple, uh, pull requests, which probably always should

85. 17:04

    have been that way. Um, but now like they're just approved automatically. And, you know, we've also worked with our auditors to ensure that we're fully SOC 2, ISO 227001, HIPAA compliant, all that.

86. 17:13

    You do not need humans in the loop to, uh, to meet these certifications. You need, you do need to know exactly what you're doing though and make sure you've got like auditing controls and everything.

87. 17:23

    Um, and so by moving approvals to an extremely well-organized, tested, and competent suite of agents, um, including codecs for code reviews, uh, I think multimodal code reviews are okay.

88. 17:31

    I just like completely went back on my platform thing. Um, uh, like, uh, we've got a high confidence that like this stuff is not de- degrading environment or adding additional risk.

89. 17:41

    In fact, I think it's removing risk because humans aren't actually as good as, uh, agents, like when they're well-defined. Um, here's like skill invocation. I actually think the earlier numbers were a bit wonky.

90. 17:51

    So like we hook up everything into Honeycomb, all the... We've got hooks all over the place, um, for basic information about like which skills are being, uh, invoked and things like that, and that's internally available.

91. 18:04

    There's no private information, uh, in this, and everyone can kind of use it to kind of get an idea of like what's being used and where. But we also pull in all s- sessions transcripts, uh, into S3 for data mining, writing reports, guiding, like also looking to see are skills effective, that kind of stuff.

92. 18:18

    Um, so we've got like a feedback loop using the session data, uh, which is, uh, we, we can get more out of it, but it's, we're, we're, we're doing some interesting stuff with it already.

93. 18:27

    And this isn't a goal, but like, and we're not particularly proud of, like, defects always increasing, uh, up till recently. But, like, defects are getting closed faster than ever.

94. 18:35

    And, like, some teams have been inspired by the move to AI to, uh, think about things like backlog zero or crunching through hundreds or thousands of, uh, defects. Um, so, like, some of this was, like, a bit deliberate and planned, but just in its...

95. 18:50

    At the same time, there's just, like, this natural deflation 'cause, uh, getting through this work, getting through all the defects so much faster these days. Um, and, uh, yeah, it's like we're, we're just seeing this naturally.

96. 19:02

    We've also been working with, like, Stanford. There's a research group there. Uh, we've, um, we give them all our code, and our, our code quality per their metrics has been increasing over the last while.

97. 19:10

    Um, okay, uh, I'm kind of running out of time at this point. Uh, we have, like, hundreds of contributors, thousands, like, thousands of lines of co- of code, uh, in our Claude Code plugins.

98. 19:24

    Um, it's very active. Uh, and, uh, yeah, c- I mean, Claude Co- Claude itself loves it. Um, here's an example skill. This is, like, not the most... Or, sorry, we got like b- base plugins, things that, like, do all the session transcripts, session syncing, um, uh, some safety hooks and things.

99. 19:40

    Um, and here's, like, a s- a skill I built which, like, it just s- it fixes flaky specs. We have hundreds of thousands of s- of tests and, you know, they get a bit flaky, uh, over time and, uh, we don't...

100. 19:53

     We ship a lot, so we just kinda bar- barge through the kinda flakes. Um, but this, this skill was not built, like, by me kinda sitting down and figuring out just like, "Oh, what are, what are all the things you need to do to fix flaky specs?"

101. 20:04

     I've worked in a feedback loop, gave the, um, gave the agent a goal, and, uh, through, like, guiding us to the right place, uh, and working with us to fix a lot of flaky specs, uh, it's written this pretty decent thing.

102. 20:18

     What are, like, these cheat codes or, like, lookup tables, and, uh, relatively well-organized, it's using pr- progressive disclosure and all that. Um, and, uh, it is like fixing stuff that if our more senior Rails engineers were doing this, I'd be like, "Wow, they're amazing."

103. 20:33

     Um, and yeah, like, other, a lot of other stuff going on, like our CI melted, we had to fix that. Um, Claude Code is actually widely used across Intercom outside of software.

104. 20:42

     It's gone completely viral. People are banging down our doors to, like, use cons- news consoles. Um, and, uh, yeah, we're, you know, we're thinking a lot about, like, the future of engineering pro- like, should we just merge all product manager design, everything?

105. 20:55

     Um, oh, yes, the single-person team product experiments have been pretty interesting as well. And I've even been shipping, like, code, like, stuff that people can use in their agents to sign up to Intercom.

106. 21:04

     Uh, this is stuff that, like, I, like, I've just been using our skills to act as a product manager, which is pretty wild. Um, so that's it. I wish you all the best of luck.

107. 21:13

     If you're not doing pretty much all of this today, you're gonna be doing it in the very near fu- future. Um, my contact details are at brian.scanlan.ie. You can interact with Fin the messenger, configurable CLI, um, and you can check out, uh, ideas.fin.ai for a lot more information about Intercom and our agents.

108. 21:28

     Thank you. [clapping] [outro music]