Building the platform for agent coordination

Talk transcript

1. 0:00

   [upbeat music] So yeah, I'm, I'm Tom.

2. 0:16

   I, uh, lead the engineering team at Linear. Um, and today I would love to talk to you a bit about our story with AI, how we think about AI as a company, uh, some of the features we've built, and then how we see software development going from here, and perhaps Le- Linear's place in that future.

3. 0:36

   Um, so just for anybody that hasn't heard of, of Linear in the room, um, [clears throat] that you might not be familiar. So Linear is a product development tool, um, it's disguised as an issue tracker, we like to say.

4. 0:49

   Um, we've spent the last five years obsessing over the speed, clarity, removing friction, making it just the best tool for ICs, um, to, to use to work every day.

5. 1:02

   So yeah, it started as a simple tracker and, and now we think of it as, as an operating system for engineering and product teams to, to build their products.

6. 1:11

   And we're used by OpenAI, Ramp, Vercel. Thousands of other modern software companies you've heard of, um, use Linear to kind of keep track of their work.

7. 1:25

   So just a little bit of, of history, uh, of, of our AI journey, as it were. Um, we spun up an internal, like, skunkworks team in early 2023, um, which I think was about GPT-3, if I remember rightly.

8. 1:41

   Um, our initial focus was on kind of summarization, uh, some similarity. We were looking at embeddings. We... Nobody on the team had any AI experience, so we're just kind of jumping in and, and figuring it out as we go.

9. 1:56

   Uh, one of the things we realized really quickly was that, uh, many of the features that we needed to build needed a really solid search foundation. Um, almost everything you need to first find the relevant stuff, right?

10. 2:08

    So right, we had Elasticsearch at that time, uh, and they didn't have a very good vector offering. I think maybe they actually had no vector offering back in, uh, 2023.

11. 2:16

    Um, so we looked around, and this was kind of a, a moment where there was, like, 100 startups suddenly came out with vector databases, right? It was like, there was Pinecone, there was this, there was that.

12. 2:25

    And uh, so we looked at these. We evaluated a few. They all had a ton of trade-offs. Um, and so we, we literally just ended up, after experimenting with a bunch of things, when we had, like, OpenAI embeddings, and we stored them in pgvector, and we put the pgvector on GCP, and it was, like, the most classic

13. 2:41

    Linear decision ever because it was so pra- so pragmatic and just use, use the solidest things. Um, so on that base we, we shipped some features, right? We shipped a, a V1 of Similar Issues, uh, where we're kind of suggesting related issues.

14. 2:55

    This was, like, in hindsight, two years later, so naive. Um, we, we were just doing simple cosine embedding op- uh, comparisons, um, against the, the vector database.

15. 3:08

    And we, we shipped natural language filters. I actually think this is one of the better ones, where, um, you can just type in natural language, "Bugs assigned to me in the last two weeks that are closed," and it will produce the filter.

16. 3:19

    So it's very one-shot, very n- naive in comparison, yeah, but, um, pretty useful. And kind of hidden, I would say, as well. And we also had another feature where if you create an issue from a Slack thread, we will, um, not just pass the text from the Slack message, we will, we will try and produce the right

17. 3:36

    issue from that automatically, and that was, like, so seamless and hidden that I think a lot of people didn't even realize it was happening. Um, and we never shipped a copilot.

18. 3:45

    We tried. It was, like... It was copilot season, and, um, we just... The quality wasn't there, you know? We, we have this quality bar, and it did not reach it.

19. 3:53

    So, um, I don't know if it was a lack of imagination for our team, 'cause we weren't, like, AI pilled enough at the time, or, uh, it was, like, the, the capability of these early models.

20. 4:03

    I think a bit of both, to be honest. Um, so you know, we, uh, I think this was the right approach at the time, in a way. Like, a, a lot of people on, on Twitter kind of noticed.

21. 4:14

    They're like, "Oh, these are very seamless features. You're not slapping AI in our face." Like, there was literally toothbrushes that said they had AI. Um, I, I think it's probably much worse now, to be honest, but, you know, people kind of appreciated this approach, uh, of, like, small pragmatic value adds.

22. 4:31

    And then, like, fast-forward to 2024, and, you know, we, we, we've added a few things since then, but it really feels like in 2024, the end of '24, we, we hit a turning point.

23. 4:40

    Um, you know, o3 coming out, the planning and reasoning models, the multimodal capabilities became available in the APIs. The context windows went through the roof. Uh, you know, have, like, million token context.

24. 4:52

    Like, you can do crazy things with that. Um, DeepSeek, of course, made a splash. And, uh, we felt like, uh, some of our experiments started to become a lot less brittle, and things actually felt smart.

25. 5:07

    Um, things kind of clicked for the team a little bit more. The, the... We, we saw how deep this could go.

26. 5:13

    So, uh, the first thing we did was [laughs] we started by, uh, rebuilding our search index again. Um, which, I don't know, if you ever, like, back, backfilled, like, million, hundreds of millions of rows of embeddings, it takes a while.

27. 5:27

    Um, so we moved to a hybrid search approach. This was something that, um, we, we had really felt was lacking over, like, the year and a half that we'd kind of had pgvector sat on its own, and we weren't, um...

28. 5:39

    We didn't put it in our main database 'cause it was so huge, so it was kind of sat in its own thing. So, uh, we moved to Turbopuffer. If you've not heard of Turbopuffer, really, really cool, uh, search index.

29. 5:48

    I'd highly recommend giving it a look. Um, and we moved our embeddings over to Cohere. After doing kind of a comparison, we felt that they were a lot, uh, a lot better for our domain at least, um, than, than OpenAI's.

30. 6:01

    Um, so this kind of filled a gap in the search, um, and this is actually just finished rolling out in, like, the last two weeks, um, because it- the, the backfill took, took such a while.

31. 6:11

    But now we thought we had, okay, we've got a really solid search foundation. What are we gonna do with this? Um, so first thing we did is, like, we're building this feature called Product Intelligence.

32. 6:21

    Um, this is basically like Similar Issues V2. So instead of just doing sim- simple cosine matching, we now have a pipeline. Um, that pipeline is using query rewriting, it's using the hybrid search engine, it's re-ranking the results.

33. 6:35

    We're using deterministic rules, and then out the other side, um, what we get is a map of relationships from an- any given issue to its related issues, and then how they are related and, and the why they are related.

34. 6:50

    Um, and then what we're able to do with that is expose this in the product. I hope that's clear enough as, you know, we have suggested, suggested labels, suggested assignees, um, possible duplicates.

35. 7:03

    And then on things like projects, it's like, why this might be, uh, uh, uh, why this might be the right person to work on this issue, or why this might be the right project for this.

36. 7:12

    So, you know, we're working with, like, the OpenAIs of the world. They have thousands of tickets coming in, and they really have to have as much help as possible to, to kind of churn through them and, and get them into the right, the hands of the right engineers.

37. 7:24

    Um, ooh, I think I skipped one. Yeah, so the next one was, uh, customer feedback analysis. This is something we're working on right now. Um, so one of the other features of Linear is you can bring in all of the customer feedback, um, from all of your channels, uh, and then use that to help to decide what

38. 7:40

    you're going to build. Um, and so obviously one of the steps there is, okay, we have hundreds of pieces of feedback.

39. 7:48

    How do we figure out what to build from this, right? So, uh, of course, LLMs are, are great at analyzing text, and we found that, um, I think our, our head of product actually said that, uh, our analysis was able to beat 90% of the candidates he talks to in the interview process, um, for what they're able

40. 8:05

    to do in terms of, uh, analysis. So, um, we're able to, yes, churn through hundreds or thousands of customer requests and then figure out, for this given project, like, how might we split this up?

41. 8:15

    What, what features might be created from this? Um, which is pretty cool.

42. 8:21

    Uh, another feature we've already shipped is a daily or weekly pulse. This, uh, synthesizes all of the updates that are happening in your workspace, um, creates a, a pulse from it, uh, like a, a summarized pulse.

43. 8:35

    Um, and then we also produce, like, an audio podcast version, which is pretty cool, 'cause you can, uh, pull open our mobile app and then listen to that on your commute.

44. 8:42

    Uh, I hope we have an RSS feed for it soon. I really wanna just subscribe to it in a podcast player. Uh, so although I put podcast here, it's not quite a podcast.

45. 8:50

    You have to have a, a mobile app or the desktop app. Uh, but this is great. You just, like, over breakfast, like, what has the team been up to while I was asleep?

46. 8:57

    Um, oh, that was a, uh... Sorry, that's the, that's the visual of it. Apologies.

47. 9:04

    Um, uh, and then, yeah, so one other feature I'll go through here is, um, is issue from video. So literally, uh, you know, so many bugs can come in as video recordings from customers, right?

48. 9:19

    Drop the video, um, we'll an- analyze it, uh, we'll figure out the reproduction steps, and then we'll create the issue for you, uh, from that. This is maybe not the finest example of that feature, but, um, an- another kind of like seamless, but, but very powerful and saves a bunch of time.

49. 9:37

    Um, so of course we're, we're, we're baking as much into the platform as we can, um, in terms of these things. But, uh, there's a limit to that, right?

50. 9:45

    We, we can't put, put in everything. We don't know... Every team is different. Every team is shaped differently. So we wanna make this pluggable. Um, and this is kind of where agents come in.

51. 9:54

    So the way we're thinking about agents is as infinitely scalable, cloud-based teammates. Um, this is a... So we launched a, a platform for this two weeks ago. Um, we figure, you know, we're already doing a pretty good job of orchestrating humans.

52. 10:08

    Um, we are a communication tool for humans, after all. Um, and if agents are going to be members of your team going forward, then they should also live in the same place where all of the, the human communication happens.

53. 10:19

    Um, so first, hopefully, if the internet stands up, I'm tethering, um, I'll, I'll do some, I've got some, some videos. Uh, yeah, so Codegen is one of the first, um, coding agents that integrated with us.

54. 10:34

    Um, so they can... Is this gonna play? Cool. Yeah, so Codegen, you can assign it, you can mention it, um, inside of Linear like any other cos- uh, any other user, and it will produce plans, it will produce PRs.

55. 10:49

    You can see here, it's gonna pop in. Boop. Uh, this is a sped up, by the way. That took four minutes, not 20 seconds. Uh, but yes, it will produce the PR.

56. 10:58

    Then you can go and review it like you would any other, any other worker, uh, any other team member.

57. 11:06

    Um, this is really powerful, by the way, and you can, because it's, uh, an agentic system in the background, you can also interact with it from, not just from within Linear, but from within Slack or from other communication tools.

58. 11:19

    And you can say, "Go and fix this ticket," uh, and give it a Linear issue, and it will know how to connect it all up. Um, or you'll be able to interrupt it, uh, partway.

59. 11:28

    Um, Bucket is a feature flagging platform that, that integrated with our, the first version of our, um, agents, uh, platform here. Let's see. Is this gonna... Oh, no.

60. 11:40

    All righty. Yeah, so in this case, you can just mention the Bucket agent, tell it to create a flag. It will create a feature flag for you. You can roll it out.

61. 11:48

    Um, you can check the status of things, um, all within here. And of course, because it's agentic, you don't have to go command by command. You could say, "Create a new flag, roll it out to 30% of users," um, and things like that.

62. 12:02

    And then Charlie is another coding agent with access, uh, to your repository. Is really good at creating plans, um, and doing, like, root cause analysis of bugs. So in this case, uh, we have an issue here.

63. 12:14

    It has a Sentry, uh, issue attached. Um, we can just, uh, mention Charlie, ask it to do some research. So it can go and look at your recent commits, um, it can go look through the codebase, and it can kind of figure out, uh, the cause of this issue.

64. 12:29

    And you can imagine immediately, right, like this has saved a lot of, um, of minutes of engineers' time. They can come in here and immediately see, uh, possible causes and regression reasons for, for this issue.

65. 12:44

    Um, so the examples I've shown so far have been, uh, kind of living in, uh, the comment area of an issue. Obviously, that's, uh, not quite where we wanna be in the long term, so, you know, we're working, uh, on build- building, uh, a sp- uh, additional surfaces for this in the product, um, so that agents aren't

66. 13:02

    just, like, the same as, as users on the team. They're, they're kind of better because you can see what they're thinking, uh, and I can't see what my teammates are thinking a lot of the time.

67. 13:10

    Um, so yeah. So we'll, we'll have this surface where the agents can send you their observations, they can send you the, the tool calls. You're able to kind of go behind the scenes of the agent.

68. 13:20

    You'll be able to, um, interrupt it. Um, and then this is kind of consistent across the whole workspace, right? So you have different coding agents, you have PM, uh, PM agents.

69. 13:32

    Um, one other company that's building an integration with us right now is Intercom with their Fin agent. So you'll be able to do things like just say, "Hey, Fin.

70. 13:39

    I fixed this bug. Can you go and, uh, can you go and reply to the 100 customers that reported it?" And, you know, how much time did that just save?

71. 13:47

    So we're building this, this interface out right now, and expect to have it in, in, in a couple of weeks. Um, but I've been really using these features a ton, and, uh, I've been hammering this for months, and I, I think it, it really changes the game and will expect kind of the amount of bugs sitting in

72. 14:03

    companies' backlogs, which we kind of take for granted that you have this giant backlog that you're never gonna get to the bottom of. Um, I think there's just not gonna be an excuse for that anymore.

73. 14:11

    Um, y- the, the agents can tackle it for you. Um, there's nothing to stop you assigning every single issue in your backlog [laughs] out to an agent, have it do a first pass.

74. 14:21

    Maybe 50% of them will be fixed by the end of the week. Um, so I think, yes, we're really in this world now where you can build more, you can build higher quality because more of the grunt work is being done, and you can build faster.

75. 14:37

    How much time we got? So, uh, I'll just talk a little bit about, like, the architecture of this. Um, so yeah. In Linear, agents are first class users. Um, they have identity.

76. 14:47

    They have history. You can see everything they do. There's a full audit trail, um, of those events. Uh, you install them via OAuth, um, and then once they're installed, kind of any admin on the team can manage, uh, manage that, that agent and its access.

77. 15:04

    And they work for- fully transparently. Um, so we have a very mature GraphQL API at this point, um, which basically enables agents to do anything in the product that a human could do, um, and granular scopes.

78. 15:19

    And then we added brand-new webhooks for this specifically, where if you are developing an agent with Linear, um, you will get webhooks when events happen that are specific to your agent.

79. 15:31

    So somebody replied to your agent, your agent was triggered on this issue.

80. 15:36

    Uh, we also added some additional scopes that you can opt into to choose whether your agent is mentionable or assignable.

81. 15:45

    And then as part of that kind of future UI that I just, uh, showed, uh, we're also working on a new SDK to be released at the same time, which will just make, uh, that really, really easy, where you can...

82. 15:56

    So right now, you can build all this stuff. It's on our existing API, um, and you kind of have to figure out a bit more, I would say. So we're kind of building this abstraction layer, this sugar, where you can, uh, very, very easily, uh, integrate with the platform.

83. 16:13

    Uh, yeah. I'll finish with, uh, some, some of the best best practices, um, that we, we found working with these partners over the last couple of months. Um, anyway, it, it really felt like we're kind of on the cutting edge here, and we're building it as the agents themselves still haven't launched in a lot of ways.

84. 16:32

    Um, you know, like Google and Codex only just launched theirs within the last couple of weeks. Um, so first is to be, to respond very quickly and very precisely, um, when folks, uh, trigger your agent.

85. 16:46

    So if I, if I mention your agent, it should respond as fast as possible. A lot of what we've seen is people using emoji reactions for that right now.

86. 16:54

    Um, excuse me. I have to cough. [coughs] Um, yeah. So and then respond in a way that kind of, like, reassures, uh, the user that you, the, the agent understood the request, you know?

87. 17:08

    So it's like, if you say, "@Cod gen, can you take care of this?" The response should be, like, something, "I will produce a PR for, for this specific thing you asked me."

88. 17:17

    And it's like, "Okay. He understood what I meant. Great."

89. 17:21

    Um, inhabit the platform. This is, like, Linear specific a little bit, but, um, in, in this example, but I think it applies anywhere. We, we really expect that these agents are not Linear agents.

90. 17:32

    They are, they are agents that live in the cloud, and one of the ways that they interact is through Linear, right? Uh, it's just another... It's a window into their behavior, and hopefully, like, a re- a really well-structured one, where they get a lot of context.

91. 17:45

    Um, but we really think that, you know, if you're, if you're working within Linear, if you're working within Slack, you should use the language of those platforms and, and not confuse things and, and put great effort into that.

92. 17:55

    And then, um, things like one of the things that we expect to happen inside of Linear is if you're working on an issue, you should move that issue to In Progress.

93. 18:04

    Don't just leave it in the backlog. Um, you expect that of your teammates, and we expect that of agents as well. Um, and then just, again, like, natural behavior.

94. 18:13

    So if, if somebody triggered you, and then they replied in that thread, you, you shouldn't need to @mention the agent again to get a response. It should be a natural behavior that if you reply to them, th- they will, they will respond.

95. 18:30

    Um, yeah, don't be clever. Uh, a lot... [laughs] Clarify your intent before acting. I think we see a lot of, like, attempts at, at one-shots. Um, one pattern that we're seeing right now coming out of a lot of the coding agents is they'll form a plan, um, before doing anything, um, and communicate that plan upfront and get clarification

96. 18:50

    on it. Uh, so that's something that we definitely expect to happen.

97. 18:58

    And finally, you know, be sure you're, you're adding value. Um, I, I think, you know, LLMs, they, they love to just produce tons of text. We don't want to see splats straight out of OpenAI into comments, into issues, into any other services.

98. 19:14

    Um, be concise, be useful, be like a good teammate [laughs] would be. Um, you can always fall back on asking, like, what would a human do in this situation, and try your best to achieve, achieve that.

99. 19:28

    Cool. That's it. Uh, thanks for listening, and if you're interested in working with us on this platform or, uh, integrating with Linear, let me know. [outro music]