Your Support Team Should Ship Code

Talk transcript

1. 0:00

   [on-hold music] I'm so excited to tell you about how at Zapier we are empowering

2. 0:25

   our support team to ship code. Before I tell you about that, has anybody here visited the Grand Canyon?

3. 0:33

   It's a good amount. Anybody rafted through the Grand Canyon?

4. 0:38

   I see one person. I just got off an 18-day trip rafting through the Grand Canyon over 200 miles. It was incredible. No internet, no cell service. The moment I got off, I found out I was giving this talk.

5. 0:52

   I didn't think about, uh, work at all on the river, but once I got off, I started thinking about the parallels between the Grand Canyon and Zapier, and we have one thing in common, and that is erosion. [laughs]

6. 1:07

   Now, natural erosion happens over millions of years with wind, water, and time. It creates the beautiful canyon that we experience, and it's never stopping, always continuing. At Zapier, we have over 8,000 integrations built on third-party APIs, and they are constantly changing, which I'm now thinking of as app erosion.

7. 1:32

   We've been around for 14 years. Some of our apps are that old. API changes and deprecations impact us and create reliability issues. Again, it never stops.

8. 1:45

   So I like to think of our apps as like layers in the Grand Canyon, and they need constant attention. So if we were to create our own Zapier canyon, and our apps would be at the walls, here's our support team flowing down the middle, watching out for app erosion, and we have a backlog crisis.

9. 2:07

   Tickets were coming in faster than we could handle them.

10. 2:11

    Creates integration reliability issues, poor customer experience, even churn. So to solve for app erosion, we kicked off two parallel experiments. The first was moving support from just triaging to also fixing these bugs.

11. 2:29

    It's experiment number one. And experiment number two, we're asking, "Can AI help solve app erosion faster?"

12. 2:37

    So let's jump into experiment one. This got kicked off two years ago. But had to start with the why. We needed to get that buy-in to empower our support team to ship code.

13. 2:49

    So app erosion is one of the major sources of bugs coming through to-- from support to engineering, so there's a big need. Support is eager [laughs] for this experience to-- A lot of them wanna go into engineering eventually.

14. 3:05

    And unofficially, many support members were already helping to maintain our apps.

15. 3:11

    This moves us into how we started this out. Put on some guardrails. We started with just four target apps to, uh, focus our fixes on. Engineering was set to review any merge requests coming from support, and we kept the focus on app fixes.

16. 3:29

    So jumping into experiment two, this is what I've been leading for the last couple of years. How can we use codegen to help solve for app erosion? And so fortuitously, the name of this project is Scout, which ties in so well to the Grand Canyon experience that I've just been through.

17. 3:48

    As any good product manager, we start with discovery. We did some dogfooding, so I shipped some app fixes. Uh, we shadowed engineers and support team members as they were going through the app fix process.

18. 4:02

    We designed out, uh, what are the pain points experienced along the way, what are the phases of the work, and how much time is spent.

19. 4:11

    One big discovery we had is how much time is spent gathering the context, going to the third party AP do- API docs, even crawling the Internet looking for information about a bug that's emerging.

20. 4:24

    Maybe somebody else has already discovered and solved for it outside of Zapier. Internal context, logs, all of this is a lot of context to go and search for as a human, uh, and a lot to grok and work through.

21. 4:41

    This is something we knew we needed to solve for.

22. 4:45

    Where we started with all this great, uh, opportunities and pain points is we started building APIs that we believed would solve for these individual, um, pain points. And some of these APIs are using LLMs to, you know, for our diagnosis tool, gathering all that context on behalf of the, uh,

23. 5:10

    support person engineer and curating that context and building a diagnosis that's using an LLM. And then some aren't. Like, we have a unit test, uh-- Well, unit test generator is, but the, um, test case finder is simply using a search query to look for the right test cases to pull in for your unit test.

24. 5:33

    We built a bunch of APIs. We had a bunch of great ideas, so there was a lot for us to test with, but we ran into some challenges in this first phase.

25. 5:42

    We had APIs, but they were not embedded into our engineers' process. So our tool, I just said they don't like to go to so many web pages to find all their context.

26. 5:54

    They would love all this information to come to them. And yet, our web interface where we've, we've created a playground we call Autocode internally

27. 6:02

    Where you can come and play around with our APIs. And our ask to the teams was, "Come try out our APIs and give us feedback." Now, this is just one more window to go to, so we didn't get a lot of engagement.

28. 6:18

    Also, because we had shipped so many, uh, APIs, our team was spread pretty thin. Cursor launched at the same time, which has gotten great adoption at Zapier. We're all huge fans of Cursor.

29. 6:32

    But from our side, it made some of our tools no longer necessary. But there was one major win in this phase, which is one of our APIs became a support darling.

30. 6:42

    It's Diagnosis, that number one pain point of needing to go out and find all of your contacts, curate it for yourself, so you can start solving the problem. We were doing that on, uh, s- the support team's behalf with the Diagnosis API.

31. 6:59

    And support loved it enough that they decided to embed it into their process. They asked us to build a Zapier integration on our Autocode APIs, so they could embed it into their Zap that creates the Jira ticket from the support issue, and now Diagnosis is included.

32. 7:19

    So embedding tools is the key to usage, as we find out, so how can we embed more of our tools? Well, then MCP spins up, and that solves our problem.

33. 7:31

    We can now embed these API tools into our engineers' workflow. Specifically, our engineers are pulling in these MCP tools as they're using Cursor. Our builders using Scout MCP tools are leaving the IDE less, spending more time in one window.

34. 7:53

    Still coming into challenges. One of our, uh, our, our key tool Diagnosis, uh, is so valuable to pull all that context and to provide a recommendation, but it takes a long time to run.

35. 8:08

    Now, we might run down that runtime, however, as you're working synchronously on a ticket in your IDE, this was frustrating. We also weren't keeping up with the customization needs.

36. 8:18

    Not only did MCP launch and we started leveraging it, Zapier MCP launched too, and some of our tools, if we weren't keeping up with the customization needs, our engineers internally mo- looked to Zapier MCP, which is great.

37. 8:33

    We're all on the same team solving the same problem, but some of our tools had a dead end.

38. 8:39

    Also, adoption was scattered. We had a whole suite of tools, and we thought there was value in each of them as a solver of different problems across, uh, different stages.

39. 8:49

    Not every engineer was using our tools, and if they were using our tools, u- they're only using a few of them.

40. 8:55

    So we have tool usage. We're happy about that. But we were under the hypothesis that true value is gonna come from tying these tools together.

41. 9:06

    So what if we owned orchestration of these tools? Rather than saying, "Here's a suite of tools, you use them as you wish," what if we combine them and created an agent to orchestrate this?

42. 9:18

    So this we are calling Scout Agent. We take that diagnosis, run that against a ticket,

43. 9:26

    uh, use that information to actually spin up a codegen tool, which will then produce a merge request using all the right context.

44. 9:35

    So who would benefit the most from orchestration? There are several integration teams at Zapier who are solving for these app fixes of various levels of complexity, and there's the support team.

45. 9:47

    So when we're saying, "Who should be our first customer of Scout Agent?" We were thinking, "Hmm. It should probably be the, the team fielding small bugs that are emergent and coming hot off the queue," which is the support team.

46. 10:00

    And now our two experiments merge, and we have Scout Agent we are building for the support team.

47. 10:09

    And this is the flow of how it works. Support is submitting an issue to Scout Agent. We first categorize the issue. We next assess its fixability. Not every issue that comes from support can be fixed.

48. 10:26

    If we think it's fixable, we'll move on to generating a merge request. At that point, the support team, this is the first time they're picking up the ticket, it already has a merge request attached to it.

49. 10:38

    They'll review and test. If it's not satisfying what they believe is the actual solution or the, the, what, what the solution should be to best address the customer's need, they will make a request for an adjustment.

50. 10:53

    That can happen right in GitLab, which is where we do our work, and Scout will do another pass. And hopefully at that point, we've gotten it right, and support can submit that MR for review from engineering.

51. 11:05

    How we are running Scout. It's all kicked off by a Zap. This is a picture of one of our Zaps. There are many Zaps that's run this whole process, and it embeds right into our support team's Zaps.

52. 11:16

    We do a ton of dogfooding at Zapier. We first run Diagnosis and post that result to the Jira ticket, saying what the categorization is, if we believe it's fixable.

53. 11:27

    And then if we do believe it's fixable, we then are kicking off a GitLab CICD pipeline, and we run three phases in that pipeline: plan, execute, and validate, generate this merge request.

54. 11:40

    The tools used in this pipeline is Scout MCP. So all those APIs we invested in a year ago now are really coming together, and we're orchestrating it, uh, within the GitLab pipeline, and we're also leveraging Cursor SDK.

55. 11:58

    Once the m- merge request has been completed, we attach it to Jira, and support picks it up.

56. 12:04

    The latest addition to this is doing a rapid iteration once a, um, uh, once a ticket has been posted with the merge request and support team is looking at it, and they say, "You know, it needs some tweaks."

57. 12:19

    To save them more time so they don't have to go pull that down to their ID, do the fixes, and push it back up, they can simply chat with the, uh, Scout Agent in GitLab.

58. 12:29

    That'll kick off another, uh, pipeline, which does that phase with that new feedback and hosts the new merge request.

59. 12:39

    On our side, we wanna make sure Scout Agent is working, so we ask three questions. Is the categorization right? Is it, was it actually fixable? Uh, and was the code fix accurate?

60. 12:50

    So far, we have two evals, 70 to 75% accuracy for categorization of fixability. As we get more feedback and process more tickets, those become our test cases, and we can move forward improving Scout Agent over time.

61. 13:06

    So what has been Scout Agent's impact on app erosion?

62. 13:11

    40% of supports, support team's app fixes are being generated by Scout, so we're doing more of the work on behalf of the support team. This is resulting in, for some of our support team, it's doubling their velocity from one to two tickets w- per week, which already is amazing.

63. 13:31

    That's going from a support team that wasn't shipping any fixes, well, unofficially they were sometimes, to now shipping one to two per week per person, to now shipping three to four with the help of Scout.

64. 13:44

    Another, uh, process improvement, Scout puts potentially fixable tickets right there in the triage flow. Takes away a lot of the friction of looking for something to grab from the backlog.

65. 13:57

    It's not just the support who's benefiting. It's also engineering. Its engineering manager said, uh, "It's a great example of when it works. This tool allows us to stay focused on the more complex stuff."

66. 14:10

    If you take away anything from this talk, I hope it is that there is a really powerful magic between support and empowering them with codegen and allowing them to ship fixes, 'cause they have three superpowers.

67. 14:23

    The first, they are the closest to customer pain, which mean that they're closest to the context that really matters for figuring out what's the problem and how to solve it.

68. 14:33

    They're also troubleshooting in real time. These tickets aren't stale. The context is fresh. The logs aren't missing. You put this ticket into a engineering backlog months later, you might not get access to those logs anymore.

69. 14:48

    And then three, they're best at validation. You've-- Again, you put the same ticket into an engineering backlog, the solution an engineer might come up with may change the behavior, and that might be good for some customers, but might not necessarily be best for that one customer who wrote in about the problem.

70. 15:10

    And one other major benefit of this is eng-- or support team members who have been part of this experiment are now engineers.

71. 15:22

    I wanna say thank you to the amazing team who's helped build this process or build all the tools and the Scout Agent. Andy is actually here in the audience, so shout out to Andy.

72. 15:32

    If you wanna talk about any of the technical bits, he's here. And I wanna impress upon you two things. We're hiring, but mostly, if you haven't rafted through the Grand Canyon, please consider it.

73. 15:43

    It's life-changing, and you should go with oars. Thank you very much. [audience applauding] [upbeat music]