AI Engineer Code 2025
Your Support Team Should Ship Code
Read the talk
Your Support Team Should Ship Code
Zapier’s Scout turns fresh support issues into proposed code fixes by gathering context, coordinating tools, and keeping support and engineering in the review loop.
From a talk by Lisa Orr
Before you start: Familiarity with APIs, support tickets, and merge requests will help; no prior knowledge of Scout or MCP is required.
A canyon of changing APIs
After 18 days rafting more than 200 miles through the Grand Canyon without internet or cell service, Lisa Orr returned to work with a useful analogy. Wind, water, and time continually reshape the canyon. At Zapier, third-party API changes continually reshape the maintenance work required to keep integrations reliable. Orr calls this app erosion.
At the time of the talk, Orr reports that Zapier had more than 8,000 integrations built on third-party APIs. The company had been around for 14 years, and some integrations were that old. Changes and deprecations could undermine previously working behavior at any time; maintenance had no finish line.
Imagine the integrations as layers in the canyon walls and support as the river running between them, encountering the places where things are breaking. Tickets were arriving faster than the organization could resolve them. The growing backlog meant unreliable integrations, poor customer experiences, and even churn. Zapier started two parallel experiments: let support fix bugs as well as triage them, and investigate whether AI could make those fixes faster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give support a bounded path to shipping fixes
The support experiment began two years before the talk. App erosion was a major source of bugs passed from support to engineering, so there was a clear maintenance need. There was also interest from support: many team members wanted engineering experience, and some were already helping maintain apps unofficially. The experiment made that work an authorized responsibility.
Zapier started with three boundaries:
- Four target apps: concentrate the initial work on a small, defined set of integrations.
- Engineering review: have engineering review every merge request submitted by support.
- App fixes only: keep the work focused on repairing integrations.
Support could now propose and develop fixes within a defined scope, with engineering remaining part of the approval path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The expensive work before writing code
The second experiment became Scout, the code generation project Orr had been leading over the preceding couple of years. Discovery started with doing the work: Orr shipped app fixes, and the team shadowed engineers and support members as they repaired integrations. They mapped the phases, pain points, and time spent throughout the process.
A major cost appeared before anyone could confidently write a fix: gathering and understanding context. A person might need third-party API documentation, reports of an emerging bug elsewhere on the internet, an external solution someone had already found, internal information, and logs. Finding those sources was only part of the task. The person also had to understand how they fit together and what they implied about the failure.
Scout initially addressed individual pain points with APIs. Some required language models; others were ordinary retrieval tools.
| Tool | Mechanism |
|---|---|
| Diagnosis | Gather and curate context, then use an LLM to formulate a diagnosis |
| Unit test generator | Use an LLM to generate tests |
| Test case finder | Use a search query to retrieve relevant test cases |
The distinction matters: useful assistance included both generating new material and finding existing evidence. There was no need to make every operation an LLM call.
The team built many APIs and exposed them through an internal web playground called Autocode. Engineers were asked to visit it, try the tools, and provide feedback. But the tools were not embedded in the app-fix process. People already disliked opening many pages to assemble context; Autocode added another window to visit. Engagement was limited.
The broad API effort also spread the Scout team thin. Meanwhile, Cursor launched and gained strong adoption inside Zapier, making some of Scout’s tools unnecessary. The first phase had produced useful capabilities, but neither their availability nor their number ensured that people would use them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring Diagnosis to the ticket, then tools to the IDE
Diagnosis was the exception. Support valued having the context assembled and curated before starting a fix, enough to request a Zapier integration for the Autocode APIs. They used it in the existing Zap that created a Jira ticket from a support issue. The ticket now arrived with Diagnosis included. The useful output appeared where support already worked.
That experience made workflow placement central to the next phase. With Model Context Protocol (MCP), the team could expose Scout’s API tools inside Cursor. Engineers could call them while working in the IDE, and Orr reports that builders using Scout MCP spent less time leaving that window.
Embedding the tools exposed another problem: Diagnosis took a long time to run. Its context gathering and recommendation were valuable, but waiting synchronously while working on a ticket in an IDE was frustrating. The team also struggled to keep up with customization requests. When the public Zapier MCP product became available, some engineers used it to meet needs that Scout’s tools did not address, leaving certain Scout tools at a dead end.
Adoption remained scattered. Some engineers used no Scout tools; others used only a few from the suite. The team’s next hypothesis was that more value would come from tying the tools together. Making each tool available still left the builder responsible for choosing the tools and coordinating the work across stages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a tool collection to a proposed fix
Scout Agent put that orchestration under the team’s control. It would run Diagnosis against a ticket, then pass the resulting context to a code generation tool that produced a merge request. This connected the investigation directly to a proposed code change.
Zapier had several integration teams handling app fixes of different levels of complexity. For the agent’s first customer, the team chose support: it was already handling small, emergent bugs arriving fresh from the queue. The experiment in giving support permission to ship code and the experiment in AI-assisted maintenance now converged.
The resulting workflow has an explicit gate before code generation and human review afterward:
- Categorize the submitted issue. Establish what kind of problem support has encountered.
- Assess fixability. Not every support issue is something Scout can fix.
- Generate a merge request if judged fixable. Support first picks up the ticket with a proposed change already attached.
- Review and test against the customer’s need. Support checks whether the proposed solution actually addresses the problem.
- Request adjustments in GitLab when needed. Scout takes another pass using the feedback.
- Submit the merge request for engineering review. Support’s acceptance of the proposal does not remove the engineering review boundary.
The automated output is a proposed fix ready for scrutiny, not an automatically approved change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run the work through Zaps, Jira, and GitLab
Scout runs through multiple Zaps embedded in support’s existing automations. Diagnosis runs first and posts its results to Jira, including the issue category and whether the issue appears fixable. A fixable result then triggers a GitLab CI/CD pipeline with three phases: plan, execute, and validate.
Those phases can be expressed as a GitLab stage declaration:
yaml
stages:
- plan
- execute
- validate
This captures the phase ordering Orr describes; the jobs that perform Scout’s work sit within those phases. The pipeline uses Scout MCP, bringing the APIs built a year earlier into a coordinated process. Orr also identifies a component as “Cursor SDK.” That historical name should not be equated with today’s @cursor/sdk: the public SDK beta was announced in April 2026, after this talk, and Scout’s exact interface is not identified here.
Once the merge request is ready, the workflow attaches it to Jira for support to pick up. Context collection and the first proposed code change have therefore happened before support begins reviewing the ticket’s solution.
The latest addition at the time of the talk shortened the revision loop. If support wanted changes, they could chat with Scout Agent inside GitLab. That feedback launched another pipeline pass and produced a new merge request, avoiding the need to pull the code into an IDE, make local edits, and push it back just to request a tweak.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the decisions separately from the code
The team asks three separate questions about Scout Agent: Was the categorization correct? Was the issue actually fixable? Was the code fix accurate? These distinguish the decisions that route work into the pipeline from the quality of the change it produces.
Orr reports 70–75% accuracy for the two existing evaluations, categorization and fixability. She does not give separate scores for those tasks, a sample size, or a code-fix accuracy result. As the team processes more tickets and receives feedback, those examples become test cases for improving Scout. The evaluation set grows out of the work the system is actually handling.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More proposed fixes, less searching for work
Orr reports that Scout generates 40% of the support team’s app fixes. That share concerns support’s app fixes, rather than all support tickets or all engineering work.
She describes two throughput changes: some support members doubled from one to two tickets per week; in the following account, support progressed from one–two fixes per person per week to three–four with Scout. The relationship between those baselines is not explained, so they should remain separate observations rather than a single calculated productivity gain.
There was also a change in how people found work. Scout placed potentially fixable tickets directly into the triage flow, reducing the friction of searching the backlog for something to take on. The system was helping surface a candidate task as well as prepare a candidate solution.
Engineering benefited too. An engineering manager’s feedback was that the tool allowed the team to stay focused on more complex work. Support’s expanded role absorbed some of the smaller app fixes while preserving engineering involvement in review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the fix close to the customer
Support brings more than additional capacity to code generation. It has three advantages that affect the quality of a fix:
- Proximity to customer pain: support is close to the context needed to understand what is wrong and what a useful solution would do.
- Fresh evidence: troubleshooting happens while the issue is active. Context is current and logs are still available; after months in an engineering backlog, some of that evidence may be gone.
- Customer-specific validation: a behavior change can help some users without solving the problem for the customer who reported it. Support can judge the proposed change against that customer’s actual need.
These strengths explain why support is well placed both to initiate the work and to test the generated proposal.
The experiment also created a path into engineering. Some of the support team members who participated had become engineers by the time of the talk. Giving people a bounded opportunity to ship fixes changed not only who handled the maintenance queue, but also what work those people could go on to do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current instructions for configuring MCP servers and connecting external tools and data to Cursor.
Connect AI clients to actions in Zapier's app integrations through the public Zapier MCP service.
Explains how jobs, runners and stages form configurable CI/CD pipelines.
Further reading
The August 2025 announcement of Cursor Agent's terminal and headless execution modes.
Updates since the talk
The April 2026 release introduces programmatic Cursor agents through the @cursor/sdk TypeScript package.
Read the complete timestamped transcript
- 0:00
[on-hold music] I'm so excited to tell you about how at Zapier we are empowering
- 0:25
our support team to ship code. Before I tell you about that, has anybody here visited the Grand Canyon?
- 0:33
It's a good amount. Anybody rafted through the Grand Canyon?
- 0:38
I see one person. I just got off an 18-day trip rafting through the Grand Canyon over 200 miles. It was incredible. No internet, no cell service. The moment I got off, I found out I was giving this talk.
- 0:52
I didn't think about, uh, work at all on the river, but once I got off, I started thinking about the parallels between the Grand Canyon and Zapier, and we have one thing in common, and that is erosion. [laughs]
- 1:07
Now, natural erosion happens over millions of years with wind, water, and time. It creates the beautiful canyon that we experience, and it's never stopping, always continuing. At Zapier, we have over 8,000 integrations built on third-party APIs, and they are constantly changing, which I'm now thinking of as app erosion.
- 1:32
We've been around for 14 years. Some of our apps are that old. API changes and deprecations impact us and create reliability issues. Again, it never stops.
- 1:45
So I like to think of our apps as like layers in the Grand Canyon, and they need constant attention. So if we were to create our own Zapier canyon, and our apps would be at the walls, here's our support team flowing down the middle, watching out for app erosion, and we have a backlog crisis.
- 2:07
Tickets were coming in faster than we could handle them.
- 2:11
Creates integration reliability issues, poor customer experience, even churn. So to solve for app erosion, we kicked off two parallel experiments. The first was moving support from just triaging to also fixing these bugs.
- 2:29
It's experiment number one. And experiment number two, we're asking, "Can AI help solve app erosion faster?"
- 2:37
So let's jump into experiment one. This got kicked off two years ago. But had to start with the why. We needed to get that buy-in to empower our support team to ship code.
- 2:49
So app erosion is one of the major sources of bugs coming through to-- from support to engineering, so there's a big need. Support is eager [laughs] for this experience to-- A lot of them wanna go into engineering eventually.
- 3:05
And unofficially, many support members were already helping to maintain our apps.
- 3:11
This moves us into how we started this out. Put on some guardrails. We started with just four target apps to, uh, focus our fixes on. Engineering was set to review any merge requests coming from support, and we kept the focus on app fixes.
- 3:29
So jumping into experiment two, this is what I've been leading for the last couple of years. How can we use codegen to help solve for app erosion? And so fortuitously, the name of this project is Scout, which ties in so well to the Grand Canyon experience that I've just been through.
- 3:48
As any good product manager, we start with discovery. We did some dogfooding, so I shipped some app fixes. Uh, we shadowed engineers and support team members as they were going through the app fix process.
- 4:02
We designed out, uh, what are the pain points experienced along the way, what are the phases of the work, and how much time is spent.
- 4:11
One big discovery we had is how much time is spent gathering the context, going to the third party AP do- API docs, even crawling the Internet looking for information about a bug that's emerging.
- 4:24
Maybe somebody else has already discovered and solved for it outside of Zapier. Internal context, logs, all of this is a lot of context to go and search for as a human, uh, and a lot to grok and work through.
- 4:41
This is something we knew we needed to solve for.
- 4:45
Where we started with all this great, uh, opportunities and pain points is we started building APIs that we believed would solve for these individual, um, pain points. And some of these APIs are using LLMs to, you know, for our diagnosis tool, gathering all that context on behalf of the, uh,
- 5:10
support person engineer and curating that context and building a diagnosis that's using an LLM. And then some aren't. Like, we have a unit test, uh-- Well, unit test generator is, but the, um, test case finder is simply using a search query to look for the right test cases to pull in for your unit test.
- 5:33
We built a bunch of APIs. We had a bunch of great ideas, so there was a lot for us to test with, but we ran into some challenges in this first phase.
- 5:42
We had APIs, but they were not embedded into our engineers' process. So our tool, I just said they don't like to go to so many web pages to find all their context.
- 5:54
They would love all this information to come to them. And yet, our web interface where we've, we've created a playground we call Autocode internally
- 6:02
Where you can come and play around with our APIs. And our ask to the teams was, "Come try out our APIs and give us feedback." Now, this is just one more window to go to, so we didn't get a lot of engagement.
- 6:18
Also, because we had shipped so many, uh, APIs, our team was spread pretty thin. Cursor launched at the same time, which has gotten great adoption at Zapier. We're all huge fans of Cursor.
- 6:32
But from our side, it made some of our tools no longer necessary. But there was one major win in this phase, which is one of our APIs became a support darling.
- 6:42
It's Diagnosis, that number one pain point of needing to go out and find all of your contacts, curate it for yourself, so you can start solving the problem. We were doing that on, uh, s- the support team's behalf with the Diagnosis API.
- 6:59
And support loved it enough that they decided to embed it into their process. They asked us to build a Zapier integration on our Autocode APIs, so they could embed it into their Zap that creates the Jira ticket from the support issue, and now Diagnosis is included.
- 7:19
So embedding tools is the key to usage, as we find out, so how can we embed more of our tools? Well, then MCP spins up, and that solves our problem.
- 7:31
We can now embed these API tools into our engineers' workflow. Specifically, our engineers are pulling in these MCP tools as they're using Cursor. Our builders using Scout MCP tools are leaving the IDE less, spending more time in one window.
- 7:53
Still coming into challenges. One of our, uh, our, our key tool Diagnosis, uh, is so valuable to pull all that context and to provide a recommendation, but it takes a long time to run.
- 8:08
Now, we might run down that runtime, however, as you're working synchronously on a ticket in your IDE, this was frustrating. We also weren't keeping up with the customization needs.
- 8:18
Not only did MCP launch and we started leveraging it, Zapier MCP launched too, and some of our tools, if we weren't keeping up with the customization needs, our engineers internally mo- looked to Zapier MCP, which is great.
- 8:33
We're all on the same team solving the same problem, but some of our tools had a dead end.
- 8:39
Also, adoption was scattered. We had a whole suite of tools, and we thought there was value in each of them as a solver of different problems across, uh, different stages.
- 8:49
Not every engineer was using our tools, and if they were using our tools, u- they're only using a few of them.
- 8:55
So we have tool usage. We're happy about that. But we were under the hypothesis that true value is gonna come from tying these tools together.
- 9:06
So what if we owned orchestration of these tools? Rather than saying, "Here's a suite of tools, you use them as you wish," what if we combine them and created an agent to orchestrate this?
- 9:18
So this we are calling Scout Agent. We take that diagnosis, run that against a ticket,
- 9:26
uh, use that information to actually spin up a codegen tool, which will then produce a merge request using all the right context.
- 9:35
So who would benefit the most from orchestration? There are several integration teams at Zapier who are solving for these app fixes of various levels of complexity, and there's the support team.
- 9:47
So when we're saying, "Who should be our first customer of Scout Agent?" We were thinking, "Hmm. It should probably be the, the team fielding small bugs that are emergent and coming hot off the queue," which is the support team.
- 10:00
And now our two experiments merge, and we have Scout Agent we are building for the support team.
- 10:09
And this is the flow of how it works. Support is submitting an issue to Scout Agent. We first categorize the issue. We next assess its fixability. Not every issue that comes from support can be fixed.
- 10:26
If we think it's fixable, we'll move on to generating a merge request. At that point, the support team, this is the first time they're picking up the ticket, it already has a merge request attached to it.
- 10:38
They'll review and test. If it's not satisfying what they believe is the actual solution or the, the, what, what the solution should be to best address the customer's need, they will make a request for an adjustment.
- 10:53
That can happen right in GitLab, which is where we do our work, and Scout will do another pass. And hopefully at that point, we've gotten it right, and support can submit that MR for review from engineering.
- 11:05
How we are running Scout. It's all kicked off by a Zap. This is a picture of one of our Zaps. There are many Zaps that's run this whole process, and it embeds right into our support team's Zaps.
- 11:16
We do a ton of dogfooding at Zapier. We first run Diagnosis and post that result to the Jira ticket, saying what the categorization is, if we believe it's fixable.
- 11:27
And then if we do believe it's fixable, we then are kicking off a GitLab CICD pipeline, and we run three phases in that pipeline: plan, execute, and validate, generate this merge request.
- 11:40
The tools used in this pipeline is Scout MCP. So all those APIs we invested in a year ago now are really coming together, and we're orchestrating it, uh, within the GitLab pipeline, and we're also leveraging Cursor SDK.
- 11:58
Once the m- merge request has been completed, we attach it to Jira, and support picks it up.
- 12:04
The latest addition to this is doing a rapid iteration once a, um, uh, once a ticket has been posted with the merge request and support team is looking at it, and they say, "You know, it needs some tweaks."
- 12:19
To save them more time so they don't have to go pull that down to their ID, do the fixes, and push it back up, they can simply chat with the, uh, Scout Agent in GitLab.
- 12:29
That'll kick off another, uh, pipeline, which does that phase with that new feedback and hosts the new merge request.
- 12:39
On our side, we wanna make sure Scout Agent is working, so we ask three questions. Is the categorization right? Is it, was it actually fixable? Uh, and was the code fix accurate?
- 12:50
So far, we have two evals, 70 to 75% accuracy for categorization of fixability. As we get more feedback and process more tickets, those become our test cases, and we can move forward improving Scout Agent over time.
- 13:06
So what has been Scout Agent's impact on app erosion?
- 13:11
40% of supports, support team's app fixes are being generated by Scout, so we're doing more of the work on behalf of the support team. This is resulting in, for some of our support team, it's doubling their velocity from one to two tickets w- per week, which already is amazing.
- 13:31
That's going from a support team that wasn't shipping any fixes, well, unofficially they were sometimes, to now shipping one to two per week per person, to now shipping three to four with the help of Scout.
- 13:44
Another, uh, process improvement, Scout puts potentially fixable tickets right there in the triage flow. Takes away a lot of the friction of looking for something to grab from the backlog.
- 13:57
It's not just the support who's benefiting. It's also engineering. Its engineering manager said, uh, "It's a great example of when it works. This tool allows us to stay focused on the more complex stuff."
- 14:10
If you take away anything from this talk, I hope it is that there is a really powerful magic between support and empowering them with codegen and allowing them to ship fixes, 'cause they have three superpowers.
- 14:23
The first, they are the closest to customer pain, which mean that they're closest to the context that really matters for figuring out what's the problem and how to solve it.
- 14:33
They're also troubleshooting in real time. These tickets aren't stale. The context is fresh. The logs aren't missing. You put this ticket into a engineering backlog months later, you might not get access to those logs anymore.
- 14:48
And then three, they're best at validation. You've-- Again, you put the same ticket into an engineering backlog, the solution an engineer might come up with may change the behavior, and that might be good for some customers, but might not necessarily be best for that one customer who wrote in about the problem.
- 15:10
And one other major benefit of this is eng-- or support team members who have been part of this experiment are now engineers.
- 15:22
I wanna say thank you to the amazing team who's helped build this process or build all the tools and the Scout Agent. Andy is actually here in the audience, so shout out to Andy.
- 15:32
If you wanna talk about any of the technical bits, he's here. And I wanna impress upon you two things. We're hiring, but mostly, if you haven't rafted through the Grand Canyon, please consider it.
- 15:43
It's life-changing, and you should go with oars. Thank you very much. [audience applauding] [upbeat music]