AI Engineer World's Fair 2026
We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI
Loading slides…
Read the talk
We Built an AI Support Agent That Resolves 80% of Tickets — AssemblyAI
AssemblyAI’s Joey combines local documentation, retrieval, agent tools and rapid deployment to handle inbound support. Adding voice turns the same agent into both a customer service channel and a working example of the product its customers build.
From a talk by Matt Lawler
At a glance
Ideas worth remembering
Owning the prompt, tools and retrieval system lets a team turn observed support failures into changes it can deploy, rather than requests on a vendor’s roadmap.
Joey combines embedding retrieval for relevant documents up front with agentic filesystem search when an answer needs additional resources.
AssemblyAI reports 80% end-to-end resolution across inbound tickets in Joey’s first week; escalations identify both necessary human work and candidates for further automation.
Voice mode retains Joey’s existing tools while the Voice Agent API connects speech-to-text, an LLM and text-to-speech over one WebSocket.
Building with the same API as customers exposes practical problems in latency and conversational timing, giving FDEs experience that improves their advice.
A thousand signups meet one onboarding engineer
AssemblyAI had around 1,000 API signups every day, and until two days before this talk, Matt Lawler had been its only onboarding engineer. That mismatch is the starting point for Joey, the company’s support agent. Lawler’s job was to help incoming customers build with AssemblyAI and win their business; personally meeting every new developer was already impossible.
The product helps explain the demand and the eventual shape of the solution. AssemblyAI supplies speech-to-text models for recorded audio and real-time applications. Its Voice Agent API extends that stack by connecting speech recognition, an LLM and speech synthesis. Customers need help building products whose behavior unfolds during a conversation, rather than simply submitting a recording and receiving a transcript.
A forward-deployed engineer, or FDE, supplies unusually close support: understanding a customer’s use case, committing code to its repository, meeting repeatedly and becoming the contact for technical and commercial questions. That relationship builds trust, but every call and custom implementation consumes human time. Hiring more FDEs in proportion to incoming customers would keep the same constraint in place.
Lawler’s deliberately provocative instruction is to “automate yourself out of a job.” In this setting, that means putting the team’s knowledge into a system that can serve customers without waiting for the person who holds it. Human-led deals still matter. Automation should free the team to work on them while giving other customers a useful path forward.
Ten percent resolution exposed an iteration problem
The first attempt followed a familiar instinct: buy a support bot, point it at the documentation and deflect simple questions. It resolved about 10% of conversations end to end. Lawler illustrates the remaining workload with a hypothetical 1,000 conversations: 100 answered by the bot still leaves 900 for the team. The signup count establishes the scale of inbound demand; it does not mean every signup created a ticket.
The more consequential problem was control. AssemblyAI could not change the bot’s system prompt, tools or retrieval-augmented generation infrastructure. A failure therefore became a request to the vendor, followed by the answer that a fix was on the roadmap. The team could see what needed to improve without being able to make that improvement itself.
Joey replaced that arrangement as the first contact across the website chat widget, contact-sales requests and support email. Pylon manages conversations and metrics, and gives Joey access to customer context so it can remember earlier interactions. The Claude Agent SDK supplies the agent foundation: a filesystem, tool calls and the ability to write and debug code. Lawler also describes infrastructure management and adding abilities, though the talk does not explain the permissions governing those capabilities.
Local files support both quick retrieval and deeper search
Joey’s knowledge starts with a local Markdown copy of AssemblyAI’s documentation. Updates to the documentation system also reach the files available to the agent. This makes the public documentation website unnecessary for serving an answer from the local copy: if that website goes down, Joey can still consult the documentation it already has.
Two mechanisms help Joey find information:
- Embedding retrieval. Voyage embeddings bring relevant documents into view first, reducing the amount of searching needed for a straightforward answer.
- Agentic file search. When those documents are insufficient, or an answer needs several resources, Joey can search its local filesystem for additional material.
The combination gives the agent a quick starting point and a way to continue investigating. Local documentation alone would provide access; retrieval helps it choose where to look.
What happens when the first retrieved documents do not answer the whole question? The diagram separates the initial retrieval from the conditional search that follows. Both routes consult the same local documentation, so a more involved question can widen the search without depending on the public site.
Deployment speed closes the gap that the vendor bot left open. The team chose Railway to avoid managing an EC2 instance and changing instance types. A bad conversation visible in Slack can lead to a pull request and a new live version in about 30 seconds. Lawler describes catching and fixing a bug while a customer conversation was still underway, with the correction helping the rest of that session. Owning the application makes a support failure something the team can act on immediately.
A quick retrieval can expand into a filesystem search
Read the diagram as text
- Documentation updates. Changes to the documentation system reach Joey’s local files.
- Local Markdown. The documentation remains available in Joey’s filesystem.
- Embedding retrieval. Voyage embeddings provide relevant documents up front.
- Joey prepares an answer. Use the retrieved material, or investigate further when it is insufficient.
- Agentic filesystem search. Find additional resources and combine information across files.
- Documentation updates → Local Markdown: Update local files.
- Local Markdown → Embedding retrieval: Documentation to retrieve.
- Embedding retrieval → Joey prepares an answer: Relevant documents first.
- Joey prepares an answer → Agentic filesystem search: Need more information.
- Local Markdown → Agentic filesystem search: Files available to inspect.
- Agentic filesystem search → Joey prepares an answer: Additional resources.
The unresolved tickets become the next capabilities
In the first week, AssemblyAI’s reported end-to-end resolution rate rose from 10% to 80%, with about $700 per month in model usage and infrastructure costs. Lawler defines the scope explicitly: Joey receives all inbound tickets, resolves 80% without a human and escalates the remaining 20%. These are first-week operational results; the recording does not specify the ticket count or how successful resolution was verified, and the cost figure does not include a stated engineering-labor budget.
Escalation remains part of the service. Customers cannot reach a human directly without going through Joey and requesting escalation. Rate changes, data opt-outs, agreements and questions requiring an FDE or legal involvement are examples of work that can still need a person. An 80% resolution rate therefore describes how much work finishes inside the agent, while the remaining requests still have a human path.
Each recurring escalation also suggests a concrete improvement:
- Agreement access. If a customer needs a BAA and Joey cannot provide it, giving the agent a resource link may remove the need for a person to send the same information.
- Pricing conversations. Joey can already accept a customer’s expected number of hours, quote a rate and negotiate. This extends support into a commercial task that otherwise consumes team time.
The roadmap comes from what customers are actually trying to finish. Some gaps need better information; others need a new capability.
Give the existing agent a voice, then keep its knowledge current
The next change serves two purposes: let customers speak to Joey, and make AssemblyAI build with the same voice infrastructure it sells. Voice mode had been integrated during the week of the talk. The agent keeps the tools it already uses in text conversations; the new interface changes how customers communicate with it.
The Voice Agent API exposes speech-in, speech-out interaction through one WebSocket connection. Behind that connection, speech-to-text feeds an LLM, and text-to-speech produces the spoken response. The API also handles pauses, interruptions and barge-in—when the customer starts speaking while the agent is talking—and supplies voices. Lawler describes the service as real-time and low latency without giving a latency measurement. Telephone access was a planned next step, rather than a capability demonstrated here.
Where does voice processing sit relative to the support tools? The diagram shows the speech pipeline around the LLM, with the existing tools still available to the agent. That relationship explains why adding voice does not require rebuilding Joey’s documentation and support capabilities as a separate bot.
The code walkthrough returns to how those capabilities stay informed. Documentation, changelog and pricing pages are converted to local Markdown as the source pages change. The files retain their URLs, letting Joey point customers to the material behind an answer. Retrieval gives the agent information to use; preserving the source URL gives the customer a way to investigate it.
Customer-facing behavior is also shaped by a large CLAUDE.md containing guardrails and operating advice. Lawler estimates it at roughly 30,000 lines. When Joey gives a wrong answer or creates a bad experience, this instruction file is a principal place the team makes corrections for future conversations. A Dockerfile packages the application for Railway, and repository updates trigger deployment without someone manually monitoring each release.
Voice surrounds the agent’s existing capabilities
Read the diagram as text
- Customer speech. Speak to Joey through voice mode.
- Speech-to-text. Voice Agent API — Recognize incoming speech.
- LLM. Voice Agent API — Produce the response with access to Joey’s existing tools.
- Existing agent tools. The same capabilities available in text mode.
- Text-to-speech. Voice Agent API — Generate the spoken response using the supplied voices.
- Spoken reply. Return speech through the same WebSocket interaction.
- Customer speech → Speech-to-text: Speech in.
- Speech-to-text → LLM: Recognized speech.
- LLM → Existing agent tools: Use existing capabilities.
- LLM → Text-to-speech: Response to speak.
- Text-to-speech → Spoken reply: Speech out.
A medical-scribe question becomes a self-service next step
The live example tests the agreement-access problem introduced earlier. Lawler runs a local simulation so the demonstration will not add work to the actual support queue. After Joey’s greeting, he asks about building an ambient medical scribe: before testing, he needs to sign a BAA. Can Joey help?
Joey first says it will check, then returns a specific path forward: complete AssemblyAI’s standard HIPAA business associate agreement. Its answer includes two consequential conditions—a paid account with a card on file is required, and signing the BAA automatically opts the customer out of model training. Joey also says the relevant documentation is linked in the chat. The demonstration supplies guidance and prerequisites; it does not show an agreement being signed or an account being changed.
The observable change is that the customer leaves with an actionable answer instead of waiting for a support engineer to send a canned response. Lawler recalls that this question required human handling when he joined the support team two years earlier. Making the agreement information available to the agent allows it to explain the next step and the conditions together, through voice, with documentation available in chat.
“Isn’t he so nice?” is Lawler’s reaction after Joey’s friendly farewell. The interaction is also a product demonstration embedded in customer support. Someone asking how to build with voice infrastructure is already experiencing the speech-based interface that infrastructure can produce. Answering the question and showing the product happen in the same conversation.
Shipping the customer’s kind of product changes the advice
The ending pushes beyond support automation. Building Joey with voice made Lawler confront latency, turn-taking, barge-in and interruption handling himself. Those are the same problems customers face. Using the same API to ship an application gives an FDE practical experience to draw on when helping someone else make it work.
The recommendation is to build the kind of product customers are building before waiting for them to report blockers. A working internal application can improve the customer experience directly, as Joey does, while also teaching the team where integration becomes difficult. Lawler closes by inviting reports of strange Joey conversations so the team can keep fixing the newly shipped voice experience. “Go build” is both the customer-support strategy and the way to learn what the product asks of its users.
Resources
Related talks
- Beating RL With Reflection: GEPA and Optimize Anything
Explores how execution traces and detailed feedback can guide changes to prompts and agent programs—a complementary approach to Joey’s manual instruction updates after bad conversations.
Full transcript
Loading transcript…