AI Engineer World's Fair 2025
The Robots Are Coming for Your Job, and That's Okay
Read the talk
Small AI Agents for a Busy Documentation Team
Twilio’s documentation workflow uses focused AI helpers to propose edits, generate metadata and simplify prose, while validation and human review keep publication under team control.
From a talk by Elmer Thomas and Maria Bermudez
Before you start: Familiarity with Markdown, pull requests and automated checks will help you follow the documentation workflow.
A small docs team facing a flood of requests
A small documentation team sits downstream of a much larger organization: Jira tickets keep arriving, first drafts need corrections, and routine grooming competes with work that requires editorial judgment. The goal for Twilio’s team is to use AI to absorb some of that repetitive work without replacing the people responsible for clarity and accuracy.
Elmer Thomas, Principal Developer Educator at Twilio, introduces the workflow alongside Maria Bermudez, a lead developer of the team’s AI Docs Buddy. Thomas describes error-prone first drafts arriving from more than 100 product teams. Style checks, image alt text and SEO metadata consume additional time. Generative AI offers help with those chores, but unconstrained generation introduces another problem: plausible content that may be wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give each helper a narrow job
The team organizes the work into six single-purpose helpers behind a simple Next.js front end. Each has a bounded output that a person can inspect.
| Helper | Assigned job |
|---|---|
| Automated editor | Grammar, formatting and accuracy edits |
| Image alt text generator | Accessibility descriptions |
| Jargon simplifier | Developer language into plain English |
| SEO metadata generator | Titles and descriptions within character limits |
| Docs outline builder | Navigation and structure recommendations; coming soon |
| Slack bot | Help-channel request triage |
The outline builder is planned work at the time of the recording. The broader division of labor is already clear: repetitive jobs go to specialized helpers, leaving humans to concentrate on judgment and clarity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn generated changes into reviewable proposals
Repeatable, high-volume work with little need for creativity is the starting point. A request enters through the Next.js UI and reaches a custom agent, with the model chosen for the particular job. The agent uses the team’s style guide and rubric, retrieved from Airtable so collaborators can maintain the guidance together. Editorial policy therefore has an explicit home outside the individual request.
The proposed changes then pass through a validation layer containing Vale linting and CI/CD tests. Vale checks configurable prose rules; it does not establish whether a product claim is true. A GitHub pull request makes the changes available for scrutiny through code-owner review. GitHub treats requesting that review and requiring approval before merge as separate settings.
Product and engineering reviewers usually participate as well, and a human merges only after accepting the changes. Generating an edit does not authorize publication. Thomas initially describes the layers as crushing hallucinations, then corrects himself: the claim is that they significantly reduce them. The protection comes from combining automated checks with several people’s judgment, rather than asking any single layer to guarantee correctness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect both the edit and its reason
Bermudez begins in the application’s overview and release notes page, which the team uses Copilot to generate. From there, she opens the automated editor. It accepts an MDX file, a Markdown file or a live URL; she chooses the URL option. This lets the editing pass start with an existing published page rather than requiring someone to prepare a separate text input.
The editor uses o1, and the interface displays a loading graphic while it works. Bermudez reports that experimentation identified it as the best model for consistently applying this team’s style guide and rubric. That choice carries a visible latency cost in the interaction.
The result offers two complementary ways to review the proposal. A diff shows what changed. The Changes Made tab provides the original text, revised text and an explanation connecting the change to a style-guide instruction or rubric item. The reviewer can inspect both the textual effect and the rule offered as its justification before deciding whether to accept it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run a separate metadata pass
The editor’s output also exposes a limit: it sometimes catches a missing SEO description and sometimes does not. Bermudez moves the same page into the dedicated SEO metadata generator. Specialization gives the missing field its own explicit task instead of relying on the general editing pass to notice every omission.
She uploads a file and returns to the Conversational Intelligence page that was just edited. The generator produces meta titles by default—users must specify if they do not want them—and a meta description that accounts for character limits. The handoff is performed by Bermudez through the interface, keeping the metadata operation separate and visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate alt text across multiple pages
The alt text generator also supports a live URL, but Bermudez demonstrates selecting multiple pages and generating descriptions for all their images in one action. Batch input reduces the repeated work of submitting each image separately.
The results display image filenames and previews beside generated text. Bermudez calls attention to the quick completion and to a second practical requirement: the output conforms to the documentation platform’s expected format. Producing a description is only part of the task; producing one in the form the publishing workflow accepts makes the result easier to use.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make revisions easy to reuse in a pull request
The jargon simplifier serves both authors drafting a page and reviewers working through a pull request. Bermudez pastes prepared text and selects Simplify. The result includes a diff, like the automated editor, plus a separate revised-text tab. A reviewer can copy that text into a pull request comment or use it to edit the file directly.
For example, a Markdown review suggestion can make a proposed simplification directly usable:
markdown
Replace:
Utilize the configuration interface to modify notification preferences.
With:
Use settings to change your notification preferences.
The useful handoff is a concrete revision that a reviewer can accept, adjust or reject. It remains a proposal until someone applies it.
These helpers are not yet an autonomous chain. Bermudez closes the walkthrough by saying the team wants the agents to communicate with one another, but substantial work remains. The demonstrated workflow consists of focused tools with a person moving between them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Maintain quality through distinct feedback loops
Tools alone do not guarantee quality. Thomas separates the remaining risks by the kind of review they need:
- Hallucinations: Combine Vale linting and CI tests with scrutiny from human stakeholders.
- Bias: Use dataset tests and prompt audits.
- Stakeholder misalignment: Review pull requests weekly, accelerating the cycle to days or hours when needed.
These checks cover different failure modes; passing a style check does not settle a factual question or establish that stakeholders agree with the result.
Slack feedback loops with product managers and engineering provide another route for corrections. The team uses those exchanges to tune prompts continuously. Quality maintenance is therefore an ongoing collaboration around actual outputs, rather than a one-time prompt-writing exercise.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with one throughput bottleneck
The adoption process begins smaller than the full set of helpers:
- Identify one pain point that is limiting throughput.
- Choose a single task that is repeatable and rule-based.
- Loop with its users at least weekly.
Ship, measure and refine that workflow. Thomas expects several such improvements to accumulate into greater team velocity; the recording offers that as the direction of travel, without a measured team-wide gain. The practical starting point is one bounded job whose outputs people can review and whose shortcomings can feed the next iteration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A configurable prose linter for enforcing editorial rules while respecting document markup.
How to assign responsible reviewers and require code-owner approval before merging.
Updates since the talk
Reusable prompts for documentation editing, jargon simplification, alt text and SEO metadata, with examples and integration guidance.
Twilio's announcement explaining how to adopt and contribute to its public documentation prompt templates.
- Making Alt Text FastArticle
Maria Bermudez explains Twilio's batch alt-text workflow, human review and subsequently reported production results.
Read the complete timestamped transcript
- 0:00
Hello, everyone. I'm Elmer Thomas, Principal Developer Educator at Twilio. Welcome to the AI Engineer World's Fair. Our talk today is called The Robots Are Coming For Your Job, And That's Okay.
- 0:11
Don't worry, the title is tongue-in-cheek. I'm here to show you why AI isn't something to fear, it's something we can embrace to work smarter. In the next few minutes, you'll see how our relatively tiny docs team uses AI agents to turbocharge workflows instead of trying to replace ourselves.
- 0:27
We'll look at the pain points, the agents we built, and the guardrails that keep us accurate. I've been a developer and builder for decades. Trends come and go. AI is the latest big one.
- 0:37
Like many of you, I wondered how it would change our day-to-day work. Today, you'll see what we've learned, warts and all. My colleague, Maria Bermudez, a lead developer of our AI docs buddy and resident backlog-busting coffee fanatic, will drive a live agent walkthrough, so you can judge the results yourself.
- 0:55
Here's a game plan. We'll cover the pain, our agent architecture, Maria's demo, and a roadmap that you can steal for your own team. Let's dive in. Picture this, a tiny docs team, a flood of Jira tickets.
- 1:11
Three headaches jump out. Number one, error-prone first drafts from hundred-plus product teams. Two, time sink grooming, like style checks, alt text, SEO. Three, hallucination risk if we let GenAI run wild.
- 1:25
We needed leverage, not burnout. Instead of one megabot, we built six single-purpose agents behind a simple Next.js front end. The automated edum- editor fixes grammar, formatting, and accuracy.
- 1:42
Image alt tech gen- text generator gives you instant accessibility wins. The jargon simplifier turns dev speak into plain English. SEO metadata provides title and description and character, keeping it character count safe.
- 1:58
The docs outline builder recommends nav and structure. That's coming soon. Super excited about that. Our Slack back bot helps triage help channel requests. Each agent tackles a repetitive, well-scoped job, so humans can focus on judgment and clarity.
- 2:16
Rule of thumb, pick tasks that are repeatable, high volume, and low creativity. That's the sweet spot for an AI helper. Here's the flow behind every request. We have a Next.js UI that feeds into a custom GPT-4-01 agent.
- 2:39
We use the appropriate model for the particular job. The custom GPT is, it has our style guide, and it has our rubric that is baked in. It's actually retrieved from an Airtable so that folks can easily collaborate on it.
- 2:56
We have a validation layer that includes Vale linting and CI/CD tests. The GitHub PR adds code owner review and makes it much easier for us to scrutinize the changes that the agents are suggesting.
- 3:12
A human hits the merge only when it's right, and usually, we have built into their product and engineering reviews as well. So we have several human eyes before we actually consider hitting that merge button.
- 3:29
This layered approach, it crushes hallucinations without showing us down. Maybe not crushes them, but it reduces them significantly. All right. It's time to prove this isn't vaporware. I'm handing the screen to Maria Bermudez, the brains behind much of the AI docs buddy code.
- 3:49
Maria, take it away.
- 3:51
Thank you, Elmer. So real quick, I wanna show you how we use Copilot to auto-generate our overview and release notes page. And from here, you can get into any of the agents that you want to work with.
- 4:02
So I'm gonna start with the automated editor. One cool thing about this is you not only can load an MDX file, uh, a Markdown file, but you can also plug in a live URL, and I'm gonna do that option right now.
- 4:16
This does take a little while, so we added this graphic here because we're using the o1 model. We've done a lot of experimentation and figured out this is the best model to consistently apply our style guide and rubric.
- 4:31
And here you go. And then on the...
- 4:35
It shows you the diff of what it changed, and, uh, in the changes made tab, you actually have a list of, um, explanation of the original text, the revised text, and what it changed and what style guidance and rubric item the changes pertain to.
- 4:53
Um, it's not perfect, as, as you can see. Like, um, sometimes it catches that there's a missing SEO description, but other times it doesn't. So luckily, we have an SEO metadata generator, and I'm going to plug in this page into that one.
- 5:12
So what I'm actually gonna do is I'm going to upload the file
- 5:18
and then go back to Conversational Intelligence. So now I'm going to generate SEO metadata for that same page that we just edited.
- 5:41
And there you go. So it generates the meta titles. You have to specify if you don't want that, and then it gives you the meta description, um, accounting for character limitations.
- 5:52
And now I'm gonna show you the alt text generator.
- 5:56
It has the same option that we have, where you can plug in a live URL, but I'm gonna show you, um, picking multiple pages and
- 6:06
Having our agent generate alt text for all of those images. So I'm gonna click on Generate alt text now.
- 6:16
And you see how quickly that got done, and it also conforms to the format that we require from our docs platform.
- 6:25
And lastly, I'm gonna show you the jargon simplifier. So you can take whatever, like, pages that you're reviewing. This is really helpful, not only for writing, but when we're re-reviewing a pull request, and it gives us, like, a good way to provide quick edits to those images.
- 6:43
So I'm gonna plug in a text that I've prepared, and I'm going to click on Simplify.
- 6:56
And there you go. It also gives you a diff like our n- our editor. And on the next tab, you have the revised text that you can quickly copy and apply as a pull request, uh, comment or just edit the file directly.
- 7:12
And that's it. We're working on making our agents talk to each other eventually, but we, we have a lot of work to do on that front.
- 7:20
Thank you, Maria. That was wonderful. Now, tools alone don't guarantee quality. Our guardrails tackle three big risks. Hallucinations, we use tools like Vale Lint and CI Test to help mitigate that in combination with the various eyes of, of human stakeholders.
- 7:41
Bias, dataset tests and prompt audits are the way to tackle those issues. Stakeholder misalignment, so weekly PR reviews. Sometimes these are compressed into days and sometimes hours.
- 7:57
And we also have Slack feedback loops, generally with the product managers and engineering team. These feedback cycles, they let us tune the prompts continuously instead of hoping the model magically stays perfect.
- 8:12
S- Now, please steal this three-step playbook. Identify one pain that's killing throughput. Two, pick a single task that's repeatable and rule-based. And then three, loop with your users weekly at least.
- 8:27
Ship, measure, and refine. Stack a few of these wins and watch your team's velocity jump. Thanks again to the AI Engineer World's Fair team for hosting us, and thanks to you for spending time with us today.
- 8:42
Big shout out to Carly Vanderwert, our manager at Twilio, who keeps encouraging us to push the boundaries with AI, along with the rest of our leadership. We appreciate your attention.
- 8:53
Now go build something amazing with AI.