The Agent Development Life Cycle

How do you improve an AI agent when its job extends from answering questions to managing subscriptions, recommending products, and helping customers buy? At Sierra, these are parts of the same customer experience. Chat support is only one interface: speaking in February 2025, Zack Reneau-Wedeen expected phone conversations to account for most Sierra interactions by the end of that year. That was a forecast, but it made the development problem immediate: improvements would have to carry across both tasks and channels.

The difficulty becomes clearer in an earlier, smaller application. After joking that other conference speakers’ histories of AI barely reached beyond the current decade, Reneau-Wedeen rewinds to his work at Google in 2016. Computer vision still struggled with distinctions that people found funny precisely because they were visually plausible: Chihuahuas versus blueberry muffins, dogs versus bagels, dogs versus mops, and dogs versus fried chicken. This was the work behind an early version of Google Lens, before its public introduction in 2017.

On his walk to the office in New York City, he passed restaurants, bookstores, dogs, and flower shops. Plant identification was one of the few early computer-vision capabilities with an obvious consumer use, so the commute became an informal test route. Could Lens identify a plant’s color as well as its species? What kind of fern or palm was this? He went into flower shops and tried the system on the plants there; the photographs in the presentation came from those walks.

Some answers were accurate. Others were not exactly wrong, but missed what he wanted to know. The experience felt like a slot machine: a run of plausible successes did not guarantee the next result. The example is not a measured failure rate. It captures the engineering problem that persists in agents today: variation in inputs and nondeterminism in outputs make a few successful trials a weak basis for confidence.

By the time of the talk, Lens supported visual search and shopping through images, YouTube, and the camera. It could translate the controls on a washing machine in Tokyo, help with math homework, and still identify flowers. Reneau-Wedeen attributes that breadth to roughly a decade of consistent iteration. The goal is to get better without also getting worse. The software development lifecycle supplies a familiar structure: implement, test, maintain, analyze, design, and repeat.

A second rewind, to 2012, establishes how much the underlying technology had changed. Google Brain’s work on learning to recognize cats from YouTube imagery had made headlines. The matching research, Building high-level features using large scale unsupervised learning, describes a network with one billion connections; Reneau-Wedeen calls it about a billion parameters. He contrasts that with a roughly trillion-parameter frontier, a thousandfold comparison offered as broad historical framing rather than a specification for any named model. At the time, even the limits of what computers might eventually accomplish looked different.

Around that period, Marc Andreessen’s 2011 essay Why Software Is Eating the World gave software businesses a memorable thesis. Reneau-Wedeen turns the history into a guessing game: which Stanford-connected startup is he about to discuss? Not Snapchat, and not DoorDash. It is Chubbies, whose proposition was that very short men’s shorts could also take over. The joke establishes something that matters to the agent that follows: Chubbies has a deliberately playful brand, and its customer experience needs to preserve that personality.

In 2024, Chubbies’ SVP of Commercial, Kit Garton, visited Sierra. The company already treated customer experience as a place to express its brand, and an agent offered another touchpoint alongside its website, social presence, and mobile app. The resulting agent is called Duncan Smothers in the talk; Sierra’s written customer story spells the name Duncan Smuthers. Duncan needed to be capable while retaining Chubbies’ sense of humor.

The customer conversations, shown with Garton’s permission, progress from guidance to operations:

• Sizing and fit: Duncan asks clarifying questions, including the customer’s waist size, and uses the answers to recommend products.
• Inventory: Duncan checks what is in stock and helps the customer choose another item.
• Package tracking: Duncan explains that an order has multiple tracking numbers, resolving confusion about where its pieces are.
• Refunds: Duncan issues a refund, moving beyond an explanation to an action that changes the customer’s situation.

The displayed thumbs-up responses show that those particular customers appreciated the exchanges. The refund demonstrates the consequential distinction: an autonomous agent can carry out work, not merely describe what someone else should do.

Reneau-Wedeen reports that Chubbies could help more customers, help them faster, and achieve higher satisfaction, without supplying numerical measurements. Sierra’s operating principle is that every agent is a product. A developer platform is only part of the requirement; customer-experience teams also need tools to operate and improve what has shipped. Sierra pairs dedicated agent engineers with agent product managers, working closely with the customer’s team day to day. The relationship resembles a team maintaining a website or mobile app, with continuing responsibility for behavior after launch.

A practical example of that development culture comes from a Deepgram workshop at the previous AI Engineer World’s Fair. Reneau-Wedeen had been building voice agents for about three months and wanted to understand how others approached multimodal models, latency, tone, and phrasing. He sat next to Sean, and their conversation quickly turned to making voice testing less cumbersome.

Their shortcut combined the Mac say command with Loopback:

1. Generate spoken input with say.
2. Route that synthesized audio into the browser with Loopback.
3. Exercise the voice interaction without repeatedly speaking aloud into a headset.

The technique made the developer loop easier to repeat in an office. Sean exchanged contact information with Reneau-Wedeen, and a few months later they were colleagues at Sierra. The anecdote ends as a recruiting invitation, but its engineering detail is concrete: reducing the friction of trying a conversation makes iteration easier.

Duncan does not have to be perfect at launch, but the team needs a process that improves him continuously. Sierra borrows from the software development lifecycle while adapting it to language models. Reneau-Wedeen’s metaphor is a foundation of Jell-O: familiar engineering practices remain useful, but the thing underneath them does not behave like ordinary application logic.

His comparison describes the engineering tradeoff rather than an absolute rule about every program or model:

The design opportunity is to combine the two: use a model’s flexibility and reasoning, then invoke traditional software where its predictable behavior is useful.

The Agent Development Life Cycle formalizes that combination as a process for building and improving agents. Its resemblance to an ordinary software lifecycle is intentional. The value lies in the operational details refined with customers in production: how teams discover a problem, express the intended behavior, test a change, and safely return it to use.

The lifecycle diagram connects Development, Testing, Release, Quality Assurance, and Alignment. Quality assurance starts with visibility into what the agent actually did. Sierra’s Experience Manager lets customer teams inspect individual conversations and view high-level reports of agent performance in real time.

Suppose Duncan gives incorrect inventory information. The failure could originate in orchestration: perhaps he queried one warehouse but omitted another, or one of the required API calls timed out. The visible answer is the symptom; the sequence of calls matters to understanding the defect.

The repair follows a concrete path:

1. A customer-team member reports the conversation’s problem.
2. That report becomes an issue.
3. The issue leads to a test for the behavior that needs to work.
4. Once the test passes, the team can make a new release.

The test retains the lesson from the production encounter. Reneau-Wedeen describes an agent’s suite growing from a handful of tests at launch to hundreds and then thousands as it improves. He does not identify a particular customer’s measured test-count history.

Production review can also reveal an opportunity to do more for a customer. Chubbies gives its agents a budget for customer delight. If shorts are unavailable online but present at a retail location, Duncan could potentially arrange a DoorDash delivery from that store. This is a proposed enhancement, not a completed delivery shown in the talk. It illustrates why the lifecycle includes alignment on what the agent ought to accomplish, rather than only correction of erroneous answers.

Reneau-Wedeen describes Sierra’s earlier process as manual while the team learned what worked at each stage. By the time of the talk, improvements in AI were allowing the company to apply AI within the lifecycle itself and accelerate that work. The agent is therefore both the product being improved and a beneficiary of better AI-assisted development practices.

Scale raises the value of both iteration speed and change management. Reneau-Wedeen contrasts Duncan’s hundreds of thousands of requests with other customers’ tens of millions, without specifying a time window. At that scale, a repeatable process for introducing changes becomes more valuable.

Changes also arrive from outside the product. Model upgrades, reasoning models, and multimodality create new capabilities and new reasons to revisit existing behavior. Reasoning models can assist development, testing, and quality assurance—not just the customer-facing conversation. The lifecycle gives the team a way to absorb those advances while continuing to improve the deployed agent.

Voice makes that need for adaptation concrete. After beginning work on voice agents roughly a year before the talk, Sierra made Voice generally available in October 2024. Reneau-Wedeen names SiriusXM as a large customer, describing its US customer population as tens of millions; that population is not a count of agent requests. He says Sierra enables SiriusXM to answer customers’ calls immediately, but provides no measured pickup-time benchmark or deployment-coverage figure.

The architectural analogy is responsive web design. Earlier mobile websites often lived separately at addresses such as m.website.com, creating two experiences to maintain. Responsive design replaced that split with a shared foundation that adapts to the device. Sierra applies the same idea to agents: the platform and agent code are shared, while the experience responds to the channel and modality.

Shared code still leaves room for channel-specific behavior. A phone interaction may need different phrasing, and requests can run in parallel to reduce latency. The Agent OS slide presents chat, email, messaging, and phone as channels for that shared agent. Reneau-Wedeen describes the basic adaptation as working out of the box; the accompanying customization options show where teams can tune the experience.

The closing design question returns to the model’s limitations. Language models can seem familiar in their unpredictability, slowness, and difficulty with arithmetic. Reneau-Wedeen uses that resemblance as an invitation to empathy: put yourself in the agent’s position and consider what information it has available when asked to respond. This is a design exercise, not an assertion that models and people are equivalent.

For voice, imagine receiving only a transcription of your conversation partner’s speech, arriving a few hundred milliseconds late, and then having to answer on the spot. The delay is part of the thought experiment, not a Sierra latency measurement. It exposes a question that merely connecting components does not resolve: does the resulting system give the agent enough conversational information to produce a good experience?

That question motivates Reneau-Wedeen’s interest in voice-to-voice models and richer inputs. His closing direction is to let language models work with more of the inputs and experiences available to a human conversation partner. He associates that richness with more robust interactions, without presenting a detailed architecture or comparative benchmark. The next improvement is therefore not only a faster response or a better instruction; it may be a better account of what the agent needs to perceive before it speaks.