# OpenAI for VPs of AI

Prashant Mital, OpenAI · Toki Sherbakov, OpenAI

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/joHR2pmxDQE-openai-vps-ai) · [Video](https://www.youtube.com/watch?v=joHR2pmxDQE)

## OpenAI for VPs of AI

Start with business priorities, build measurable use cases, and let observed failures guide the move from a single agent to specialized networks with explicit safety gates.

Basic familiarity with language-model APIs, prompts, and retrieval will help with the agent architecture examples.

### Getting models into everyday work

[0:17](https://www.youtube.com/watch?v=joHR2pmxDQE&t=17s)

How does an enterprise turn access to powerful models into useful work for employees, internal operations, and customers? The problem extends beyond choosing a model: capabilities must become products, those products must fit real workflows, and deployment must generate feedback that improves the next iteration.

OpenAI’s division of responsibilities follows that path. Research builds foundational models. Applied engineering turns them into products such as ChatGPT and the API. Go-to-market teams help enterprises put those products into employees’ hands, automate operations, and integrate AI into customer-facing products. Feedback from those deployments then returns to both product teams and model research, closing the loop.

![Three panels labeled Research, Apply, and Deploy, with deployment covering workforce, operations, and product, and dotted arrows looping back.](https://ai.engineer/api/talk-essay-media/d7504fe1dda3a57f2f5373fc84a63f97ba80418eda241804243d39a669bf2fb9)

OpenAI’s research, application, and deployment loop.

### Three surfaces for enterprise adoption

[1:49](https://www.youtube.com/watch?v=joHR2pmxDQE&t=109s)

Enterprise adoption commonly begins by giving employees AI they can use every day. That builds AI literacy through practice. Internal automation follows: copilots and workflows begin doing more of the organization’s work. Customer-facing integration brings those capabilities into the product itself. These are common phases, not a mandatory sequence.

| Adoption surface | Purpose | Typical product surface |
| --- | --- | --- |
| Workforce | Daily assistance and AI literacy | ChatGPT |
| Internal operations | Automation and customized copilots | ChatGPT; API for greater customization |
| Customer products | AI inside the end-user experience | Primarily the API |

The transition to the API is driven by the integration and customization a use case requires. Internal operations can begin within ChatGPT; more complex workflows need closer control over how AI fits into the surrounding application.

### Business strategy before organizational scale

[3:02](https://www.youtube.com/watch?v=joHR2pmxDQE&t=182s)

**Start with the business strategy, then identify where AI can advance it.** A top-down mandate is useful when it establishes priorities and constraints, rather than treating AI adoption as an independent objective. From those priorities, select one or two substantial, high-impact use cases and scope them tightly enough to deliver.

Once those initial use cases are underway, build the organization’s capacity to repeat the work. That can mean enabling teams, establishing a center of excellence, or creating a shared technical platform that other divisions can build on. The progression is from strategic direction to bounded delivery, then to broader capability—not an organization-wide platform built before the first use case is understood.

### Define success before development

[4:26](https://www.youtube.com/watch?v=joHR2pmxDQE&t=266s)

Toki Sherbakov describes an illustrative three-month use-case journey, not a delivery guarantee. Its first steps establish what the system must do and how the team will know whether it works:

1. **Ideate and scope.** Choose the workflow and bound the problem.
2. **Review the architecture.** Determine how AI fits into the existing stack.
3. **Define success.** Establish metrics and KPIs before implementation becomes the main activity.

These decisions give development a target beyond producing a convincing demonstration.

Most of the work then goes into iteration: changing prompts, incorporating retrieval-augmented generation, and improving whichever parts of the system constrain the use case. OpenAI’s collaboration with engineering teams can include workshops, office hours, paired programming, and webinars to accelerate that development.

Testing returns to the evaluations defined upfront. A/B tests and beta rollouts establish how the application behaves in practice before a broader launch. Production rollout adds scale optimization testing, followed by continuing maintenance. Deployment is therefore another stage of the use-case lifecycle, with its own checks, rather than the point at which development simply stops.

### Dedicated teams, bounded roadmap visibility

[5:54](https://www.youtube.com/watch?v=joHR2pmxDQE&t=354s)

This partnership requires dedicated teams on both sides. Early access to models and features can help an enterprise prepare for capabilities that are approaching release, but that access does not imply a long, dependable forecast. Sherbakov describes his visibility at the time as roughly six months, or two quarters; he explicitly cannot supply an eighteen-month roadmap.

Research engineering and product experts can support the work, while joint roadmap sessions connect the customer’s plans to the capabilities OpenAI can see coming. The useful planning horizon is the one the participants can actually discuss, with ongoing alignment as those plans develop.

### Morgan Stanley: improving the whole retrieval system

[6:50](https://www.youtube.com/watch?v=joHR2pmxDQE&t=410s)

[Morgan Stanley’s internal knowledge assistant](https://openai.com/index/morgan-stanley/) gives wealth managers a way to ask questions across a large collection of information, including research reports and stock-related data. The operational requirement is accurate information that advisers can use when responding to clients. A fluent answer alone does not satisfy that requirement.

Sherbakov reports an initial accuracy of 45%. The team introduced [HyDE retrieval](https://arxiv.org/abs/2212.10496), fine-tuned embeddings, and different chunking strategies. These changes address the retrieval side of the application: how a question connects to relevant material and how that material is represented and divided for use.

| Reported milestone | Accuracy | Development context |
| --- | --- | --- |
| After further retrieval work | 85% | Added reranking and classification |
| Customer target | 90% | Desired accuracy |
| Final reported result | 98% | Further prompt engineering and query expansion |

These are Sherbakov’s deployment-specific accuracy figures. The talk does not define the evaluation set or scoring protocol, and the sequence does not isolate the contribution of any individual technique. It illustrates cumulative improvement against a customer metric, rather than a general benchmark for retrieval systems.

The next step in this progression is moving from systems that answer questions toward systems that carry out work. The speakers frame 2025 as an expected year of agents, then turn from enterprise delivery to the architecture lessons emerging from those deployments.

### The model controls an execution loop

[8:21](https://www.youtube.com/watch?v=joHR2pmxDQE&t=501s)

Prashant Mital draws on customer work and OpenAI’s own agentic products, including [deep research](https://openai.com/index/introducing-deep-research/) and the original [Operator research preview](https://openai.com/index/introducing-operator/). His framing is a transition from an assistant to a coworker: an application that can take successive steps toward an objective.

An **agent** combines a model, instructions, access to tools, and an execution loop. Instructions usually arrive through a prompt. Tools let the application retrieve information or interact with external systems. The defining feature of this account is that the model controls when the loop terminates.

Within that loop, the application proceeds through a recurring cycle:

1. Receive and interpret natural-language instructions.
2. Determine whether to call tools.
3. Execute the requested tools.
4. Incorporate their return values into a response.
5. Decide whether the objective has been met and execution should end.

The tool results become information for the next model decision. That makes the trajectory of the interaction—not just one generated answer—part of the system’s behavior.

### Understand the primitives before abstracting

[9:56](https://www.youtube.com/watch?v=joHR2pmxDQE&t=596s)

Suppose an agent must orchestrate models, retrieve data, reason over it, and generate an output. Building directly with primitives means making API calls and recording outputs and failures yourself. A framework can handle many of those details and make a proof of concept much faster to assemble. The risk is accepting its design choices before understanding the system’s behavior or constraints.

**Build enough of the system directly to understand what needs improvement.** Observe how the task decomposes, where failures arise, and which constraints prevent better results. Abstractions become useful when they remove repetition you have actually encountered—for example, repeatedly implementing an embedding strategy or model graders.

The decision is therefore not primarily which framework to choose. It is whether an abstraction improves a system whose data, failure points, and constraints the team understands. Start simply, optimize the parts that need it, and introduce abstraction when it makes the application better.

### Learn from one focused agent in production

[11:43](https://www.youtube.com/watch?v=joHR2pmxDQE&t=703s)

The same reasoning applies to agent count. Agents calling other agents, coordinating tasks, and reasoning across long trajectories create many interacting unknowns when introduced too early. Mital recommends beginning with one agent, purpose-built for one task, and deploying it to a limited set of production users.

That release exposes bottlenecks that an architectural diagram cannot resolve:

- **Conversation reliability:** hallucinations may emerge over an extended interaction.
- **Latency and adoption:** a system may be capable but too slow for users to adopt.
- **Retrieval accuracy:** poor evidence retrieval may constrain the quality of answers.

Once the team understands how the application underperforms and what users value, it can improve the relevant parts incrementally. Complexity grows as new intents, failure cases, and constraints become visible.

### Handoffs preserve context while changing specialists

[12:57](https://www.youtube.com/watch?v=joHR2pmxDQE&t=777s)

When the task calls for multiple specializations, a network of agents can divide a larger workflow into related subflows. Each agent handles a particular part of the request. A **handoff** transfers control of the active conversation from one agent to another—like a phone transfer, but with the ability to carry the conversation history forward so the user does not have to start again.

The sample customer-service architecture assigns models according to the work they perform:

| Role | Model | Responsibility |
| --- | --- | --- |
| Triage | **GPT-4o mini** | Route the incoming request |
| Dispute agent | **GPT-4o** | Manage the conversation with the user |
| Eligibility specialist | **o3-mini** | Check refund eligibility |

This is a sample architecture, not a reported production result. It illustrates how a small model can handle triage, a conversational model can manage a dispute, and a reasoning model can handle an accuracy-sensitive decision within the same service flow.

The mechanism is straightforward: retain the conversation history and context while replacing the active model, prompt, and tool definitions. The new agent receives the prior interaction but operates with the instructions and capabilities appropriate to its specialization. Full-history preservation is the pattern illustrated here, rather than a requirement that every handoff implementation must follow.

### Run guardrails separately, gate consequential actions

[14:53](https://www.youtube.com/watch?v=joHR2pmxDQE&t=893s)

**Guardrails** enforce safety, security, and reliability around the application. Keeping the main instructions simple and focused on the target task makes the system easier to compose and its accuracy and performance easier to improve predictably. Safety checks need not all become extra instructions inside that main prompt; they can run separately, in parallel.

Faster, cheaper models such as GPT-4o mini make separate checks more accessible. But parallel checking must be paired with an explicit action gate: a high-stakes tool call, such as issuing a refund, or a response disclosing personal-account information should wait until the relevant checks have completed. **A proposed action is not yet an authorized action.**

For a refund workflow, the separation can be expressed with a small Python orchestration function. Here, `propose_refund` only prepares a proposal; the side effect belongs exclusively to `issue_refund`. Every check returns a Boolean approval.

```python
import asyncio


async def guarded_refund(
    request,
    *,
    propose_refund,
    input_check,
    output_checks,
    issue_refund,
):
    proposal, input_allowed = await asyncio.gather(
        propose_refund(request),
        input_check(request),
    )
    if not input_allowed:
        return {"status": "blocked"}

    approvals = await asyncio.gather(
        *(check(proposal) for check in output_checks)
    )
    if not approvals or not all(approvals):
        return {"status": "blocked"}

    return await issue_refund(proposal)
```

The input check overlaps with proposal generation. Output checks require the proposal and then run concurrently with each other. Rejection or a raised exception prevents execution from reaching `issue_refund`; speculative work before that point must remain free of consequential side effects.

The illustrated guardrail arrangement uses an input check for prompt injection and multiple checks on the agent’s response. The slide shows a user and an appointment agent, with hallucination and misinformation detection and a content-policy filter around the output. The architecture separates the task from its checks while retaining the requirement to defer consequential tool calls and responses until those checks return.

![Diagram linking a user to an appointment agent, with prompt injection detection, hallucination and misinformation detection, and a content policy filter. A note describes deferring tool calls and responses until guardrails return.](https://ai.engineer/api/talk-essay-media/7c7a69da7759eb082bcfec5869c307f8912e902eeb00a1fd012d2707be6c231f)

Keep prompts simple, with lightweight guardrails running in parallel.

These design choices fit together: understand the primitives before adding abstractions, learn from one focused agent, introduce specialists as the set of intents expands, and keep task prompts focused while guardrails handle edge cases. The system becomes more capable through observed needs, with explicit boundaries around what it may actually do.

## Resources

- [Morgan Stanley’s evaluation-driven AI deployment](https://openai.com/index/morgan-stanley/) — A customer case study covering expert evaluations, retrieval improvements, regression testing and advisor adoption.
- [HyDE: Precise Zero-Shot Dense Retrieval without Relevance Labels](https://arxiv.org/abs/2212.10496) — The original HyDE paper explains retrieval using embeddings of generated hypothetical documents.
- [HyDE implementation and notebooks](https://github.com/texttron/hyde) — Original research code with a demonstration notebook and TREC DL19 experiment. Uses historical GPT-3 and Contriever tooling.
- [Implementing handoffs with the OpenAI Agents SDK](https://openai.github.io/openai-agents-python/handoffs/) — Current guidance for transferring conversation control and configuring the history passed to another agent.
- [Guardrail execution modes and boundaries](https://openai.github.io/openai-agents-python/guardrails/) — Current SDK guidance on parallel and blocking input checks, output validation and function-tool guardrails.
- [Introducing deep research](https://openai.com/index/introducing-deep-research/) — OpenAI’s launch account of multistep research in ChatGPT, with later product updates marked separately.
- [Introducing Operator](https://openai.com/index/introducing-operator/) — The original browser-agent research preview, including human oversight and safety controls, with a later integration notice.

## Slides

### How OpenAI operates: Research, Apply, Deploy, Repeat

Research advances models, the applied team builds and tests products, and deployment reaches workforces, operations, and products. Feedback loops return to research and application.

[0:46](https://www.youtube.com/watch?v=joHR2pmxDQE&t=46s) [A diagram shows research, application and deployment, with feedback loops returning to earlier stages.](https://assets.ai.engineer/sites/aiecode2025/sha256/b5d4be3d01a09c9eeb79752f462f985cc43e2a32d34756522a18c84a6a9763fd.webp)

How OpenAI operates:
Research, Apply, Deploy, Repeat

Research
Advance foundations
New model capabilities

Apply
Build products and apps
Test and iterate

Deploy
Workforce
Operations
Product

OpenAI Confidential & Proprietary
2

### Scope: Identify pathway to implement AI into your organization

OpenAI maps ChatGPT to workforce adoption, ChatGPT and the API to internal operations, and the API to user-facing products. The speaker says these pathways need not happen in sequence.

[1:54](https://www.youtube.com/watch?v=joHR2pmxDQE&t=114s) [Three AI implementation pathways: an AI-enabled workforce, automated operations, and AI in products.](https://assets.ai.engineer/sites/aiecode2025/sha256/06f81684ac0124f7e6f7decaf9e960082ae0bd6e09d4193371bb0b96e887605b.webp)

Scope: Identify pathway to implement AI into your organization

01 Building an AI-enabled workforce
02 AI automated operations
03 Infusing AI into products

OpenAI Confidential & Proprietary
3

[2:27](https://www.youtube.com/watch?v=joHR2pmxDQE&t=147s) [Three AI implementation pathways: ChatGPT for workforce enablement, ChatGPT and the OpenAI API for operations, and the OpenAI API for products.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f0e4ce7ea9ef6cfc6bf028cdcd5bfcc74e03877e71a28bd29e27801f4f43bc5.webp)

Scope: Identify pathway to implement AI into your organization

01 Building an AI-enabled workforce
ChatGPT

02 AI automated operations
ChatGPT
OpenAI API

03 Infusing AI into products
OpenAI API

Confidential & Proprietary
3

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[2:48](https://www.youtube.com/watch?v=joHR2pmxDQE&t=168s) [Three pathways: an AI-enabled workforce using ChatGPT, automated operations using ChatGPT and the OpenAI API, and AI in products using the API.](https://assets.ai.engineer/sites/aiecode2025/sha256/686b814166d2261cbd6d6ad8448040f87694cfa6356716752dac474baa88ea7c.webp)

Scope: Identify pathway to implement AI into your organization

01 Building an AI-enabled workforce
ChatGPT

02 AI automated operations
ChatGPT
OpenAI API

03 Infusing AI into products
OpenAI API

Confidential & Proprietary
3

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Joint Strategy

The diagram spans broad strategic guidance, divisional capability, and deeper use-case delivery. The speaker recommends aligning AI with business strategy and starting with one or two high-impact use cases.

[3:05](https://www.youtube.com/watch?v=joHR2pmxDQE&t=185s) [Diagram relating strategic guidance, divisional capability, and tactical use cases across breadth and depth.](https://assets.ai.engineer/sites/aiecode2025/sha256/7895e21bcb8a5655ba6eaddd02507d921a27df93a2b2ed4cb66dba178f85edc1.webp)

Joint Strategy: Determine where to intervene first, demonstrate value, and build internal capability

Strategic guidance
• Setting an AI strategy
• Sharing roadmap
• Prioritizing top initiatives

Divisional capability
• Building internal capability
• Delivering workshops
• Establishing GenAI Center of Excellence (CoE)

Tactical delivery
• Solution design
• Evaluation and optimization
• Hands-on support

Breadth
Strategic
Divisional | Divisional
Use case | Use case | Use case | Use case | Use case | Use case
Depth

OpenAI Confidential & Proprietary
4

### Execution collaboration

The illustrative three-month plan spends six weeks in development after scoping, then moves through testing and production. Dedicated customer and OpenAI teams collaborate throughout, with maintenance ongoing.

[4:27](https://www.youtube.com/watch?v=joHR2pmxDQE&t=267s) [Three-month collaboration plan covering scoping, development, testing, production, and ongoing maintenance.](https://assets.ai.engineer/sites/aiecode2025/sha256/096bf16102638574f7738759750ac3b784c172986145f1bdf39029cc43ec1215.webp)

Execution: Define engagement model to accelerate time to value with close collaboration and defined use cases

Illustration of a 3-month execution collaboration

Phase
Ideation & Scoping — 3 weeks
Development — 6 weeks
Testing & Evaluation — 2 weeks
Production — 1 week
Maintenance — Ongoing

Focus
Ideation & Scoping:
- Use case ideation
- Prioritization/scoping
- Architecture review
- Defining success and KPIs

Development:
- Workshops / onsites
- Office hours
- Paired programming
- Webinars

Testing & Evaluation:
- A/B testing
- Eval / guardrail deployment
- User interviews
- Beta rollout

Production:
- Security and privacy
- Change management
- Scale & optimization
- Launch planning

Maintenance:
- Accuracy improvements
- Cost / latency optimization
- KPIs / metrics realization
- Ongoing support

Team
Dedicated teams from your org and OpenAI with close regular collaboration
Project Manager, Product/Engineering Lead, Data Science
Account Director, Solutions Architect, Solutions Engineer
OpenAI

Enablers
Early access to new models/features
Collaboration with internal OAI experts
Joint roadmap sessions

OpenAI Confidential & Proprietary
5

### Defining agents

An agent combines a model, instructions and tools in a runtime. The model controls when the execution loop ends; tools can retrieve information and interact with external systems.

[8:58](https://www.youtube.com/watch?v=joHR2pmxDQE&t=538s) [An agent is defined as a model with instructions and tools, encapsulated in a runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/14f6482ed6595ef4d88c868fafb9f093811477dd8bb392ac9019830bab3c61bc.webp)

Defining agents

An agent is an AI application consisting of a model equipped with instructions that guide its behavior, access to tools that extend its capabilities, encapsulated in a runtime with a dynamic lifecycle.

Agent =
+ Model
+ Instructions
+ Tools
+ Runtime

OpenAI
Confidential and proprietary.
8

### Defining agents

An agent combines a model, instructions, tools and a runtime. In each cycle, it can call tools and use their results to answer; it can also end the loop when it determines its objective is met.

[9:23](https://www.youtube.com/watch?v=joHR2pmxDQE&t=563s) [An agent is defined as a model with instructions and tools, encapsulated in a runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/1dd7daf5719bdcdd24f6b127da7b6d0f9a0451be04751060ebac141034081cfa.webp)

Defining agents

An agent is an AI application consisting of a model equipped with instructions that guide its behavior, access to tools that extend its capabilities, encapsulated in a runtime with a dynamic lifecycle.

Agent =
+ Model
+ Instructions
+ Tools
+ Runtime

[illegible]prietary.
8

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

[9:29](https://www.youtube.com/watch?v=joHR2pmxDQE&t=569s) [An agent consists of a model, instructions, tools, and a runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/e77974f3f196110e6ff67576c5fce01995f9476606a8b083922936b56e963445.webp)

Defining agents

An agent is an AI application consisting of a model equipped with instructions that guide its behavior, access to tools that extend its capabilities, encapsulated in a runtime with a dynamic lifecycle.

Agent =
+ Model
+ Instructions
+ Tools
+ Runtime

[illegible]prietary.
8

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Lessons learnt

Frameworks can make an agent proof of concept quick to build, but may obscure how the system behaves and which primitives it uses.

[9:47](https://www.youtube.com/watch?v=joHR2pmxDQE&t=587s) [Lesson 01: Abstraction is a tool, not a crutch.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3241d189c4796b64cf6a87b465403c13a383adb41b41eef92391c1959d96713.webp)

Lessons learnt

01  Abstraction is a tool, not a crutch

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Start simple

A single appointments agent handles user messages with four tools. The speaker recommends deploying a purpose-built agent to a limited set of users, then improving it based on observed bottlenecks.

[12:11](https://www.youtube.com/watch?v=joHR2pmxDQE&t=731s) [A user communicates with an appointments agent connected to scheduling, availability, reminder, and adjustment tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/1c65b8bf0a6ae74a214554768a77fc08a26c82e81722dccfcd28785c7f7bd1c8.webp)

Start simple
Begin with a single agent

You are an Appointments Agent. Your job is to assist users with questions related to scheduling, changing, or canceling appointments. You have access to several to tools ...

User
Message dictionary

Appt. Agent
Routine
Tools

Schedule Appointment Tool
Availability Inquiry Tool
Appointment Reminder Tool
Adjust Appointment Tool

[12:50](https://www.youtube.com/watch?v=joHR2pmxDQE&t=770s) [A user sends a message to an appointments agent connected to scheduling, availability, reminder, and adjustment tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/22626db544ac2052ccad5458c1fd004d7cb7f0ddca25c592306b59b8115bb128.webp)

Start simple
Begin with a single agent

You are an Appointments Agent. Your job is to assist users with questions related to scheduling, changing, or canceling appointments. You have access to several to tools ...

User
Message dictionary

Appt. Agent
Routine
Tools

Schedule Appointment Tool
Availability Inquiry Tool
Appointment Reminder Tool
Adjust Appointment Tool

Interface text:
AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Lessons learnt

After starting simple, the speaker introduces networks of agents and handoffs for more complex tasks.

[12:55](https://www.youtube.com/watch?v=joHR2pmxDQE&t=775s) [The third lesson, use a network of agents for complex tasks, is emphasized; the first two lessons are faded.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ab66520adba7177b88cacb9f13770c63bd58470c176cb43453076ee70d950bb.webp)

Lessons learnt

01 Abstraction is a tool, not a crutch
02 Start simple
03 Use a network of agents for complex tasks

AIE
AI LEADERSHIP
HTTPS://AI.ENGINEER

### Network of agents and handoffs

A triage agent connects the user to a network of specialized agents. Handoffs transfer control between agents while preserving conversation history, so the next agent can continue with context.

[13:24](https://www.youtube.com/watch?v=joHR2pmxDQE&t=804s) [A user sends a message to a triage agent, which routes work through a network of agents. Text defines agent networks and handoffs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ffd3ff3582056b1677130471a57f9f0ef7e94c1e910bb7035ddea8a2ff88f4c.webp)

Network of agents + handoffs

A network of agents is a collaborative system where multiple agents work in concert to resolve complex requests or perform a series of interrelated tasks.

User
Message dictionary

Triage agent
Routine
Tools

Agent network:
Routine | Tools
Routine | Tools
Routine | Tools
Routine | Tools
Routine | Tools
Routine | Tools
Routine | Tools

A handoff is when one agent passes control of an active conversation to another agent.

### Customer service bot

GPT-4o mini triages requests, GPT-4o handles disputes, and o3-mini checks refund eligibility. Feedback branches to order details and knowledge-base checks; dispute paths can escalate to a human.

[14:07](https://www.youtube.com/watch?v=joHR2pmxDQE&t=847s) [A triage agent routes requests to dispute and feedback agents. Specialized agents check refund policies, information, order details and knowledge base entries; dispute paths can escalate to a human.](https://assets.ai.engineer/sites/aiecode2025/sha256/defce089825680bc647d971e6628b7188cf839ddfe4d7f77ff378de8baeaf681.webp)

Example: Customer service bot

Message dictionary

gpt-4o-mini
Triage agent
Routine

gpt-4o
Dispute agent
Routine
Tools

gpt-4o
Feedback agent
Routine
Tools

o3-mini
Refund policy check agent
Routine
Tools

gpt-4o
Information agent
Routine
Tools

gpt-4o-mini
Order details agent
Routine
Tools

o1-mini
KB checks agent
Routine
Tools

Escalate to human
Escalation Agent
Tools

OpenAI Confidential & Proprietary

## Full transcript

[0:00](https://www.youtube.com/watch?v=joHR2pmxDQE&t=0s) [on-hold music] Hello.

[0:17](https://www.youtube.com/watch?v=joHR2pmxDQE&t=17s) Uh, thanks for having us here. And today, we're gonna talk a bit about building and scaling use cases with OpenAI and what this means in terms of enterprises working with OpenAI to bring use cases to production, and a little sneak peek into agents and how we've seen some of our experience building these use cases and now agentic

[0:34](https://www.youtube.com/watch?v=joHR2pmxDQE&t=34s) workflows, uh, in the field. So, uh, on our side, um, just a quick introduction into OpenAI. I'm sure folks have probably heard of OpenAI, but just in terms of how we operate, we have two core engineering teams.

[0:48](https://www.youtube.com/watch?v=joHR2pmxDQE&t=48s) We have our research team, which is twelve hundred researchers that are inventing these models, right? We-- They, they build and deploy these foundational models. These kinda come down from the heavens.

[0:59](https://www.youtube.com/watch?v=joHR2pmxDQE&t=59s) Our applied team, our second engineering team, take this and build it into product. So this is where you see things like ChatGPT, you see things like the API, where our GPT models are available, and that's where we actually deploy this.

[1:10](https://www.youtube.com/watch?v=joHR2pmxDQE&t=70s) Finally, in the go-to-market sense, where we take these products and put it in end users' hands. That's kinda where our team comes into play with go-to-market, where we actually help get this in the hands of your workforce, in the hands of your product, and really start to automate these internal operations.

[1:26](https://www.youtube.com/watch?v=joHR2pmxDQE&t=86s) And once we finally deploy these, there's kinda this iterative loop where we take feedback from the field to improve our product directly, and then also improve our core models through this research flywheel.

[1:38](https://www.youtube.com/watch?v=joHR2pmxDQE&t=98s) So that's kinda the last step of getting it back to research. So this is typically how OpenAI operates. Um, in terms of the enterprise, we see the kind of AI customer journey happen typically in three phases.

[1:49](https://www.youtube.com/watch?v=joHR2pmxDQE&t=109s) It doesn't have to happen in sequence in this way, but this is what we usually see, is first and foremost, building an AI-enabled workforce. This is getting AI in the hands of your employees to become AI literate, to use AI every day in their day-to-day work.

[2:04](https://www.youtube.com/watch?v=joHR2pmxDQE&t=124s) That's the first and foremost, that first step typically that we see. Then from there, you typically graduate to- towards automating your AI operations. This is actually more internal use cases to build an automation or maybe some co-pilot type use cases into the workforce.

[2:20](https://www.youtube.com/watch?v=joHR2pmxDQE&t=140s) Then the last step here is actually infusing AI into end product. This is end user facing. So when it comes to OpenAI's product, specifically enabling your workforce, typically starts with something like ChatGPT.

[2:33](https://www.youtube.com/watch?v=joHR2pmxDQE&t=153s) So this is our, you know, first party product to put in the hands of users to use day in and day out. Then when you talk about automating operations internally, you can do this partially with ChatGPT.

[2:42](https://www.youtube.com/watch?v=joHR2pmxDQE&t=162s) For the more complex use cases or more, more customization is needed, that's where something like the API comes in. And then finally, infusing this into your end user products is where it's primarily API use cases.

[2:53](https://www.youtube.com/watch?v=joHR2pmxDQE&t=173s) But just to give a flavor of how these products come into play when actually as-- executing this across your AI customer journey.

[3:02](https://www.youtube.com/watch?v=joHR2pmxDQE&t=182s) So in terms of how we see enterprises actually craft this strategy and practice, it kinda happens in a few different ways. I'd say first and foremost, you determine a little bit from a top-down level of what should the strategy be.

[3:15](https://www.youtube.com/watch?v=joHR2pmxDQE&t=195s) And one core thing that we acknowledge here, it's not actually what's your AI strategy, it's actually what's your broader business strategy. And what OpenAI does is help figure out where does the technology meet that broader business strategy, first and foremost.

[3:28](https://www.youtube.com/watch?v=joHR2pmxDQE&t=208s) So that kinda top-down strat-- uh, strategic guidance is really important to start with. And then once you start with that top-down guidance, you then move to use cases. Like let's identify one or two meaty use cases that are high impact to start with and scope those out to really just deliver on, um, kind of that scoped scale.

[3:47](https://www.youtube.com/watch?v=joHR2pmxDQE&t=227s) So once you have the strategy, you execute upon those two use ca-- one to two use cases, and then you think about how to build divisional capability across your enterprise.

[3:55](https://www.youtube.com/watch?v=joHR2pmxDQE&t=235s) This is where you start to enable the team and to infuse AI throughout the organization. And this happens in many ways. This comes through enablement, this comes through building centers of excellence, this comes with building maybe a centralized technological platform that other people in the enterprise can build on.

[4:10](https://www.youtube.com/watch?v=joHR2pmxDQE&t=250s) And I feel like that's typically the journey we see is, again, set the strategy, pick those one to two use cases, and then build that capability across your organization through enablement.

[4:20](https://www.youtube.com/watch?v=joHR2pmxDQE&t=260s) So that's usually the, the type of journey we see. And just to illustrate this a little bit with an example

[4:26](https://www.youtube.com/watch?v=joHR2pmxDQE&t=266s) is this is how we've seen the use case journey play out. So, um, this is ill- illustrative of a three-month type of example of a use case. But when you've identified that one to two use cases that you want to tackle, first and foremost, you have to ideate upon that, do some initial scoping, do some architecture review

[4:45](https://www.youtube.com/watch?v=joHR2pmxDQE&t=285s) to understand how does AI gonna fit into your current stack, and then really clearly define what the success metrics and KPIs are. Once you have that established, the bulk of the time is really spent in development.

[4:55](https://www.youtube.com/watch?v=joHR2pmxDQE&t=295s) This is where you iterate. This is where you are iterating on prompting strategies, incorporating RAG, what- whatever it may be, to constantly improve the, uh, use case that you're tackling.

[5:06](https://www.youtube.com/watch?v=joHR2pmxDQE&t=306s) When it comes to engaging with OpenAI, this is where our team, like Prashant and myself, really interact closely with your engineering team through things like workshops, things like office hours, paired programming sessions, webinars, whatever it kinda takes to accelerate the use case forward.

[5:21](https://www.youtube.com/watch?v=joHR2pmxDQE&t=321s) Once we do that development phase, we kinda move to testing and evaluation, which is with the evals we've typically defined upfront, we're able to actually now do some A/B testing, do some beta rollout to understand how this actually works in practice.

[5:36](https://www.youtube.com/watch?v=joHR2pmxDQE&t=336s) And then finally, we go to production. This is where you just do some launch rollout, do some, uh, scale optimization testing to make sure it's gonna work once you deploy to m- many end users.

[5:44](https://www.youtube.com/watch?v=joHR2pmxDQE&t=344s) And then we have kind of constant maintenance that's, that's ongoing. So that's like the typical phase you'll see. And again, the bulk of the time, especially in partnership with OpenAI, will be around development.

[5:54](https://www.youtube.com/watch?v=joHR2pmxDQE&t=354s) Um, in this, we bring a dedicated team. We ask you bring us also a dedicated team to make this work in practice. And the things that we deploy also to enable you are things like early access to new models and features.

[6:07](https://www.youtube.com/watch?v=joHR2pmxDQE&t=367s) That's one of the key things of working closely with, uh, OpenAI is that we can see a little bit into the future, not much. Like, our roadmap, I, I don't see beyond much, maybe, like, six months.

[6:17](https://www.youtube.com/watch?v=joHR2pmxDQE&t=377s) People ask, "What's your eighteen-month roadmap?" I cannot tell you. I can tell you basically what's gonna happen in the next two quarters. But that, like, purview into the future is really important to bring forward to these use cases and en- enable customers to build and innovate for what's coming next.

[6:32](https://www.youtube.com/watch?v=joHR2pmxDQE&t=392s) So that's a really critical part of our partnership. Um, also, we bring in, you know, internal experts from our research engineering team, our product team, to help kind of accelerate you on this path.

[6:41](https://www.youtube.com/watch?v=joHR2pmxDQE&t=401s) And then lastly, just kind of do joint roadmap sessions to make sure that we're on track for what your future roadmap is as well. So that's hopefully an illustration of how we partner together.

[6:50](https://www.youtube.com/watch?v=joHR2pmxDQE&t=410s) And then one concrete example on this is something we did with Morgan Stanley. So Morgan Stanley, here based in New York, uh, was building a internal knowledge assistant. So what this was, was giving their wealth managers the ability to que-- uh, ask questions of their large corporus-- corpus of da-- uh, knowledge, which was research reports, like, live

[7:10](https://www.youtube.com/watch?v=joHR2pmxDQE&t=430s) views on stock ticker data, whatever it may be, and they wanted to get highly accurate information back to be able to respond to their end clients, right? And accuracy was pretty bad to start, right?

[7:21](https://www.youtube.com/watch?v=joHR2pmxDQE&t=441s) It was forty-five percent typically what they saw. So interacting with us, we introduced new methods throughout the use case development, things like HyDE retrieval. We did some fine-tune embeddings, different chunking strategies, which improved performance.

[7:32](https://www.youtube.com/watch?v=joHR2pmxDQE&t=452s) And then once we kept introducing more and more methods, we saw accuracy go up. We introduced things like reranking and classification stuff that got it to eighty-five percent, and ultimately, their goal was ninety percent.

[7:41](https://www.youtube.com/watch?v=joHR2pmxDQE&t=461s) We got to ninety-eight percent accuracy through other things like prompt engineering and query expansion. So more of just an example of how we introduced methods throughout this use case journey to, uh, improve their core metric for Morgan Stanley in this case.

[7:54](https://www.youtube.com/watch?v=joHR2pmxDQE&t=474s) Um, so this is hopefully one illustration of how OpenAI's partnered with customers. And one common use case we're seeing more and more of is now building in this agent space.

[8:03](https://www.youtube.com/watch?v=joHR2pmxDQE&t=483s) You maybe hear that twenty twenty-five is the year of agents. Agentic workflows has been a buzzword for a long time. I think we're seeing that actually come to reality this year.

[8:11](https://www.youtube.com/watch?v=joHR2pmxDQE&t=491s) And, um, I think with that, we've seen, uh, some ba-- we have some battle scars and some best practices of what we've seen in the field, and I'll hand it off to Prashant to talk about what we've seen on the agent side.

[8:21](https://www.youtube.com/watch?v=joHR2pmxDQE&t=501s) Thanks, Toki. So at OpenAI, we are lucky to work alongside customers who are building state-of-the-art agents and working alongside team members who are building our own agentic products, like deep research and Operator.

[8:36](https://www.youtube.com/watch?v=joHR2pmxDQE&t=516s) Like Toki said, we expect twenty twenty-five to be the year of agents, the year Gen, GenAI truly graduates from being an assistant to being a coworker. And to help usher in this era, we've been hard at work identifying the patterns and anti-patterns prevalent in agent development.

[8:54](https://www.youtube.com/watch?v=joHR2pmxDQE&t=534s) I'm excited to share four of those with you today.

[8:58](https://www.youtube.com/watch?v=joHR2pmxDQE&t=538s) Before we can go further, I'd like to quickly define, uh, what we mean by the term agent. So we think of an agent as an AI application that consists of a model that has some instructions, usually in the form of a prompt, access to some tools for retrieving information and interacting with e- external systems, all encapsulated in

[9:18](https://www.youtube.com/watch?v=joHR2pmxDQE&t=558s) a, in an execution loop whose termination is controlled by the model itself.

[9:22](https://www.youtube.com/watch?v=joHR2pmxDQE&t=562s) So one way of thinking about this is that in each execution cycle, the agent can be thought of as an entity that's receiving instructions in natural language, determining whether or not to issue any tool calls, running those tools, synthesizing a response with the tool return values, and then providing an answer to the user.

[9:41](https://www.youtube.com/watch?v=joHR2pmxDQE&t=581s) Additionally, the user may determine-- Sorry. The agent may determine that it's met its objective, and therefore terminate the execution loop. So with that definition, let's move on to some of the lessons that we've learned, uh, building these agents in the field.

[9:56](https://www.youtube.com/watch?v=joHR2pmxDQE&t=596s) So for the first insight, imagine you're designing an AI agent. You need to orchestrate multiple models. You need to retrieve data, reason over it, and generate an output. You have two choices.

[10:07](https://www.youtube.com/watch?v=joHR2pmxDQE&t=607s) You can start with primitives, making raw API calls, logging results yourselves, um, and logging outputs and failures,

[10:15](https://www.youtube.com/watch?v=joHR2pmxDQE&t=615s) or you can start with a framework. You can pick an abstraction, you can wire it up, and you can let it handle a lot of the details. And I have to say, starting with a framework is pretty enticing.

[10:25](https://www.youtube.com/watch?v=joHR2pmxDQE&t=625s) It's how I got started building agents. It's really easy to get started, have a proof of concept, concept stood up in no time. But the problem is that if you start with a framework, you often don't actually know how your system behaves or what primitives it uses.

[10:41](https://www.youtube.com/watch?v=joHR2pmxDQE&t=641s) You've deferred design, design decisions before you've understood your constraints, and if you don't know your constraints, you can't optimize your solution. So we believe a better approach is to first build with primitives, understand how your task decomposes, where the failures happen, and what actually needs improvement.

[11:02](https://www.youtube.com/watch?v=joHR2pmxDQE&t=662s) Then introduce abstraction. When you find that you're reinventing the wheel, for example, by re-implementing an embedding strategy or re-implementing model graders, that may be a good time to bring in some abstractions.

[11:17](https://www.youtube.com/watch?v=joHR2pmxDQE&t=677s) Many teams today are spending a lot of time picking the right framework. Um, we actually believe that developing agents in a scalable way isn't so much about choosing the right abstraction.

[11:27](https://www.youtube.com/watch?v=joHR2pmxDQE&t=687s) It's really about understanding your data, understanding your failure points and your constraints. So in summary, the first lesson is to start simple, optimize where needed, and abstract only when it makes your system better.

[11:43](https://www.youtube.com/watch?v=joHR2pmxDQE&t=703s) Which leads us straight to our second insight, starting simple. So too often, teams are jumping straight into designing multi-agent systems, agents calling agents, coordinating tasks, dynamically reasoning over long trajectories.

[11:57](https://www.youtube.com/watch?v=joHR2pmxDQE&t=717s) It all sounds really powerful, but when it's done too soon, it creates a lot of unknowns, and it doesn't give you all that much insight.

[12:07](https://www.youtube.com/watch?v=joHR2pmxDQE&t=727s) We like a different approach. We generally recommend starting with a single agent that's purpose-built for a single task. Put that into production with a limited set of users and observe how it performs.

[12:20](https://www.youtube.com/watch?v=joHR2pmxDQE&t=740s) Doing this allows you to identify the real bottlenecks: hallucinations over conversation trajectories, low adoption due to high latency, or maybe inaccuracy due to poor retrieval performance.

[12:34](https://www.youtube.com/watch?v=joHR2pmxDQE&t=754s) Then, knowing how the system underperforms and knowing what's important to your users, we can work to incrementally improve it.

[12:43](https://www.youtube.com/watch?v=joHR2pmxDQE&t=763s) In a nutshell, we should think of complexity as something which increases as we discover more intents, failure cases, and constraints. Because the goal isn't really to build a complicated system, it's just to build a system that works.

[12:57](https://www.youtube.com/watch?v=joHR2pmxDQE&t=777s) So starting simple sounds great, uh, but we all know that complexity is where true value is realized. So how should we handle more complex tasks?

[13:08](https://www.youtube.com/watch?v=joHR2pmxDQE&t=788s) This is where a network of agents and the concept of handoffs comes in.

[13:14](https://www.youtube.com/watch?v=joHR2pmxDQE&t=794s) So you can think of handoffs... Uh, sorry. Let's start with a network of agents. So a, a network of agents is a collaborative system where multiple agents work in concert to resolve complex requests or perform a series of interrelated tasks.

[13:28](https://www.youtube.com/watch?v=joHR2pmxDQE&t=808s) You can think of this as a series of specialized agents handling sub-flows within a large agentic workflow.

[13:37](https://www.youtube.com/watch?v=joHR2pmxDQE&t=817s) On the topic of handoffs, you can think of these

[13:40](https://www.youtube.com/watch?v=joHR2pmxDQE&t=820s) as the process by which one agent transfers control of a active conversation to another agent. It's pretty similar to how you get transferred to someone else on a phone call, except in this case, you can preserve your entire conversation history, and the new agent just magically knows everything you've talked about already.

[13:59](https://www.youtube.com/watch?v=joHR2pmxDQE&t=839s) So let's see an example of this. In this sample architecture, we are showing how a fully automated customer service flow may be implemented with a network of agents and handoffs.

[14:10](https://www.youtube.com/watch?v=joHR2pmxDQE&t=850s) This approach is allowing us to build, bring the right tools to the right job. So for example, on the left-hand side, we are using a GPT-4o mini call to, uh, perform triage on the incoming request.

[14:23](https://www.youtube.com/watch?v=joHR2pmxDQE&t=863s) We're then using GPT-4o on the dispute agent to actually manage the conversation with the user, and finally, we are using a o3-mini reasoning model to perform accuracy-sensitive tasks, like checking whether the customer is eligible for a refund.

[14:38](https://www.youtube.com/watch?v=joHR2pmxDQE&t=878s) It turns out that handoffs work really well, and keeping the entire conversation history and context while swapping out the model, the prompt, the tool definitions, provides sufficient flexibility to solve a wide range of scenarios.

[14:53](https://www.youtube.com/watch?v=joHR2pmxDQE&t=893s) So our final lesson pertains to guardrails. And just to level set, guardrails is a catch-all term today for any mechanism that enforces safety, security, and reliability within your application, and it's generally used to prevent misuse and ensure that your system maintains integrity.

[15:11](https://www.youtube.com/watch?v=joHR2pmxDQE&t=911s) So keeping the model instructions simple and focused on the target task ensures maximum interoperability of your system and also ensures that we are able to hill climb on perf-- on accuracy and performance most, uh, predictably.

[15:26](https://www.youtube.com/watch?v=joHR2pmxDQE&t=926s) Guardrails should not necessarily be made part of your main prompts, but should instead be run in parallel,

[15:33](https://www.youtube.com/watch?v=joHR2pmxDQE&t=933s) and the proliferation of faster and cheaper models like GPT-4o mini is making that m- making this more accessible than ever. Tool calls and user responses that are high-stakes, for example, issuing a refund or showing a user what information, uh, some information from their personal account, these can be deferred until all of the guardrails have returned.

[15:56](https://www.youtube.com/watch?v=joHR2pmxDQE&t=956s) In this example, we see that we're running a single input guardrail to prevent prompt injection, and then a couple of output guardrails, uh, on the use- on the agent's response.

[16:07](https://www.youtube.com/watch?v=joHR2pmxDQE&t=967s) So to recap, we have four lessons from our time building agents: use abstractions minimally, start with a single agent, graduate to a network of agents when you have more intents, and finally, keep your prompts simple and focused on the happy path, and use guardrails to handle edge cases. [outro music]

[16:28](https://www.youtube.com/watch?v=joHR2pmxDQE&t=988s) Thank you. [audience applauding] [outro music]
