OpenAI for VPs of AI

Talk transcript

1. 0:00

   [on-hold music] Hello.

2. 0:17

   Uh, thanks for having us here. And today, we're gonna talk a bit about building and scaling use cases with OpenAI and what this means in terms of enterprises working with OpenAI to bring use cases to production, and a little sneak peek into agents and how we've seen some of our experience building these use cases and now agentic

3. 0:34

   workflows, uh, in the field. So, uh, on our side, um, just a quick introduction into OpenAI. I'm sure folks have probably heard of OpenAI, but just in terms of how we operate, we have two core engineering teams.

4. 0:48

   We have our research team, which is twelve hundred researchers that are inventing these models, right? We-- They, they build and deploy these foundational models. These kinda come down from the heavens.

5. 0:59

   Our apply team, our second engineering team, take this and build it into product. So this is where you see things like ChatGPT, you see things like the API, where our GPT models are available, and that's where we actually deploy this.

6. 1:10

   Finally, in the go-to-market sense, where we take these products and put it in end users' hands. That's kinda where our team comes into play with go-to-market, where we actually help get this in the hands of your workforce, in the hands of your product, and really start to automate these internal operations.

7. 1:26

   And once we finally deploy these, there's kinda this iterative loop where we take feedback from the field to improve our product directly, and then also improve our core models through this research flywheel.

8. 1:38

   So that's kinda the last step of getting it back to research. So this is typically how OpenAI operates. Um, in terms of the enterprise, we see the kind of AI customer journey happen typically in three phases.

9. 1:49

   It doesn't have to happen in sequence in this way, but this is what we usually see, is first and foremost, building an AI-enabled workforce. This is getting AI in the hands of your employees to become AI literate, to use AI every day in their day-to-day work.

10. 2:04

    That's the first and foremost, that first step typically that we see. Then from there, you typically graduate to- towards automating your AI operations. This is actually more internal use cases to build an automation or maybe some co-pilot type use cases into the workforce.

11. 2:20

    Then the last step here is actually infusing AI into end product. This is end user facing. So when it comes to OpenAI's product, specifically enabling your workforce, typically starts with something like ChatGPT.

12. 2:33

    So this is our, you know, first party product to put in the hands of users to use day in and day out. Then when you talk about automating operations internally, you can do this partially with ChatGPT.

13. 2:42

    For the more complex use cases or more, more customization is needed, that's where something like the API comes in. And then finally, infusing this into your end user products is where it's primarily API use cases.

14. 2:53

    But just to give a flavor of how these products come into play when actually as-- executing this across your AI customer journey.

15. 3:02

    So in terms of how we see enterprises actually craft this strategy and practice, it kinda happens in a few different ways. I'd say first and foremost, you determine a little bit from a top-down level of what should the strategy be.

16. 3:15

    And one core thing that we acknowledge here, it's not actually what's your AI strategy, it's actually what's your broader business strategy. And what OpenAI does is help figure out where does the technology meet that broader business strategy, first and foremost.

17. 3:28

    So that kinda top-down strat-- uh, strategic guidance is really important to start with. And then once you start with that top-down guidance, you then move to use cases. Like let's identify one or two meaty use cases that are high impact to start with and scope those out to really just deliver on, um, kind of that scoped scale.

18. 3:47

    So once you have the strategy, you execute upon those two use ca-- one to two use cases, and then you think about how to build divisional capability across your enterprise.

19. 3:55

    This is where you start to enable the team and to fuse AI throughout the organization. And this happens in many ways. This comes through enablement, this comes through building centers of excellence, this comes with building maybe a centralized technological platform that other people in the enterprise can build on.

20. 4:10

    And I feel like that's typically the journey we see is, again, set the strategy, pick those one to two use cases, and then build that capability across your organization through enablement.

21. 4:20

    So that's usually the, the type of journey we see. And just to illustrate this a little bit with an example

22. 4:26

    is this is how we've seen the use case journey play out. So, um, this is ill- illustrative of a three-month type of example of a use case. But when you've identified that one to two use cases that you want to tackle, first and foremost, you have to ideate upon that, do some initial scoping, do some architecture review

23. 4:45

    to understand how does AI gonna fit into your current stack, and then really clearly define what the success metrics and KPIs are. Once you have that established, the bulk of the time is really spent in development.

24. 4:55

    This is where you iterate. This is where you are iterating on prompting strategies, incorporating RAG, what- whatever it may be, to constantly improve the, uh, use case that you're tackling.

25. 5:06

    When it comes to engaging with OpenAI, this is where our team, like Prashant and myself, really interact closely with your engineering team through things like workshops, things like office hours, paired programming sessions, webinars, whatever it kinda takes to accelerate the use case forward.

26. 5:21

    Once we do that development phase, we kinda move to testing and evaluation, which is with the evals we've typically defined upfront, we're able to actually now do some A/B testing, do some beta rollout to understand how this actually works in practice.

27. 5:36

    And then finally, we go to production. This is where you just do some launch rollout, do some, uh, scale optimization testing to make sure it's gonna work once you deploy to m- many end users.

28. 5:44

    And then we have kind of constant maintenance that's, that's ongoing. So that's like the typical phase you'll see. And again, the bulk of the time, especially in partnership with OpenAI, will be around development.

29. 5:54

    Um, in this, we bring a dedicated team. We ask you bring us also a dedicated team to make this work in practice. And the things that we deploy also to enable you are things like early access to new models and features.

30. 6:07

    That's one of the key things of working closely with, uh, OpenAI is that we can see a little bit into the future, not much. Like, our roadmap, I, I don't see beyond much, maybe, like, six months.

31. 6:17

    People ask, "What's your eighteen-month roadmap?" I cannot tell you. I can tell you basically what's gonna happen in the next two quarters. But that, like, purview into the future is really important to bring forward to these use cases and en- enable customers to build and innovate for what's coming next.

32. 6:32

    So that's a really critical part of our partnership. Um, also, we bring in, you know, internal experts from our research engineering team, our product team, to help kind of accelerate you on this path.

33. 6:41

    And then lastly, just kind of do joint roadmap sessions to make sure that we're on track for what your future roadmap is as well. So that's hopefully an illustration of how we partner together.

34. 6:50

    And then one concrete example on this is something we did with Morgan Stanley. So Morgan Stanley, here based in New York, uh, was building a internal knowledge assistant. So what this was, was giving their wealth managers the ability to que-- uh, ask questions of their large corporus-- corpus of da-- uh, knowledge, which was research reports, like, live

35. 7:10

    views on stock ticker data, whatever it may be, and they wanted to get highly accurate information back to be able to respond to their end clients, right? And accuracy was pretty bad to start, right?

36. 7:21

    It was forty-five percent typically what they saw. So interacting with us, we introduced new methods throughout the use case development, things like hide retrieval. We did some fine-tune embeddings, different chunking strategies, which improved performance.

37. 7:32

    And then once we kept introducing more and more methods, we saw accuracy go up. We introduced things like reranking and classification stuff that got it to eighty-five percent, and ultimately, their goal was ninety percent.

38. 7:41

    We got to ninety-eight percent accuracy through other things like prompt engineering and query expansion. So more of just an example of how we introduced methods throughout this use case journey to, uh, improve their core metric for Morgan Stanley in this case.

39. 7:54

    Um, so this is hopefully one illustration of how OpenAI's partnered with customers. And one common use case we're seeing more and more of is now building in this agent space.

40. 8:03

    You maybe hear that twenty twenty-five is the year of agents. Agentic workflows has been a buzzword for a long time. I think we're seeing that actually come to reality this year.

41. 8:11

    And, um, I think with that, we've seen, uh, some ba-- we have some battle scars and some best practices of what we've seen in the field, and I'll hand it off to Prashant to talk about what we've seen on the agent side.

42. 8:21

    Thanks, Toki. So at OpenAI, we are lucky to work alongside customers who are building state-of-the-art agents and working alongside team members who are building our own agentic products, like deep research and operator.

43. 8:36

    Like Toki said, we expect twenty twenty-five to be the year of agents, the year Gen, GenAI truly graduates from being an assistant to being a coworker. And to help usher in this era, we've been hard at work identifying the patterns and anti-patterns prevalent in agent development.

44. 8:54

    I'm excited to share four of those with you today.

45. 8:58

    Before we can go further, I'd like to quickly define, uh, what we mean by the term agent. So we think of an agent as an AI application that consists of a model that has some instructions, usually in the form of a prompt, access to some tools for retrieving information and interacting with e- external systems, all encapsulated in

46. 9:18

    a, in an execution loop whose termination is controlled by the model itself.

47. 9:22

    So one way of thinking about this is that in each execution cycle, the agent can be thought of as an entity that's receiving instructions in natural language, determining whether or not to issue any tool calls, running those tools, synthesizing a response with the tool return values, and then providing an answer to the user.

48. 9:41

    Additionally, the user may determine-- Sorry. The agent may determine that it's met its objective, and therefore terminate the execution loop. So with that definition, let's move on to some of the lessons that we've learned, uh, building these agents in the field.

49. 9:56

    So for the first insight, imagine you're designing an AI agent. You need to orchestrate multiple models. You need to retrieve data, reason over it, and generate an output. You have two choices.

50. 10:07

    You can start with primitives, making raw API calls, logging results yourselves, um, and logging outputs and failures,

51. 10:15

    or you can start with a framework. You can pick an abstraction, you can wire it up, and you can let it handle a lot of the details. And I have to say, starting with a framework is pretty enticing.

52. 10:25

    It's how I got started building agents. It's really easy to get started, have a proof of concept, concept stood up in no time. But the problem is that if you start with a framework, you often don't actually know how your system behaves or what primitives it uses.

53. 10:41

    You've deferred design, design decisions before you've understood your constraints, and if you don't know your constraints, you can't optimize your solution. So we believe a better approach is to first build with primitives, understand how your task decomposes, where the failures happen, and what actually needs improvement.

54. 11:02

    Then introduce abstraction. When you find that you're reinventing the wheel, for example, by re-implementing an embedding strategy or re-implementing model graders, that may be a good time to bring in some abstractions.

55. 11:17

    Many teams today are spending a lot of time picking the right framework. Um, we actually believe that developing agents in a scalable way isn't so much about choosing the right abstraction.

56. 11:27

    It's really about understanding your data, understanding your failure points and your constraints. So in summary, the first lesson is to start simple, optimize where needed, and abstract only when it makes your system better.

57. 11:43

    Which leads us straight to our second insight, starting simple. So too often, teams are jumping straight into designing multi-agent systems, agents calling agents, coordinating tasks, dynamically reasoning over long trajectories.

58. 11:57

    It all sounds really powerful, but when it's done too soon, it creates a lot of unknowns, and it doesn't give you all that much insight.

59. 12:07

    We like a different approach. We generally recommend starting with a single agent that's purpose-built for a single task. Put that into production with a limited set of users and observe how it performs.

60. 12:20

    Doing this allows you to identify the real bottlenecks: hallucinations over conversation trajectories, low adoption due to high latency, or maybe inaccuracy due to poor retrieval performance.

61. 12:34

    Then, knowing how the system underperforms and knowing what's important to your users, we can work to incrementally improve it.

62. 12:43

    In a nutshell, we should think of complexity as something which increases as we discover more intents, failure cases, and constraints. Because the goal isn't really to build a complicated system, it's just to build a system that works.

63. 12:57

    So starting simple sounds great, uh, but we all know that complexity is where true value is realized. So how should we handle more complex tasks?

64. 13:08

    This is where a network of agents and the concept of handoffs comes in.

65. 13:14

    So you can think of handoffs... Uh, sorry. Let's start with a network of agents. So a, a network of agents is a collaborative system where multiple agents work in concert to resolve complex requests or perform a series of interrelated tasks.

66. 13:28

    You can think of this as a series of specialized agents handling sub-flows within a large agentic workflow.

67. 13:37

    On the topic of handoffs, you can think of these

68. 13:40

    as the process by which one agent transfers control of a active conversation to another agent. It's pretty similar to how you get transferred to someone else on a phone call, except in this case, you can preserve your entire conversation history, and the new agent just magically knows everything you've talked about already.

69. 13:59

    So let's see an example of this. In this sample architecture, we are showing how a fully automated customer service flow may be implemented with a network of agents and handoffs.

70. 14:10

    This approach is allowing us to build, bring the right tools to the right job. So for example, on the left-hand side, we are using a GPT-4.0 Mini call to, uh, perform triage on the incoming request.

71. 14:23

    We're then using GPT-4.0 on the dispute agent to actually manage the conversation with the user, and finally, we are using a o3-mini reasoning model to perform accuracy-sensitive tasks, like checking whether the customer is eligible for a refund.

72. 14:38

    It turns out that handoffs work really well, and keeping the entire conversation history and context while swapping out the model, the prompt, the tool definitions, provides sufficient flexibility to solve a wide range of scenarios.

73. 14:53

    So our final lesson pertains to guardrails. And just to level set, guardrails is a catch-all term today for any mechanism that enforces safety, security, and reliability within your application, and it's generally used to prevent misuse and ensure that your system maintains integrity.

74. 15:11

    So keeping the model instructions simple and focused on the target task ensures maximum interoperability of your system and also ensures that we are able to hill climb on perf-- on accuracy and performance most, uh, predictably.

75. 15:26

    Guardrails should not necessarily be made part of your main prompts, but should instead be run in parallel,

76. 15:33

    and the proliferation of faster and cheaper models like GPT-4.0 Mini is making that m- making this more accessible than ever. Tool calls and user responses that are high-stakes, for example, issuing a refund or showing a user what information, uh, some information from their personal account, these can be deferred until all of the guardrails have returned.

77. 15:56

    In this example, we see that we're running a single input guardrail to prevent prompt injection, and then a couple of output guardrails, uh, on the use- on the agent's response.

78. 16:07

    So to recap, we have four lessons from our time building agents: use abstractions minimally, start with a single agent, graduate to a network of agents when you have more intents, and finally, keep your prompts simple and focused on the happy path, and use guardrails to handle edge cases. [outro music]

79. 16:28

    Thank you. [audience applauding] [outro music]