The Demo I Wish I'd Had: OpenAI's Agents SDK... serverless!

Talk transcript

1. 0:01

   What does full stack AI engineering even look like today in a serverless environment? Let's talk about it. My name is Brook Riggio. I'm a technology partner with thryv.com, and excited to share with you some things I've learned about running production AI engineering workloads in a full stack and serverless environment.

2. 0:18

   So let's dive in. The question of twenty-twenty five is: How do we deploy zero-ops, resilient, agent-powered, user-ready apps today? How do we make it happen to get a- agentic workflows into the hands of users as AI engineers?

3. 0:34

   This is what, uh, our job is. This is our goal. Let's think about how we do this in a modern way.

4. 0:41

   There are, of course, so many options. We are, uh, overwhelmed with different pieces of the infrastructure, which basically is gonna need to look like this: Some type of client app, an agent framework, an orchestration layer, all running somehow serverlessly in the cloud, so we don't have to fiddle with all the details of managing ops.

5. 1:02

   Of course, each one of these options themselves has so many options. Uh, you could be running things in Remix, Astro, Vue, Svelte, Next.js, and so many more. For agent frameworks, um, uh, there's probably a new one announced every day.

6. 1:18

   Now we're probably hearing more and more about new agent frameworks like LangChain, Vercel's AI SDK, Flowise Agents, OpenAI's Agents SDK as well, and so many more.

7. 1:31

   There are also dozens of orchestration layers that we could be playing with. Temporal, AWS Step Functions, LangSmith, Inngest, and so many more. And of course, so many different serverless environments to deploy to.

8. 1:43

   Lambda, API Gateway plus Bedrock on AWS. Google has Vertex. A- Azure has AI Studio. Vercel, and dozens of other serverless cloud environments for us to deploy to. So many options, and probably in combination, at least forty-two million combinations of these things.

9. 2:04

   I don't even know what's gonna be the best, but I'm gonna tell you about my favorite. Because the question AI, AI engineers are asking is: How does it all work together?

10. 2:13

    How do we get the right pieces of the right layers to actually function? Uh, and if you run this through an LLM, you're gonna get lots of different answers.

11. 2:22

    You're gonna get, uh, some code that maybe might kinda work. Uh, even if you pick your favorite stack, there's, there's not really likely to be a bunch of reference code in the training data.

12. 2:33

    Uh, and even leveraging or- original docs can prove very difficult. I put this talk together because I struggled with this very thing, trying to make it happen, uh, over and over again in a production environment.

13. 2:45

    And I went through, um, a bunch of different combinations and landed on my favorite, which is what I wanna tell you about today. Uh, as maybe I gave it away in the talk description, but my favorite is indeed a combination of using Next.js with its built-in streaming, first-class server actions, file-based and nested routing in Next.js fifte- fifteen,

14. 3:07

    and a deep inte- integration with Vercel hosting makes this my choice for client apps. For agent frameworks, uh, from the first day that OpenI- OpenAI announced their Agents SDK, uh, I was all in on it.

15. 3:21

    It's got native tool calling, one-shot multi-agent calls. Uh, it built in tracing and eval hooks, so you can really get a lot of observability out of this thing. And of course, strong backing from OpenAI.

16. 3:35

    Uh, they're supporting this. It's not going away, and they allow you to interchange models, so you're not stuck in, uh, their ecosystem, even if you're using their open source Agents AI SD-- Agents SDK framework.

17. 3:47

    So, uh, overall then for orchestration, my choice is Inngest.

18. 3:54

    Inngest uses events to trigger things. It's-- You don't have to manage JSON documents of state machines or anything like that. Uh, it works entirely on demand. There's no worrying about keeping your servers warmed up.

19. 4:06

    Uh, it's got automatic retry, step-level observability. You can see what's happening step-by-step and where the errors are, and a really nice one-click integration with Vercel. So you can probably guess what I'm gonna say for serverless.

20. 4:19

    Let's go with Vercel. Vercel's got preview deployments with every push. This makes continuous deployment, uh, no longer a difficult concern, but an actual just integrated part of your workflow.

21. 4:32

    You can have your whole team reviewing your latest changes on preview branches. Uh, and, uh, even as you're getting that merged into main, uh, Vercel deploys it all for you, so you can see it functioning.

22. 4:45

    It's an incredible experience. If you haven't tried it yet, I'm here to tell you, you need to try it.

23. 4:50

    They also have a really strong edge network, automatic CDNs, uh, a bunch of integrated database options from, uh, Postgres to, uh, Redis to blob storage. And of course, since Vercel is the company that's behind Next.js, there's a really strong alignment with, uh, with developing the ideal cloud platform for the

24. 5:15

    Next.js infrastructure. Between Vercel and Next.js and the whole suite of tools that you can find in Next Forge, uh, there's a really strong ecosystem of, uh, pretty much everything you need to make things happen in the client-based app.

25. 5:31

    And with these other parts, you get to integrate your AI workflows right into the client app. So here's the beauty of it. This set of tools actually works. Now, the LLMs still don't know how to piece this all together, but that's what I'm here to tell you.

26. 5:47

    Maybe they can start training on my repo, and they'll get there. But for now, you're gonna need to understand how to get this implemented so you can guide your, uh, your, your vibe coding session in the right way.

27. 5:59

    Here's what the architecture looks like. The-- Or your user's gonna connect to your Next.js app. That is gonna be maybe checking the database. Is work done? Is there something cached?

28. 6:09

    Do I need to do something new? We can trigger a new workflow with an Inngest service, sending an event to Inngest. Inngest is your orchestration layer. That manages your connection to Python serverless functions where your AI agents are running.

29. 6:25

    The OpenAI Agents SDK is Python only right now, uh, but that's okay because when you deploy your Python functions to Vercel, Vercel n-recognizes it as Python and automatically hosts it for you.

30. 6:41

    Those Python functions can handle all your inference talking to AI, uh, talking to OpenAI, uh, to get the work done and return results to the orchestration layer, which can, again, inform your, your front end, your client app that the work is done, caching things in the database as needed.

31. 7:00

    So this is the essence of the architecture. But really, I'm sure what you wanna see is the code. So I built an example app that uses AI agents to create a newsletter, and I wanna show you how it works so that you can get a sense of how all of these pieces work together.

32. 7:18

    So let's take a look. Here's the README. Uh, this code, there's a link to this code in the notes below. Um, it's up on GitHub and, uh, you'll get to explore it there.

33. 7:28

    But let me run you through it so you can see what's happening. So first of all, we're highlighting that we are focusing on serverless scalability, the ability to support long-running jobs so your agent tasks don't just crash out or blow out the, uh, time limits available for cloud functions.

34. 7:45

    That lets you really pay attention to costs. You can adjust. You can pay only for what you're actually using and not keeping servers idle. Uh, all of these tools work together for a really nice local developer experience.

35. 7:59

    I'll get to show you a little bit of that right now. And of course, you have full type safety with Pydantic and Python and, and TypeScript across Next.js. So how do you make it happen?

36. 8:10

    Well, once you've cloned this repo, all you need to do is fire up three terminals: one to run your Python agents, one for Next.js, and one for your Inngest dev server.

37. 8:20

    This is a really nice feature of this orchestration layer. They give you a specific development environment to manage it. So let's see what it looks like. We'll just go like this.

38. 8:31

    Fire up three terminals here. Get Next.js going,

39. 8:48

    and we get npm going. All right. With those running, we can pop over and take a look at our app. Like I said earlier, we've got a README. Let's see what the app looks like.

40. 9:02

    Here it is up and running. This is the homepage of my Next.js app.

41. 9:07

    As you can see, running on [REDACTED:url], uh, and it provides a place to enter some topics, comma-separated list, as many topics as you want, and it will attempt to generate a newsletter that looks at what's going on at the intersection of those topics.

42. 9:21

    Let's see what's going on with AI engineering in Seattle. That's where I'm based, and we can generate a newsletter that will look up the latest, what's happening at the intersection of these topics.

43. 9:33

    So we fire this off, and it creates a, uh, a page that's starting to just pull against the database to see if the database has the answers for us yet, and we're using Vercel's blob storage for our data store.

44. 9:48

    Can use anything you like, but that's easy to plug in. And, uh, in the meantime, we can look at the Inngest server and see it running. Uh, it looks like one of our agent calls has already completed, and it got back some data.

45. 10:01

    Uh, and there's the second agent call that's fired off. It has saved it to the blob storage and did a little finalization. So here you can see our orchestration layer has broken the whole task down into individual steps.

46. 10:16

    Each one of those steps can scale up and down to the full length of what Vercel allows for a cloud function to run. Um, you can configure that for, uh, up to a minute, I think, on the free plan and even higher on pro plans, so you get plenty of bandwidth to play around with individualized tasks when

47. 10:34

    you break it out that way. Uh, and then overall, as the page refreshes, it finds the results in the database and presents back to us our newsletter about what's going on in Seattle and, uh, and with AI transformation of engineering.

48. 10:52

    All kinds of interesting things about the A- Seattle AI community and, uh, its own take on what we're talking about with engineering. All right, so there is the results of our page and, uh, formatted very nicely as a newsletter, and we can go back and generate more as needed.

49. 11:13

    So let's take a quick look through the code to get a sense of how all of this works. So we were looking at the README, and now we can see the project structure here that lays out everything in our app.

50. 11:25

    Inngest itself has an endpoint that that cla-- that the, uh, the orchestration server,

51. 11:33

    uh, connects to and shows us what's happening. We get all the detail and observability we need about what's going on step-by-step in our code base, uh, because it's served right out of our app.

52. 11:45

    We have then the newsletter API endpoints that Inngest calls, and that in turn invokes the Python agents, which are over here at a top-level API directory. When Vercel finds this top-level API directory, it looks in and serves each file as its own independent function.

53. 12:04

    So we get this as a cloud function. Uh, otherwise, the workflow is defined in a Inngest folder that's within our source- Code folder. And we can have as many different types of workflows as we want, each one triggering off of its own event.

54. 12:21

    So there you have it. Those are the pieces that work together to make this happen. Uh, ultimately, it's made up, once again, of, uh, AI agents that are powered by, uh, FastAPI, a really quick and easy way to put them together, right?

55. 12:36

    So here's a look at our AI agents. We have one that's doing research. We have one that's formatting. And, uh, each of those has its own endpoint in the FastAI app, which can spin it up really quickly.

56. 12:49

    All Vercel needs to know to get this going is this one line of code at the bottom. It fires it off as an independent file. And you'll note here that this repo does not even have a vercel.json config file.

57. 13:01

    Vercel automatically picks this up when you have things structured this way. Um, that's something I went in circles around with the LLM on, uh, in a numerous tries to get this, get this whole stack deployed correctly.

58. 13:14

    So we've got the AI agent, and I'll show you two also here, the Inngest workflow,

59. 13:20

    uh, which is found here in our Inngest functions. Uh, we define them here, and I really like to structure these, so we keep each step defined at the top, and each step is essentially just doing one thing of calling out to a function that we define below.

60. 13:41

    Uh, and of course, in JavaScript, we can define those functions later on in our code, uh, but this keeps our overall high-level orchestration very clear what's going on. We're essentially invoking each step and then doing some type checking on it.

61. 13:54

    Step one, call the agent. Step two, format the newsletter with another agent. And step three, save this to our blob storage, so the front end can find it and, and load it up.

62. 14:06

    Uh, so that's, that's the-- that's what Inngest provides as very nice lightweight orchestration. One thing to remember is that every step.run will be invoked in order, but code that's outside of step.run that you have, uh, for example, up here, may well be executed multiple times as it comes back to this function to figure out exactly which step

63. 14:30

    to run. So keep an eye out for, uh, duplication of, of execution outside of those step functions.

64. 14:39

    But inside the step functions, you can count on them to re-uh, reliably run in sequence and passing the results to a variable that you can then send into, uh, your next, your next step.

65. 14:53

    All right, so here we have the results of the initial research agent. Uh, after type checking, we pass that into formatting our newsletter. We take the results of that formatted content after type checking and pass that into saving to the blob.

66. 15:12

    And, uh, and then, of course, that's our final result, which, uh, Inngest shows us right there in that Inngest dashboard. So that's it. All of this fits within a Next.js repository deployed to Vercel.

67. 15:27

    It fits together very well and allows for all the scalability you would expect from a serverless environment, all the resilience that you need from a robust orchestration layer, and most significantly, all the agentic power that you can get out of OpenAI with the Agents SDK.

68. 15:45

    So this is my favorite combination, uh, and, uh, it's working well. It's working well. Uh, it deploys really nicely onto Vercel. You can see here it is running on a Vercel URL.

69. 15:59

    Uh, and Vercel shows us the full logs of how things get built. If you're building this, here's a place to check. Under the initial deployment, you can see your API agents folder gets deployed as a Python function.

70. 16:12

    It's running Python three dot twelve in Vercel's current Python environment, uh, all laid clearly, laid out clearly for you here. Full build logs are available if you do any troubleshooting, but ultimately, it just is as simple as actually running the Vercel command right from your terminal to deploy this code to Vercel.

71. 16:34

    I'll show you that too. We got one more shell to pop open.

72. 16:40

    The Vercel CLI command lets you sign in, connect to your environment variables. You can actually pull down your environment variables from production and run them locally. For this project, you need an OpenAI API key.

73. 16:53

    Of course, you need a Vercel blob storage token to use for storing your data. Uh, and, uh, that's about it. Those two environment variables are all you need to get going.

74. 17:05

    And you can fire off your build process right here.

75. 17:10

    That gives you everything you need to make it happen. So that's a tour of the app. That code is up on GitHub. Feel free to check it out, make some contributions.

76. 17:21

    It's open sourced. It's ready for you. Uh, I'd love to know what you wanna do with it. Uh, if you build on top of it, take it and replace the agents with, with your own workflows, uh, then link back to it, and we'll build a whole section in there with examples of what other people are doing with

77. 17:37

    that same basic structure. So that's what I've got for you. Thank you so much for your time. I hope you find this helpful, and I can't wait to see what you do with full stack agents in the cloud.

78. 17:49

    Thanks so much.