Building Agentic Applications with Heroku Managed Inference and Agents — Julián Duque and Anush DSouza

Talk transcript

1. 0:00

   [upbeat music] Welcome, welcome you all to this Lunch & Learn.

2. 0:18

   I hope you are enjoying lunch right now that the conference brought you. Um, we are going to be talking about our recent, uh, product that we announced at Heroku, which is Heroku Managed Inference and Agents or part of the Heroku AI offering.

3. 0:35

   And we will be talking about the fundamentals of building agentic applications with this service. This is not a session, this is a workshop, so if you wanna follow along, just bring your laptop.

4. 0:47

   You only need a browser. You are not going to install anything in your computer. I swear that it's going to be, uh, easy to follow along. Also, this content is going to be available for you if you wanna continue learning at home and, um, having access to the Heroku platform during the event and the weekend.

5. 1:09

   So initially, uh, we are going to do the setup while you get ready by signing up to an Heroku account, or if you already have an Heroku account, you can just get access into the platform.

6. 1:22

   I will give you a link for you to follow. Then my friend here is going to give you an overview of what we release, and then we will go and do the hands-on.

7. 1:32

   My name is Julián Duque. I'm a principal developer advocate for Heroku. I'm here, I am with Anush.

8. 1:37

   Yeah. Hi, my name is Anush. I work as a product manager for Heroku AI. Uh, thank you everyone for joining us. Very excited to talk to you guys as well as walk you through how simple Heroku is.

9. 1:48

   Of course. And let's get, uh, started. You can join the, uh, Workshop Heroku AI channel in the AI Engineer Slack, uh, workspace. There I share the slides and some links.

10. 2:02

    Also, this is the QR code that is going to take you to the workshop site. So in the workshop site you are going to find a link that has the content and a form when you enter your Heroku account email.

11. 2:18

    If you already have an Heroku account, just enter your email that you used to sign up to Heroku. If you don't have an Heroku account, go ahead, sign up for one.

12. 2:28

    You don't need to put, like, any credit card information. And with that form you are going to get access to Heroku. You will be able to deploy applications, use our services during the duration of this week.

13. 2:42

    If you have any problems with the sign-up process, let us know. We are here to help. And this is a follow-along workshop. What are we going to do is deploy a Jupyter Notebook to Heroku, and then we are going to load all of the workshop contents into this Jupyter Notebook, and from there I will take you, uh,

14. 3:03

    with the workshop material. So let's take a picture of this. Also make sure you join the Slack channel. There is the information in there. And while you are doing the setup, my friend Anush here is going to giving...

15. 3:16

    give you an overview of Heroku AI and what we release.

16. 3:21

    Okay. Um, thank you for joining us, friends. Uh, how many of you are familiar with Heroku? Have you used Heroku before? Show of hands. Oh, nice. So, um, to set the stage,

17. 3:36

    this is the most exciting time to be building right now, and especially building with AI.

18. 3:42

    And this point has come before, right? There have been inflection points with technology that have fundamentally changed the way we build things. There was the internet, there was the cloud, there were web apps, and now there's AI.

19. 3:54

    Previously when Heroku started, we took a similar challenge. People wanted to build and deploy their apps, especially with Ruby on Rails, but it was hard. It was hard to operate it, it was hard to deploy it, it was hard to scale it.

20. 4:06

    Heroku made that super simple with Git push heroku main, and a whole new host of developers could easily build, push, and scale their apps. We are doing a similar thing right now where we have taken on the challenge with AI.

21. 4:19

    We have seen that people wanna build with AI. They wanna build agentic applications. They wanna build agents that scale and operate in a way that is very simple. We wanna make sure that every software engineer right now is an AI engineer, and it is simple as attaching agents and AI to your apps.

22. 4:37

    So how are we doing that? Currently a lot of solutions are only for the day one problems. But what happens on day two? How do you operate it? How do you scale it?

23. 4:47

    There are so many models out there. How do you know this is the right model for your problem? How do you know your tools are running safely? Heroku has taken a very opinionated and curated set of models that we believe work best for our customers and developers will enjoy.

24. 5:04

    We have expanded that further by deeply entrenching these models in an agentic control loop that runs on Heroku that has access to tools like code execution, access to your data, all under the trust layer of Heroku.

25. 5:16

    And for these extension of agents, we are using the Model Context Protocol.

26. 5:22

    You might have seen online that people keep asking that who's gonna build the Heroku of AI or who... uh, this is the Heroku of X. So the question for who's building the Heroku of AI, it's Heroku of course.

27. 5:34

    Why wouldn't we? Okay. So what are the challenges right now people are facing? One of the things that I can see is that how do you, how do you figure out that this model works best for you?

28. 5:47

    How do you know that it's evaluated and traced and has the right technologies to make sure that it is performing the way you want it to perform? So these are the challenges that we're taking on.

29. 5:57

    We are curating these things such that whenever you work with the agentic applications you don't feel like it's a bunch of knobs and bells and whistles like a plane, but it's actually s- pretty simple.

30. 6:07

    By taking on, uh, the opinionated approach of having those defaults for you, you can get started with as simple as one CLI command. You, you just do He-Heroku AI models create, and that attaches AI to your app as a resource.

31. 6:21

    Uh, moving on to the next slide.

32. 6:23

    Of course.

33. 6:24

    Okay. So we have three major things that work great for our AI application or building agentic applications. One is we offer primitives like inference, where you can curate and take a set of models and access this in your apps.

34. 6:38

    We also have Model Context Protocol to extend your apps. You can build remote MCP servers on Heroku simplified way, and you can also build standard IO MCP servers that run in Heroku's trusted compute.

35. 6:50

    And they'll... They can also scale to zero so that you do not pay for things that you're not using. And we also have had pgvector, which is a great vector database for embeddings.

36. 7:00

    So all, all these together give you all the primitives that you need to build agentic applications.

37. 7:08

    So how do we do this? Heroku has a trusted compute layer called dynos, and the-these compute is w- the one that runs tools for you. So ex-example, we f- we provide first-party tools, like code execution, that run on Heroku's compute, and these can run on the compute and stream the data back to you and solve your problems.

38. 7:32

    We plan to offer our additional, uh, first-party tools, such as web search for grounding. Uh, memory. Memory is really important right now, so in the future we'll probably offer memory and other compute tools that can run on Heroku's compute and provide to you.

39. 7:46

    You can bring your own tools as well using MCP that can run on our compute and stream things back to agents.

40. 7:53

    Okay. With that, I'll hand over to Julián for a more hands-on workshop, and you can build all the things I spoke about within the next 30 minutes.

41. 8:02

    Okay, let's go back to this last, uh, picture for the folks that just joined so you get access to the workshop content. And remember, we have the Slack channel in the ai.engineer workspace where I already shared these slides and the link so you can follow along.

42. 8:19

    You can follow along with me right now, or you can follow along at home with more time.

43. 8:25

    So what are we going to do is the following, so let me show you. That link is going to take you to this page. In this page, you have access to the workshop resources.

44. 8:36

    This is a website that has the instructions step by step of what you are going to be doing today. I'm going to be following these same steps right here, right now.

45. 8:45

    And second, we have a way for you to get access to Heroku. If you already have an Heroku account, just put your email, the email that you use to log into Heroku.

46. 8:57

    If you don't have one, go and sign up first. You can sign up for Heroku. You don't need to enter your credit card information. With just this invitation, you are going to get access to the workshop.

47. 9:09

    These services are going to be enabled until, uh, the seventh, like this weekend, and I can extend it if you like the content we have for you. This is going to send you an invitation to a team.

48. 9:21

    The, the, the team is called AI Engineer World's Fair. Go accept that invitation, and then you have access to this dashboard. So from the Heroku dashboard, you go to dashboard.heroku.com, you log in, and you should get access to this view.

49. 9:39

    We already have, like, some pre-deployed applications here. The workshop that I will be showing you, the Jupyter Notebook, the Brave Search MCP. We are going to use a preexisting MCP to give you an example of how to call that MCP from the Managed Inference and Agents endpoint.

50. 9:59

    And I also have another application here, which is the AI Engineer Data that has an Heroku PostgreSQL database, and we are going to see how can we build agents that has access to this database as well.

51. 10:11

    But we start the workshop first by deploying an Heroku application. So we have this Heroku Jupyter template that you can deploy to Heroku. Go to the repository.

52. 10:26

    After you have the account and logged into Heroku, just click to deploy. Click to deploy it.

53. 10:37

    You are getting this, uh, page that will ask you for an application name. Make sure that this application name is unique.

54. 10:47

    And the internet situation is a little bit slow. Let's hope it gets better

55. 10:54

    to show you the things working live. Otherwise, I, I need to start getting access somewhere.

56. 11:02

    If you join this link, please go to Slack channel. It's called workshop Heroku.

57. 11:07

    Heroku-

58. 11:07

    Yeah, AI. Yeah

59. 11:08

    ... Heroku AI workshop. And in the AI Engineer Slack channel, you can find the how to get started instructions. If you guys are stuck somewhere or you don't know how to get started, please raise your hands and I can come help you out.

60. 11:20

    Beautiful. This is the deployment page. We are going to create a new application. Remember to have a, like, a unique name. Let's do... This is [REDACTED:username]. This is going to be my Jupyter.

61. 11:32

    It is not unique. So if it is not unique, you are going to get an error. So let's do workshop to get a, a unique. And for the app owner, if you want this workshop to work without you paying anything, make sure to select the AI Engineer World Fair, which is the team we have invited you to.

62. 11:55

    These, uh, Jupyter Notebooks are password-protected, so there is an environment variable here for you to define the password to this workshop. I'm going to use a super secure password, lab, and then click to deploy.

63. 12:10

    So this is going to deploy the application with a, a dyno where the Jupyter Notebook is going to run. The dyno is pretty much the- Container unit where your application runs, like the virtual machine on Heroku.

64. 12:25

    And it also has Heroku Postgres, so all the work that you are going to do in the Jupyter Notebook is going to get persisted into this database. And this is going to take a little bit of time to, to install.

65. 12:38

    This is pretty much, uh, fetching all the source code of this Jupyter Notebook, uh, template to Heroku, building that application and all of the dependencies, and then you have this available for you to use.

66. 12:52

    So basically you are going to get something like this. I already have this deployed, but you don't have any workshop here. So we are going to be loading that notebook.

67. 13:03

    And to load that notebook, you go and keep following the instructions. Oh, the third step that I miss is how can we provision Managed Inference main- mainly how can we provis- provision an AI model to an Heroku application?

68. 13:22

    So Managed Inference is a service that let you run AI models within the same infrastructure where your application is running. So the data is not going to third parties.

69. 13:33

    If you are using, let's say, like, OpenAI or Anthropic APIs, the data is going outside your application. This is keeping, uh, everything running inside the same network that your application is running.

70. 13:47

    And to provision this you can use the Heroku CLI, but we are going to make things easy. Once the application is deployed... This is taking, uh, time to deploy, so let's go to something that is already deployed.

71. 14:00

    Let's say this one that I, I already, uh, created before. You go to your application in the dashboard, you click to, uh, the management page, then you go to Resources.

72. 14:14

    And in Resources is where you can provision add-ons. Add-ons like Heroku Postgres, or the Key Value store, or a- any other third party add-on that we support on our marketplace.

73. 14:25

    The one that we are going to provision is called Heroku Managed Inference and Agents. This is our AI offering. So you are going to click on this add-on, and then you will select the model that you want to use in your application.

74. 14:40

    We support text-to-text- models from Anthropic, Claude 3.5, 3.7, and 4. We support Cohere Embed for embeddings, and we support, uh, Stable Image Ultra for image generation. We are going to be working mostly with text-to-text with inference models, so let's do Claude 4.

75. 15:01

    We submit, and that's it. Now my application can access AI services. That's the only thing that you need to do. And how this works, this is going to enable your application to have access to environment variables that contain the API URL, the API key, and the model ID.

76. 15:22

    And with those three things you can use, like, an existing SDK that supports the OpenAI specification or the OpenAI API, or you can perform, like, an HTTP request or build your own solution to, uh, run these applications.

77. 15:41

    So now we provision AI into our app.

78. 15:45

    Now our another app is deployed, so we can do the same with that, so I'm going to keep following. Then let's go and load the workshop. So I'm going to be copying this URL.

79. 15:57

    That's the Jupyter Notebook we are going to load on our Jupyter template that we deploy. And here you go to File, Open from URL. You paste that URL, click on Open, and voila.

80. 16:13

    We have our, uh, notebook ready to start.

81. 16:19

    So remember that we provision the Managed Inference and Agents service into our application. That is going to give us three environment variables. If I go here to Settings,

82. 16:33

    Reveal config vars, we see the environment variables there. The INFERENCE_KEY, the MODEL_ID, and the INFERENCE_URL, the API key that we will use to call the service.

83. 16:47

    So obviously the first step that we will do is to load those environment variables on our Jupyter environment. So we will do this workshop in four parts. The first one is set up the environment.

84. 17:01

    We are going to just load those environment variables so we can continue doing our activities. Second, we are performing managed inference. This is your basic chat completions endpoint. And we will see two examples, one doing, like, basic chat completions, and the other one doing a streaming version of those chat completions endpoint.

85. 17:24

    The third, which is one of my favorite ones, is the Heroku tools or the agents part. We have another endpoint that lets you run agents that we support natively.

86. 17:36

    We have code execution agents, we have database access agents, document conversion, uh, execute commands on a dyno, and we are going to continue adding more to those. And this is just an endpoint that performs the execution of the agent in the Heroku infrastructure in our dynos, and it is just an one-off dyno.

87. 17:58

    What is a one-off dyno? It just spins up, runs the code, and it scales back to zero. So you are not paying for an application that is constantly running.

88. 18:07

    You are just paying for the amount of compute that that tool took to execute. And that endpoint also supports MCPs. MCP is that you can deploy to Heroku

89. 18:19

    And then attach them to your, uh, inference agent endpoint. And last but not least, we are going to take a look at that MCP support that we have. How can you deploy an MCP to Heroku, attach that MCP to the agent's endpoint, and also use those MCPs externally, remotely through the MCP gateway or MCP toolkits that we

90. 18:43

    have? I know it sounds like a lot, but it's going to be easy. I have a couple of exercises for you if you wanna write some code. But basically, what I will be doing here is just, like, running the code that I already have implemented here.

91. 18:58

    My good friend, thank you for coming. Okay, so first thing I told you, we need to load these three environment variables, inference URL, inference key, and inference model ID.

92. 19:08

    And there is a fourth one that we will use for certain, uh, tool execution on Heroku, which is the target application name. So what are going to put you here in target application name, the name of the application you deploy, the name of the application, the Hero- the, the Heroku Jupyter application that you deploy.

93. 19:33

    In this case, my worsh- my Jupyter Notebook is called AI Engineer Workshop. This is the one that I'm going to use here. I am giving permission to my tool to run commands and perform compute operations on this dyno, on this application.

94. 19:51

    So here you just replace to the name of your application so you can do those examples. And later, I will show you also how can you give access to the application that has the database to run the other examples that we have for you.

95. 20:07

    So let's, uh, load the environment variables. Now the environment variables are ready, so I can continue with the examples.

96. 20:18

    Then let's go with Managed Inference. I mentioned that for Managed Inference we have the basic chat completions endpoint. This is an endpoint that you will find on services like OpenAI or Cloud Anthropic.

97. 20:31

    Basically, you ask a question, you send a message or an array of messages, and you are going to get a response. This endpoint also supports custom tool execution, but this tool execution is something that is happening on your code.

98. 20:47

    So you'll need to specify the function and capture, like, the information to execute those. Or if you are using an agent framework like ADK or LangChain, it also has, like, custom tool execution internally.

99. 21:01

    But for the example, we are going to perform a basic inference. To perform the basic inference, we are, we are going to do a basic HTTP request. So we have the inference key, the model ID on the payload, so we are going to use Cloud Four, which is the model that I provisioned here on my app.

100. 21:19

     And the message that I'm going to send as an user is, "Explain the conte- the concept of Managed Inference in one sentence." You can adjust the parameters that the OpenAI API supports like temperature, maximum tokens, uh, top P, et cetera.

101. 21:36

     You can take a look at the documentation by clicking on the, uh, path endpoint, and it will take you to the documentation of that endpoint and all of the parameters we support.

102. 21:50

     And then after performing the HTTP request, we get the response as JSON, and let's run this.

103. 21:58

     And let's see how this work. So we get a response like any other AI endpoint works. So we have a chat completions object with the different choices, like the amount of messages that it returns.

104. 22:11

     Basically, we have just one message here as an assistant, and the response is Managed Inference is a cloud service that handles the deployment, scaling, and maintenance of machine learning models for real-time predictions, et cetera.

105. 22:25

     So that's a basic example. So this is something that you can do everywhere, getting inference. It became a commodity. And also to show that we also support the streaming.

106. 22:37

     Of course, we wanna have, like, a better experience, uh, while building our applications. We need to support streaming, meaning that it's not going to return the whole answer as just one JSON object.

107. 22:49

     It's going to start giving you chunks so you can get that real-time feedback when getting the response from the inference service. So here I'm just consuming chunks and printing out those chunks on the s- uh, on the screen.

108. 23:05

     The first ex-sample is going just to show you the raw version of this request. So this is using a server-sent event approach. It's sending you message events. Every message events has data, and each data is a piece of that completion object.

109. 23:25

     But in this case, you are getting like a, like a delta, like a version of that message. So you see here in the content we get nothing, then streaming is crucial for this, and then it continues giving me the response.

110. 23:42

     We get a bunch of different responses, but let's take a look how that looks like on an application when you are rendering the chunks. This is just the pure objects to understand how the API works.

111. 23:54

     Here I have another example where I am doing some parsing of that information, extracting the delta content of each response, and then at the end rendering everything as a beautiful markdown.

112. 24:07

     When I execute this, the inference service is going to start responding on real time, chunk by chunk, all the messages that I'm getting from the service. As you can see, it is a markdown response.

113. 24:19

     So at the end when we finish, we use the markdown, uh, display object here in Jupyter to get the beautiful answer here. Sometimes it will give you like a, a code example depending on what the inference, uh, service is giving you, and that's how this works and how...

114. 24:39

     This is how you can build an application to consume these real-time streams.

115. 24:46

     But now let's talk about the good part, which is the agents. Who has built an agent before, like an application that executes a tool or run, run some code?

116. 24:57

     Perfect. We have like couple hands there. Amazing. So the tools that we support today, and Anush here as product manager can tell us what we are going to be supporting in the future, are, uh, dyno run command.

117. 25:10

     Dyno run command allows you to execute a command on a dyno, like a Unix command or a script that you are, you, you already have pre-deployed on your application.

118. 25:21

     It will execute that and give you the response. So this is pretty good to run trusted code, code that you have written, that you know that it works, that it has a predictive, um, uh, predictive result, and you can just have that code running on your dyno and execute it through this, uh, this agent, this tool.

119. 25:43

     We have query databases. We have two tools here. One is Postgres get schema, and the other one is Postgres run query. Postgres get schema, mean, the LLMs doesn't know the shape of your database.

120. 25:56

     If you ask for data, it will hallucinate an SQL query, or if you have some sort of retrieval augmented generation, or if you are grounding the prompt with the shape of your database, it will generate something that is close to the shape of the data of your, uh, database.

121. 26:17

     But with this tool, it will get exactly what you have on the, on the database, and then we can pass that schema to the next tool, which is run query.

122. 26:26

     It will generate a query that will run on your database, get the result, and make inference over those, uh, data. Then we have a couple other tools for document transformation, HTML to markdown, PDF to markdown.

123. 26:43

     You pass an URL, like a website or a URL of a PDF that is hosted somewhere, and it will give you the m-markdown response so the inference can work over that text content.

124. 26:55

     It's going to perform that text extraction for you. And my favorites are the code execution one. We support for today Python, Node, Ruby, and Go. So the LLM will generate code that will run on Heroku on a one-off dyno, and it will return that response back to the inference service.

125. 27:15

     And these also support dependency installation. So the input parameter has like a, a packages array parameter where the LLM automatically say, "Okay, my Python script is going to use, uh, Pandas or NumPy.

126. 27:32

     Let's install it before I attempt to execute the tool." And all of these are just MCPs that we natively support. But you can also extend this agent endpoint by deploying your own, your own MCPs.

127. 27:48

     So now let's take a look. The agent's endpoint is different to the chat completions. It works similarly with, uh, some minor differences. For example, all of the responses are a stream, so it doesn't support like the synchronous call where it waits to give you like the full response because code execution, access to database, and these tool execution

128. 28:10

     take time. So we prefer to have it like a stream. And, um, depending on the tool you are using, you might need to give access to the tool to your database or to an application, for example, to execute a command.

129. 28:27

     So let's try that first one. Let's ru- run a demo to run a command on an Heroku dyno.

130. 28:36

     So here we are setting up the payload similar to chat completions. We have the model ID. We have the message. We are asking,

131. 28:46

     "What is the current date and time on, on the server?" LLMs doesn't know anything about like the real time. They need to use tools to be able to have context about what's going on right now.

132. 28:58

     I have asked the LLM about like information, and it gives me like information from twenty twenty-four because that's the date when they stop training that model. But with this, it knows exactly what time is 'cause it's running this command on the server.

133. 29:12

     So the tool is an Heroku tool, so I'm specifying the type to run this on Heroku. The tool that I'm executing is the dyno run command, and then the parameters for this tool are, okay, the target app name.

134. 29:28

     Remember I told you use the name of the application you deploy. And then the command to run is date and a description of pretty much what's going on with the, with the output of this, uh, command.

135. 29:46

     And I'm calling it. I'm just calling the agent's, uh, Heroku endpoint. I'm passing that payload that has the tool. This is also a, a stream endpoint, so I am having the, the stream output here.

136. 29:58

     And I'm just extracting the tool calls, and this is a standard also shape, like API shape that other, uh, API support for tool calling. But this is specifically running the tools on Heroku.

137. 30:13

     So let's execute and see the example. And I'm hoping you can execute this tool in your, in your Jupyter Notebook. So it is running this on my AI Engineer Workshop application.

138. 30:26

     It's going to execute the dyno run command. So now it is waiting until it runs the command on my app. We get the, the date and time, and then the inference response.

139. 30:39

     Cool. I mean, it's a basic example but powerful. Instead of running data, imagine if you deploy your own screen that does your data extraction or connects to external services, you can just call it, call it from here from this endpoint as well.

140. 30:56

     The other one, code execution. Uh, with code execution, I am asking the inference service to perform a operation, and I am passing that code execution tool. So now that it has that tool enabled, it will try to generate code in that language, run it on Heroku, and give you the response back.

141. 31:18

     For this example, I will invite you to change Node to Python, Ruby, or Go to see different, uh, responses. I'm a Node developer, even though I have a bunch of Python here.

142. 31:31

     Some of you might recognize this is LLM generated. So sorry, not sorry. That's what tools are for. And I'm going to be executing that, uh, CodeExec Node to perform this operation.

143. 31:44

     What is the 30th Fibonacci number? This is a basic algorithm. I just wanted to do something, uh, easy. I will invite you to change this to perform a different operation and see what you can do, like break this thing.

144. 31:58

     And then for the execution, I am just parsing the response so I get like beautiful, uh, markdown code highlighting and everything. So let's perform the operation.

145. 32:11

     And, um, it will generate, uh, the algorithm.

146. 32:18

     Let's see. Okay, I will execute CodeExec Node. This is the code that I'm passing as the input. This is the beautiful markdown syn- syntax highlight code that it will execute.

147. 32:29

     That's JavaScript. Yes, that's JavaScript. And at the end, it's going to execute that code on Heroku, and then I am getting the response back. The Fibonacci number is two, three, three.

148. 32:41

     But I want to do that in Go, so just change

149. 32:45

     from Node to Go. Execute this, and it will do the same but now the code that we are going to see here is a, is a Go code, and it will do the same.

150. 32:57

     Get this Go code on an Heroku dyno, compile it, run it, and get you the response back. That looks like a Go code to me. I am not a, not a Go developer, but that definitely looks like Go.

151. 33:09

     And it execute the response, and it gave me the same thing, and an explanation, like the inference operation over that tool execution.

152. 33:19

     But that's cool. So far we have just called one tool. The good thing about agents is that we can, uh, chain calls together, mix and match. We, we have like different agents acting together.

153. 33:31

     So now let's complicate things a little bit more, and we are going to use two tools. One is the HTML to markdown. So go to a website, do something, and then get that result and use it on this other agent.

154. 33:46

     For that, we are going to use two tools, HTML to markdown and code execution Python. So the prompt here is use the Python snippet from the Wikipedia page for Euclidean algorithm to calculate the common divisor of two hundred and, uh, two hundred and fifty-two and one hundred and five.

155. 34:10

     So it doesn't know the algorithm. I am telling directly, "Go fetch the exact one that is on Wikipedia, then run this code on Heroku and give me the response."

156. 34:22

     I'm enabling those two tools, HTML to markdown and code execution Python,

157. 34:28

     and let's run this. And now we are going to see multiple tool calling. Getting the page in markdown. So now it recognize that the Euclidean algorithm page is this one.

158. 34:39

     Now it is reading the whole content of that page in markdown, attempting to get the algorithm from there, then it's going to generate the code, then it's going to run the code, and at the end I'm going to get the response.

159. 34:52

     This execution takes a little bit more time because it is performing multiple, uh, tool runs at the same time.

160. 35:01

     Any questions so far? Is anybody doing it? If not, you have access to the workshop. You can do it, uh, at home with more time.

161. 35:09

     Yeah.

162. 35:09

     Modify the code, analyze it, and make sure, uh, the concept is understood. So it got the algorithm here,

163. 35:19

     and it execute the thing on Heroku. Now I'm just waiting for the last inference step, and that execution is done. There you go. I found the Python implementation from Wikipedia.

164. 35:33

     I calculate that number that you asked me. This is the Python snippet from Wikipedia, and then it implemented in this code, and this is how I did it. These are the explanation.

165. 35:46

     Beautiful. It works. Now, before I get into Postgres execution, we need to have access to the database, and right now the application that you have doesn't have access to that database.

166. 36:00

     If you wanna give it a try, you are going to go to

167. 36:07

     the Heroku dashboard. Where is the Heroku dashboard? Here.

168. 36:14

     To the AI Engineer World's Fair. Click on AI Engineer data.

169. 36:26

     Go to Resources. And we have one thing here. The Postgres database tools only work on followers. Followers on Heroku are read-only. For security reasons, we don't wanna give an LLM tool write access to your database because, of course, uh, LLMs are not

170. 36:51

     trusted for that. If you want to give write access to your production database to an agent, I will invite you to deploy an MCP that does that, like the PostgreSQL MCP, give the access under your own risk.

171. 37:05

     The ones we maintain, we wanna make sure we don't, like, break production. So we have two different databases here. The master one or the main one is database. Don't touch that one.

172. 37:18

     The other one that has attachments is the follower. So you are going to expand these, manage attachments, and add your application,

173. 37:30

     like Julián Jupyter, and that's it. Now I gave my application access to the database. Second, you will need to get the name of that database.

174. 37:45

     So go to your application. In this case it's, uh, Julián Jupyter.

175. 37:54

     Go to Resources, and this database here, AI Engineer Data,

176. 38:07

     in my application is called Database, so that's the name I'm going to use in code.

177. 38:15

     Since I am working on AI, um, AI Engineer Workshop, the name is different, so I'm going to show you so you can rec- recognize the differences in names. This is just the name of the environment variable on Heroku that has the connection string.

178. 38:33

     So here I have access to that database. On this application, my database is called Heroku PostgreSQL Aqua, so that's the name you are going to give and change it in your code.

179. 38:47

     Heroku PostgreSQL Aqua. So now I have access to that specific database on my app. I cannot access databases from other applications. This is for security reasons, so this is why I have to do the attachment first to be able to give permissions to my database.

180. 39:05

     If the database lives in your application, you don't need to do, to do this because your application already has access to it. That's the only thing that we need to do here to, to set up.

181. 39:16

     Uh, then we are going to enable two tools, Postgres Get Schema and Postgres Run Query. These are going to run on my application, remember the target app name, and these are going to access that specific database.

182. 39:34

     So I'm giving permission to go to those two, uh, places. So the database that we have here is a database for a solar energy company. It contains a table full of metrics with, uh, energy consumed and energy produced every hour in kilowatts per hour, and we have metrics for maybe two months.

183. 39:58

     So now what type of applications can we build with this? Like asking questions like how much energy has been saved in the last thirty days?

184. 40:09

     It's going to understand the shape of the database. It will see that it has a metrics table with these columns, then it will go and generate an SQL query to be able to get this information, and then run that query and give you the response.

185. 40:27

     And let's execute that to see how it works.

186. 40:34

     So first step is, "I don't understand your database. Let me get the schema."

187. 40:41

     So to get the schema, it executes the tool, and then you are going to see the full schema being printed here on the screen in a moment. So now we have the schema of my database.

188. 40:53

     It see the different tables, metrics with the fields, uh, products, systems, users, et cetera. So I have the information that I need here.

189. 41:05

     Beautiful. Now it will run the, the query. So it already generated the query that it is a hundred percent sure will run on my database because it knows the shape.

190. 41:18

     This is the response. It got the data. And now that after it got the data, it's going to perform the inference. It's going to extract the answer from that data, and at the end it's going to tell me the information.

191. 41:31

     It wanted to run more queries to get a breakdown of the energy by system. I have three different systems, one system that performs well, the other one that doesn't perform pretty good, the other one that it's totally horrible.

192. 41:46

     So it is also trying to get more information. Just from one, one question it is doing inference on my data, and I just enabled two tools. I didn't need to do anything else.

193. 41:58

     And there we go, the report. These are the key metrics. This is how much you have saved, and this is the breakdown per system, the best performer, good performer, and the energy deficit.

194. 42:11

     And this is ga- getting access to your data and acting over your data. We can add a third tool. Let's say Code Execution Python. I want to generate a graphic.

195. 42:23

     So it will now do a Matplotlib, uh, graphic with that data, and you can keep adding more, more things. But also the more agents and tools that you add, it will take more time to perform the whole operation.

196. 42:38

     But that's an example how can you mix and match these tools that we have here and, uh, how can you give access to agents to your data. I have a couple of exercises.

197. 42:50

     Just try to come up with an example to use a PDF and extract something from that PDF and run code. And you can do this on your own time.

198. 42:58

     We are running low on code. I want to show you now how can we

199. 43:04

     deploy and run MCPs with the same endpoint. So those are the tools we maintain, but what about the tools you are building and the tools that already exist on the MCP ecosystem?

200. 43:20

     So with Manage, Manage Inference and Agents, uh, let's say you go to the dashboard, to the configuration page, and here

201. 43:31

     you get access to the, um, model configuration, and here it has the toolkit integration and the MCP server list. So basically, this MCP Brave that I have here, the Brave Search MCP, and you might have already have access to this, so you can attach it to your own Jupyter Notebook.

202. 43:54

     Pretty much to attach an MCP, you click on Manage MCP Servers. You attach it as an application similar to what we did with the database. It's just another local application.

203. 44:05

     And there you go. We have an MCP server here that is exposing to tools, web search and local search. So now we can use these tools on the agent's endpoint.

204. 44:19

     So let me go and take a look at that. So these are the instructions, step-by-step, how can you enable an MCP. Then

205. 44:27

     it's just another tool. This time it is not a Heroku tool, it is just an MCP, and this is the name of the tool that I will execute. MCP Brave

206. 44:42

     is the name of the namespace. You can have multiple MCPs here, so this is kind of like a na- like a namespace for your MCPs. And I am going to execute Brave Web Search.

207. 44:57

     This requires an API key, and I already have the API key on my application. If you want, you can go ahead and take it. I'm going to, like, remove that API key in a moment.

208. 45:08

     Don't perform like, uh, two thousand queries, otherwise I, I'm going to get, uh, charged. But now I have access to it and the security to also enable those, those keys to your MCPs, and I will execute it as another tool.

209. 45:24

     And the prompt that I'm sending is, what is the most recent news about AI agents? Let's execute this tool.

210. 45:32

     And this MCP is running on Heroku the same thing as code execution or the Postgres, uh, tools. It will spin off a dyno, run the MCP as a standard input and output, and scale back to zero, so you are not paying for something that is constantly running.

211. 45:51

     And there we go. That's the search from the Brave Web Search. So this is the response from the tool, and now it's going to, uh, render that after the inference, uh, operation.

212. 46:07

     So I'm running MCPs on Heroku. A quick example how can you deploy an MCP.

213. 46:14

     I have, uh, an exercise here. Deploy an MCP to Heroku. I have a Perplexity Ask MCP that I just forked. This is the official Perplexity MCP. I fork it to my repository to just do one thing.

214. 46:32

     To make it Heroku-compatible, you use the Procfile, which is the file on Heroku that defines how an application is executed,

215. 46:43

     and I added the entry point as an MCP. So this is a new process type we support. Everything that starts with MCP is going to be recognized as an MCP on Heroku.

216. 46:54

     It will execute this standard input and out- and output code. It also requires an API key that I don't have,

217. 47:02

     but you can deploy it. To deploy to Heroku, I added a button to make it super easy. Click to deploy. I deploy this, uh, MCP to my space, and then I attach it to my application.

218. 47:15

     This is my Perplexity MCP. Let's add available to the space.

219. 47:27

     I don't have one. I deploy. There are certain MCPs, if you don't have an API key, it fails when you run them, so this is why I'm specifying something.

220. 47:38

     It doesn't need to be valid. And this is now a Node.js application. It will just deploy this, build this, and keep it available on Heroku, and you will be able to attach it to your, to your app.

221. 47:49

     So this is an exercise you can do and write the code to run that, that MCP that you just deployed.

222. 47:58

     Now we got the response from the previous, uh, Brave Search,

223. 48:03

     and this is pretty much the most recent news about AI agents. But remember that I told you that you can also use those MCPs outside of Heroku, not only for Heroku agents.

224. 48:16

     Let's say you are using Cursor or Cloud Desktop, or you are writing your own agent in a different platform, but you wanna have those MCPs available remotely, you can use those too.

225. 48:30

     So here on the management dashboard, you see the Toolkit Integration page. So all of the MCPs that I deployed to my app are going to be available through this endpoint.

226. 48:42

     So this is a server-sent event endpoint, and it is, um, authenticated behind a bearer token, and we are working on OAuth support.

227. 48:52

     We're getting, we're getting-

228. 48:54

     So that you can run it securely. You can build remote MCP servers on Heroku that are accessible without a bearer tokens. We don't like bearer tokens. [laughs] Perfect. So now that the MCP is there, let's add, like, my, my Perplexity really quick.

229. 49:09

     Let's refresh. I need it. Perfect. I have it. It's available. I go to my toolkit, copy the token. It is the same, the same token that you use for inference, so I don't need to do this.

230. 49:23

     You just need the URL. And in my Jupyter code, I have a basic MCP client. I'm using...

231. 49:32

     So I need to install the dependency here. I'm using the MCP package from, uh, from Anthropic.

232. 49:42

     And I'm creating an MCP client. That MCP client is going to my Heroku endpoint, passing that API key as a, a authorization header, and let's run it to list the tools that are available, and then execute Brave Web Search.

233. 50:02

     So I'm executing the MCP that I deploy on Heroku outside of Heroku.

234. 50:08

     Let's run this demo. And it connected. We have the following tools: Brave Local Search, Brave Web Search, and Perplexity Ask, the one that we just deployed and enabled with just two clicks mostly.

235. 50:26

     And it executed the web search. I am not processing the result here. This is just a very basic example, but that's how can you deploy an MCP and use it externally.

236. 50:37

     And I think, and I, I have also, like, a cursor, uh, MCP gateway that I use for MCPs that I deploy. I just use the same approach to use my MCPs and cursor.

237. 50:50

     And last but not least, uh, I told you that the, at least the chat completion endpoint is compatible with OpenAI API, 95%. We are working to bring it to 99%.

238. 51:03

     There are, like, certain parameters that are not supported, but you can just use the SDK. So let's use the SDK to perform a basic operation. I am just using the OpenAI SDK with the API key and URL from Heroku.

239. 51:20

     I perform an inference, and we should get a response

240. 51:30

     in a moment. And there you go. We have the response. And that's pretty much what we had for you today. You get access to this notebook. Keep playing with it.

241. 51:45

     I extended access to, to the Heroku platform, that team, until the weekend, 'cause right now, unfortunately, we don't have a free tier. We are working hard to bring it back.

242. 51:56

     Uh, but you can go deploy, try it out, and if you have any questions, please, um, uh, connect with us on social.

243. 52:06

     Uh, we have, um, the dev center docum, uh, site for documentation, the Heroku AI website. Uh, we created a Heroku community on Twitter or X. These slides are on the Slack, so you can get them from the Slack.

244. 52:23

     And thank you very much. I hope you enjoyed this workshop. [audience applauding] [upbeat music]