AI Engineer World's Fair 2024
From Software Developer to AI Engineer
Read the talk
From Software Developer to AI Engineer
Build on software engineering skills by learning model behavior, using AI development tools, testing models through a common API, and composing tools into agent workflows.
From a talk by Antje Barth and Mike Chambers
Before you start: Familiarity with application development, APIs, and basic Python will help with the examples; no machine learning research background is required.
Learn what sits behind the API
If an AI application can begin with a few API calls, what does a software developer still need to learn? The opening role diagram places AI engineering on the application side of the API boundary. The role associated with swyx’s The Rise of the AI Engineer makes room for developers who build with models without becoming ML researchers or data scientists first. Access to a model removes a substantial barrier to getting started.
An accessible API does not remove the need to understand model behavior. You need enough foundation-model knowledge to understand why an output might differ from what your application expects. You also need to recognize when to customize a model for a particular use case or dataset, including through fine-tuning, and how application functions can give it access to additional systems. Those are separate engineering decisions: understanding the model, adapting it, and connecting it to software.
For that foundation, Antje Barth recommends Generative AI with Large Language Models, a course developed with colleagues, Andrew Ng, and DeepLearning.AI. It introduces generative AI fundamentals with the goal of building real applications and is offered through DeepLearning.AI and Coursera.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use AI across the development lifecycle
The second step is to use AI developer tools in everyday work. Natural language becomes another interface to development: a request can produce a rewritten README, while existing code can become an English explanation or function documentation. The useful direction runs both ways—from a description toward a software artifact, and from code toward understanding.
That expands the opportunity beyond typing code faster. Planning, design, testing, deployment, and maintenance surround implementation. Barth offers a rough workload framing: sometimes up to 70% goes to boilerplate, documentation, and maintenance, leaving perhaps 30% for creative development. This is an illustrative split, not a measured productivity result for Amazon Q; the point is that assistance can target the work around feature implementation too.
Amazon Q Developer is introduced as an assistant for that broader lifecycle. Its agents can tackle feature development and code transformation, in addition to code completion. Moving an older Java codebase to a newer Java version is one proposed transformation use case; the demonstration that follows instead builds up from IDE assistance to a feature change spanning project files.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move from IDE guidance to a reviewed feature change
Mike Chambers begins inside the IDE with an ordinary chat question about creating a serverless application. Q responds with a sequence starting at AWS SAM CLI installation and continuing through project creation. Once generated project code is available, he selects it, opens the context menu, and asks Q to explain it. The selected code becomes part of the explanation prompt. The same interaction can help with an unfamiliar legacy system, where the original author is no longer available to explain the implementation.
Next, Chambers supplies a comment describing an input-checking function, then narrows the requirement: trim any incoming string. Q generates the small function from that local description. This is the familiar completion workflow: the developer states a small intent close to the code, and the assistant supplies an implementation.
The demonstration then grows from a function into a feature. Using the recording’s /dev interface, Chambers requests a DynamoDB search by category and specifies how its output should be formatted. The workflow proceeds through distinct stages:
- Describe the feature. Provide the search requirement and output details in chat.
- Inspect the plan. Q identifies project changes, including work in
template.yamland a proposedsearchByCategory.mjsfile. - Review the change set. Q follows the plan to generate proposed code, exposing differences from the current project.
- Accept the changes. The demonstrated
Insert codebutton applies the generated code to the project if the developer approves it.
A generated plan becomes useful when it leads to inspectable changes. The review step separates a proposal from code accepted into the project; this is more than a longer autocomplete suggestion.
For further examples, Barth points to Amazon Q Developer Center, along with the conference’s Q sessions and AWS booth demonstrations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prototype around a use case, then evaluate models
The third step shifts from using an AI assistant to building an AI application. Start by defining the use case, then choose candidate models and decide how to adapt them. Prompt engineering, retrieval-augmented generation (RAG), and fine-tuning are options to consider against that use case, rather than mandatory stages every application must pass through.
Responsible AI policies, privacy, security, and guardrails belong throughout this workflow. After integration, agents and deployed models also need ongoing maintenance—the operational work often described as GenAI Ops or LLM Ops. Model selection sits within this larger lifecycle: evaluate candidates thoroughly because different application tasks may call for different models. A common application platform does not imply a single best model for every use case.
Amazon Bedrock is presented as the managed platform for this experimentation. It combines access to foundation models with tooling for fine-tuning, RAG workflows, and agents. Barth emphasizes a secure environment and control over application data. The talk’s model lineup includes providers AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, and Stability AI, alongside Amazon’s Titan models.
The release context is June 2024: Barth highlights Claude 3.5 Sonnet as having arrived on Bedrock the previous week. That historical lineup sets up the next integration problem: how to experiment across providers without rewriting the invocation code for each one.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep model invocation consistent with Converse
The Converse API, newly introduced at the time of the recording, provides a common invocation structure. Instead of making the application translate parameter names and message formats for each model, Bedrock handles that translation. The interface accommodates system instructions and user/assistant messages and returns a consistent output structure.
The Python example follows a short sequence: initialize the SDK client, construct messages, and call converse with a model ID and inference parameters. The following uses that same structure, with the model ID supplied through the environment:
python
import os
import boto3
client = boto3.client("bedrock-runtime")
messages = [
{
"role": "user",
"content": [{"text": "Explain how an API helps developers use AI models."}],
}
]
response = client.converse(
modelId=os.environ["BEDROCK_MODEL_ID"],
messages=messages,
inferenceConfig={
"maxTokens": 256,
"temperature": 0.2,
},
)
for block in response["output"]["message"]["content"]:
if "text" in block:
print(block["text"])
The request separates the conversation in messages from generation settings in inferenceConfig. In the recording, the selected model is from Anthropic; the common call structure is what lets the developer experiment with other supported models by changing the model selection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Declare the functions a model can request
Converse also supports function calling for models that provide that capability. The setup described here is to define a list of tools representing application functions, then pass that list with the Converse call. A common invocation interface does not make tool support universal across models. The tool list describes the functions available for use; the next demonstration makes that connection concrete with code-backed actions in an agent.
For more detailed examples across languages, Barth points to the generative AI space on community.aws and names Dennis Traub as a tutorial author.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give an agent instructions and code-backed actions
The fourth step is application integration, illustrated through a Minecraft agent. Chambers opens an agent configured in the AWS console, describing the agent workflow as fully serverless. In the agent builder, the selected model is Claude 3 Haiku, and the instructions tell the Minecraft bot how to play. Those instructions are operational prompts, not notes for the developer.
The agent also needs actions it can perform. Chambers has organized these into Minecraft actions and an experimental action group. Inside the actions group, tools such as jump and dig connect to code. Their descriptions explain their intended use, making the tool descriptions another part of prompt engineering. Required parameters give the agent information it must obtain before using an action.
Two more actions prepare the central demonstration:
| Action | Role in the workflow |
|---|---|
| Get a player’s location | Obtain the destination associated with a player |
| Move to a location | Move the bot toward that destination |
These are small, reusable capabilities. Together, they let the agent address a request whose wording does not correspond to a dedicated action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose actions to answer “Come to me”
Inside Minecraft, Chambers first changes nighttime to daytime so the movement will be visible. Rocky, the Bedrock-powered bot, stands in the game world and can receive messages through chat. Chambers backs away to make the request more interesting. There is no dedicated tool for coming to the player.
He types “Come to me.” The described workflow connects the chat request to the available actions:
- Identify which player sent the request.
- Use that player’s name to obtain their location.
- Use the destination to move Rocky from its current position toward the player.
Chambers describes the final step as mapping a path to him. Rocky reaches the player, and the game chat confirms arrival. The useful result is composition of existing tools into a requested behavior, without adding a special-purpose come-to-player action.
Barth points viewers to further demonstrations at the AWS booth and in the agents track, and says the project code is available on GitHub for people who want to explore the integration themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep learning with other builders
The fifth step is continued learning through community participation. As models, tools, and application patterns change, workshops and conversations with other builders become part of staying current. Barth closes by announcing plans to turn AWS’s San Francisco Loft into an AI engineering hub, with workshops, events, and meetups.
The invitation includes shaping that program: suggest topics through a survey, volunteer to speak, or offer to host a meetup. At the time of the talk, an Anthropic happy hour at the Loft was near or at capacity, and additional events were planned for the following weeks and months. The closing opportunity is to participate in the learning community as a contributor as well as an attendee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- The Rise of the AI EngineerArticle
Swyx’s essay introducing the AI engineer role and application development around foundation models.
The May 2024 announcement explains unified model invocation, conversation history and supported function calling.
AWS’s June 20, 2024 announcement documents the model release discussed in the talk.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everyone. I'm so excited to be part of this conference and share with you five practical steps from software developer to AI engineer.
- 0:26
And if anyone is wondering here, this avatar on the slide, this is what happens if you ask AI to make you look a little bit more agentic. [laughs]
- 0:36
All right, let's get started. So I'm pretty sure everyone is familiar with this image here and the post from swyx that defines the new role of the AI engineer.
- 0:47
And as you've experienced, uh, probably daily in your jobs, you don't need to be a full ML researcher anymore or data scientist. Things that took month or years before to get AI projects into production is now able to be in just a couple of API calls.
- 1:06
Super exciting. But still, if you're working with AI, it still makes sense to understand the basics of the technology,
- 1:18
and this involves a couple of things, right? So you have to understand at a basic level how foundation models work, why they're sometimes producing output that you don't expect in your application code, right?
- 1:32
You have to understand how you can customize the models, how you can, you know, for example, sometimes fine-tune models to adapt them to your specific use cases and data sets, how to include functions in your application code to give them access to additional systems.
- 1:50
The good news is, if you're just starting on this journey to becoming an AI engineer, there's plenty of resources now these days available to you to learn. And I wanted to call out one specific course here, which is called Generative AI with Large Language Models.
- 2:07
A few colleagues and mine, we actually collaborated with Andrew Ng and the team at DeepLearning.AI to put this course together and help you understand the fundamentals of generative AI to help you build real-world applications.
- 2:22
If you're curious, it's available on DeepLearning.AI and on Coursera.
- 2:29
Now, the second step in this journey is to start get hands-on with the AI developer tools to help you increase your productivity.
- 2:42
And I think we're all seeing this quote here, and we experienced it in our daily jobs, that how we do work, how we develop applications, has changed a lot.
- 2:55
These days, we can literally use natural language inputs to interact with applications, and really, English has become one of the most, um, popular and hottest programming languages, I think, um, we can see this happening.
- 3:12
For example, you can go these days from English to code by asking AI to, for example, rewrite a README file.
- 3:24
We can also do code to English, for example, asking AI to document functions in our code.
- 3:35
But this is not all. If we look at the software development life cycle, I think many of us can agree that the majority of time we usually spend not writing valuable code, but all the other things around it.
- 3:48
So sometimes up to seventy percent of unvaluable tasks, which is writing boilerplate code, writing documentation, trying to maintain old code bases, right? And sometimes we only have like a fraction of the time, maybe thirty percent, that we're spending on actually what, you know, creates joy and kind of the creative tasks in software development.
- 4:12
And this is what led us at AWS, this inspired us to create Amazon Q. Amazon Q is a generative AI-powered assistant specifically developed for software development,
- 4:27
and this is much more than just a coding assistant. Q Developer actually uses agents to perform much more complex tasks and help you automate those, for example, feature development and also code transformation.
- 4:44
Think about working with old Java-based code bases that you need to migrate maybe to a newer Java version.
- 4:53
And to show you how this works, I asked my colleague, Mike Chambers, to put together a quick demo. Let's have a look.
- 5:02
With Amazon Q installed inside of my IDE, I can go to New Tab, and I can start a conversation with Amazon Q Developer. And I can do the kinds of things that maybe you'd expect, such as, "How can I create a serverless application," and, "How do I get started?"
- 5:19
And the chat session brings back a list of instructions of what I should do, starting off by installing AWS SAM CLI, how to do that, where to get that from, and how to step through the creation of a project.
- 5:34
Now, if I've done that, then Serverless SAM, for example, might actually come back with some generated code, and here is that code. Maybe I don't quite know what this code does, so I can right-click on the code and send it to Amazon Q, asking Amazon Q to explain.
- 5:52
And the code then will go into a prompt along with explain and generate an answer. And this is great for code that's been generated for us. But also imagine code for legacy systems, something that was worked on years ago by somebody else, where you can get Amazon Q to help explain it.
- 6:11
We can also get Amazon Q to generate code. Now, this is, again, probably the kind of thing you'd expect. I can put in a comment line inside of my code.
- 6:20
In this case, I want to create an input checking function. I'm going to give it some more definition here that I actually want it to trim any string that's being sent into this function.
- 6:31
And yes, Amazon Q can generate this small function. Well, that's great, but what about if I've got more code that I need to have generated? Well, I can go to the chat and put in /dev, and I can put in a much more comprehensive description of something that I would like.
- 6:49
In this particular case, I'm gonna ask for it to write a function to search by category in DynamoDB with a bunch of details about the way that I want the output to be formatted.
- 7:02
So this is much more than just a single or a few lines of code. And in this particular case, what's gonna happen is it'll come back again with a step-by-step list of what's required.
- 7:14
So I need to add in template.yaml. It's recommending that I create searchByCategory.mjs and many more things. But this isn't just a big shopping list of things that I need to do.
- 7:29
This is actually a plan, and it's a plan that Amazon Q can actually follow for us. So it generates some code as a change set, something that we can look at the difference between our current code and what it suggests.
- 7:44
And if we like that, we can actually click on the Insert code button, and it will add all of that code into our project, way more than just a couple of lines.
- 7:55
So Amazon Q Developer is much more than just code completion.
- 8:03
All right. If you're curious to learn more about Amazon Q, Amazon Q Developer, we have a couple of more sessions throughout this day, so make sure you're checking, um, those expo sessions, and we also do have a session at our AWS booth here.
- 8:18
You can also visit our Amazon Q Developer Center for much more examples what you can do with it.
- 8:26
All right, let's come to step three, and this is where the fun starts. Start prototyping and building with AI.
- 8:35
And the fun includes a couple of steps, right? Everyone developing with AI knows this. It starts all with defining a use case, and then really you're on this road trying to choose from different models.
- 8:47
You're trying to, you know, customize them to your use case, decide whether it's prompt engineering, whether you do s- RAG, whether you need to do a little bit of fine-tuning there with your data.
- 8:58
And of course, across the whole development workflow, you have to incorporate responsible AI policies, making sure data is private and secure, and also implementing guardrails into your application. And then when you're integrated, another fun part, obviously working with the agents, what we're hearing a lot here throughout this conference, and the fun topic of, you know, how to
- 9:21
keep them up to date, GenAI Ops, I think there's a lot of terms for that, MFM Ops, LLM Ops. So really kind of, um, a lot of things to consider here.
- 9:31
I wanna dive in briefly into the topic of models to choose, and this is really an important topic. When you're evaluating models, you have to really evaluate them thoroughly because most likely there's not just gonna be one size fits all for you.
- 9:49
In fact, if you look at all your use cases you want to implement, there's likely no one model to rule them all.
- 10:00
And this is why we developed Amazon Bedrock. Bedrock is a fully managed service that gives you access to a wide range of leading foundation models that you can start experimenting with, implementing into your applications.
- 10:17
It also integrates the tooling you need to customize your model, whether it's fine-tuning, also to include RAG workflows, to build agents, and of course, everything in a secure environment where you are in full control of your data.
- 10:36
And speaking of choice, just to give you a quick overview, as of today, this is the selection of models you can choose from. We're working with leading companies such as AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and we also offer our own Amazon Titan models for you to choose from.
- 10:59
And I'm super excited just to call this out. Last week, together with Anthropic's launch, we integrated Claude 3.5 Sonnet on Amazon Bedrock as well. So you can also, since last week, use this model.
- 11:13
Super exciting. Now, with choice also comes responsibility, right? And we continuously innovate and trying to make it easier for you to build applications across the different model types.
- 11:29
And just a few weeks ago, we introduced a new unified Converse API in Amazon Bedrock. What does this do? The unified Converse API helps you with a new unified method structured invocation, meaning you can use the same parameters and bodies regardless of which mo- model you choose.
- 11:52
And we are, on the platform side, we're handling this translation if parameters are called different for the different models, handling the system, user, assistant prompts for you, and also giving you a consistent output format.
- 12:05
And as well having native function calling support in here. But let me show you how this looks in code. So here's the Python example that shows how you can use the new API This is Python, so we're starting by just integrating the Python SDK client here.
- 12:22
And then you can define this list of messages. And here's, for example, where you put in your user message prompts. You can put in system prompts as well. And then this message list, you can just pass in this single API call using the Converse API here.
- 12:40
In the model ID, you can choose which model you wanna test. Here I'm using an Anthropic model. And then pass the messages and also the inference parameters. And again, in this API, all those parameters are standardized, and we're gonna make the work behind, um, the covers to convert this to the specific format that the model is expecting.
- 13:01
So you have an easy way to work across different models.
- 13:06
Similarly here for n- function calling, we do have the support built in that with the models that support it. So how we implement this is by defining a tool list.
- 13:17
So tool here equivalent to, to the functions you wanna give access to. And then when you're doing the Converse API call, you can pass this list of tools.
- 13:30
All right. If you wanna find out more about Converse API, here's the link to our generative AI space on community.aws, which has a lot more co- tutorials, code examples, not just for Python, but across different languages as well.
- 13:45
So check it out. The author here, Dennis Traub, is also somewhere here in the audience here this week, so if you wanna connect with him, talk about different code examples and how to use the API, um, feel free to reach out.
- 13:58
All right. Now, let's integrate AI into our applications. And this can be a whole session in its own, but I wanna focus on one of the hottest topics right now that we're discussing during the conference, which is of course agents.
- 14:13
And I have one more demo here, and I asked my colleague Mike last time to put together an exciting demo to show you what you can do with agents.
- 14:24
Mike. And we need sound.
- 14:28
To be able to create agentic workflows right inside of the AWS console and inside of the service. It works fully serverless, and I've used it to create an agent that plays Minecraft.
- 14:40
Let me show you [laughs] how I did it. If we jump into the AWS console, dro- go down the menu on the left-hand side to agents, um, you can see the agents screen here, and I can open up my agent, my Minecraft agent.
- 14:52
Now, if I just go into agent builder, uh, and just expand the screen out a little bit, you get to see some of the parameters that I used to create this agent.
- 15:01
So you can see the large language model I used, in this case, Claude 3 Haiku. And you can also see this, the instructions for the agent. Now, this is not some notes for myself.
- 15:10
This is actually prompt engineering that we're doing to explain how we want the agent, in this case, the Minecraft bot, to play the game. And then we also have to add some tools in, right?
- 15:21
Some Minecraft tools. So we do that through actions and inside of action groups. So I've got a couple of different action groups. We've got Minecraft actions and Minecraft experimental.
- 15:30
Let's have a look at actions. And inside of here, we can see some really simple things, some actions that the bot will be able to do. And these are all linked up to code.
- 15:40
So we've got the action to jump. We've got the action to dig. And you can see the description here for action to dig. It's got some instructions. Again, this is prompt engineering.
- 15:50
And then we've got some parameters that we can select, c- collect. In fact, we require these parameters. So the bot needs to get these for us. Um, if I scroll down a little further, there's a couple of really simple actions in here.
- 16:02
Action to get a player location and action to move to a location. And I'm gonna show you those in action because the bot can actually problem solve and reason its way to be able to use these tools to solve simple problems.
- 16:18
Let's jump into the game. Um, and so it is nighttime. So let's set it to be the daytime, um, so that we can see what's going on. So set time to day.
- 16:28
Okay. And there in the middle of the screen, you can see Rocky. Rocky is the Bedrock agent running inside of the game. And we can talk to it, and we can have a chat session, but what about if we want it to come to us?
- 16:40
Now, there is no tool to come to us. So if I just... I'm just gonna back up a little bit further, make it a little bit more of a challenge.
- 16:48
And I'm gonna say, "Come to me" in chat. And what's gonna happen now is that the agent's going to reason through a whole set of actions. It's going to look to see who requested that.
- 16:58
It's then gonna take that name, and that's my name, and it's going to find the location of that player. And then it's going to map a path from where it currently was to me.
- 17:08
All of those things happened all in that blink of an eye, and there's agentic workflows making all of that happen. This is super exciting. I'm discovering new things that this bot can do every day.
- 17:19
Um, but with that, it's back to you. [laughs]
- 17:23
All right. Thank you, Mike. If you're curious to know how we did this, check out our booth session. We, we're running the demo there as well. And we have another session in the agents track later today, so make sure you're popping in there if you wanna know more.
- 17:38
And of course, you can find the project code for this on GitHub. So if you wanna play, play it on your own in how you can integrate agents into a fun thing, check out this project.
- 17:50
All right. We're f- almost there. So the last step I really wanna call out is stay up-to-date. There's so much happening in this space, as you all know. And a really good way to do that is to engage with the community.
- 18:04
Speaking of community, I have one last announcement to make, and I'm super excited to announce that we're transforming our AWS [REDACTED:location] here in San Francisco into the AI engineering hub for the community.
- 18:20
So we're super excited to host workshops, events, and meetups there.
- 18:26
If you wanna suggest a couple of topics you're most interested in to make those events most valuable to you, fill out this quick survey here. Also, if you're interested in speaking or hosting a meetup yourself, you can let us know.
- 18:39
And also, we do have another event tonight, which I think we're reaching capacity or just half reach capacity, but we do have a happy hour with Anthropic tonight at the [REDACTED:location].
- 18:52
In case you didn't make it anymore and we're at capacity, um, don't worry. We're working on putting together much more events like this in the upcoming weeks and month.
- 19:02
So keep an eye out for those. And with that, I'm coming to the end of our presentation. This wraps it up, the five practical steps to become an AI engineer.
- 19:13
And let's innovate together. And I'm looking forward, and I'm excited to see what you build with AI. Thanks so much. Make sure you're checking out the rest of the sessions here, and also pop by our booth outside.
- 19:27
Thanks so much. [upbeat music]