AI Engineer World's Fair 2024
Claude plays Minecraft: Introducing a real world serverless AI to a virtual world
Read the talk
Claude plays Minecraft: from chat requests to blocks in the world
Derek Bingham’s Minecraft bot Rocky connects conversational requests to game actions, using managed agent orchestration while a stateful client handles the world.
From a talk by Derek Bingham
Before you start: Basic familiarity with LLM tool calling and client–server applications will help; no Minecraft programming experience is required.
Giving Minecraft chat a body
How do you turn a message in Minecraft chat into an agent that actually does something in the game? Derek Bingham begins with the familiar agent loop: accept a request, use an LLM to select tools, run one or more rounds of tool use, and return a response. Minecraft makes that loop visible. A successful request can change where a bot stands or which blocks remain in the ground.
The game already supplies the input interface: its chat function lets players communicate with the world and with one another. It also supplies a rich action space—building, digging, farming, and interacting with animals. Put the agent inside a bot, connect those actions as tools, and a chat message becomes a request for behavior. Both the request and the resulting behavior can be open-ended, which is part of the risk of demonstrating an LLM live.
After one or more orchestration steps, the agent returns a response to Minecraft. Bingham names the bot Rocky, after Bedrock, and gives it tools including jumping, moving to a position, and locating a player. These concrete operations are the bridge between a conversational request and an observable result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a weather question to a hole in the ground
Bingham introduces Rocky as an open-source demonstration that attendees can try with a Minecraft client, then plays a recording made at the AWS booth. It is raining in the game. Asked about the weather, Rocky addresses the player by name, reports the local rain, and suggests taking shelter. The response connects friendly conversational behavior to the state of the world.
The requests then move from observation to action. Rocky jumps on command. Bingham hides behind a building Rocky previously constructed and asks the bot to find him; Rocky announces its approach and comes around to the player. Next, he asks Rocky to find a pig. The Minecraft plugin supplies entity locations, so Rocky can report where a pig is before a subsequent request sends it to that location. Finding an entity and navigating to it are distinct operations in this exchange.
Once Rocky reaches the pig, Bingham asks it to hit the animal, and it does. He notes that repeated requests to hit pigs have been a conspicuous feature of conference demonstrations—human behavior proving at least as interesting as the model’s. He then calls Rocky back and asks for a two-by-two hole. Unlike a bare request to dig, this instruction supplies dimensions that must become action parameters. Rocky digs the hole, then later digs its way out to reach the player. Bingham reports that the team had not expected that escape behavior; the demonstration does not isolate which part of the system produced it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping game state close to the bot
Rocky began as a project for a serverless conference, with three constraints: run in a managed environment, use serverless infrastructure, and make an engaging demonstration. Minecraft and Mineflayer, the bot framework, could run alongside one another in a container. Mineflayer retained the bot’s state, so Bingham kept it out of Lambda. That is an architectural choice about a continuously running game client, rather than a blanket claim that Lambda can retain nothing: AWS documents execution-environment and connection reuse, but reuse is not the same contract as a persistent bot process.
The first orchestration layer used LangChain on Lambda. As the team added actions and tools, Bingham found it increasingly complex, particularly given his limited Python experience. That prototype used Cohere models through Amazon SageMaker. The next version moved orchestration to Agents for Amazon Bedrock and kept Minecraft and Mineflayer on Amazon ECS.
| Version | Minecraft and Mineflayer | Agent and model layer |
|---|---|---|
| Initial prototype | Container | LangChain on Lambda; Cohere through SageMaker |
| Revised hosted design | Amazon ECS | Agents for Amazon Bedrock |
| Onstage demonstration | Bingham’s laptop | Remote Agents for Amazon Bedrock |
The live demonstration uses the same separation between the game client and managed orchestration, but runs the game components locally. Its remote agent calls therefore depend on the venue’s internet connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Managed orchestration, local execution
Amazon Bedrock provides a common API over models from Amazon, Anthropic, Cohere, and other providers, including the Llama family. Rocky’s revised implementation uses Claude, with Bingham choosing the Haiku variant for speed. He gives this as a selection rationale, without reporting a measured latency comparison.
Around that model interface, Bedrock offers agents, knowledge bases for retrieval-augmented generation, guardrails, and evaluations. For Rocky, the relevant managed capabilities include help with prompt creation and orchestration across multiple tasks. Bingham also describes inspecting the agent’s orchestration trace in the console or logs, calling it the agent’s chain of thought. These are the service’s exposed traces, useful for following its actions and responses.
Return control connects the managed agent to the stateful game client. Bingham singles it out as essential to the demonstration. With return control, an action selection comes back to the application for execution. Bedrock can decide what operation to request while the client that maintains the Minecraft connection carries it out.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Defining Rocky’s personality and action parameters
In the software walkthrough, Bingham shows Mineflayer and the project’s source, then opens the Bedrock console. The console exposes model access and the Minecraft agent, but manual configuration is not required: he says the repository includes CloudFormation and CDK definitions. This is the historical Bedrock Agents implementation; the current AWS invocation reference calls the service Agents Classic and says it is closed to new customers, while existing customers may continue using it.
The selected model is Claude 3 Haiku. Its system prompt describes Rocky as a playful, friendly, creative Minecraft agent whose goal is to entertain players and collaborate with them. Bingham explains that the team developed this prompt over time. Personality is configured alongside the task, so Rocky’s chat responses can remain consistent while its tools perform quite different operations.
All the displayed actions use return control. More precisely, the application receives the selected action’s inputs and an invocation ID—not the output of an action already completed in Minecraft. It executes the action and can send the result back through sessionState in another InvokeAgent request. That distinction matters: a request to dig is still pending until the game client performs it.
The dig action exposes depth and width. The earlier two-by-two request supplies those values explicitly; for a small hole, Bingham says the model infers one by one. The boundary between language interpretation and execution can therefore be expressed as a small parameter contract. In JavaScript, a validation helper for those demonstrated dimensions could look like this:
javascript
function readDigDimensions(parameters) {
const values = Object.fromEntries(
parameters.map(({ name, value }) => [name, value])
);
const depth = Number(values.depth);
const width = Number(values.width);
if (!Number.isInteger(depth) || depth < 1 ||
!Number.isInteger(width) || width < 1) {
throw new Error('Dig depth and width must be positive integers');
}
return { depth, width };
}
const requestedHole = readDigDimensions([
{ name: 'depth', value: '2' },
{ name: 'width', value: '2' }
]);
requestedHole holds the requested dimensions; it does not itself dig anything. The model interprets the player’s language, while the client needs usable parameters before it changes the world. Jumping and checking whether it is raining are separate action definitions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Asking for a double-decker couch
A separate action set, Minecraft experimental, contains a single build action. Construction is harder than jumping or reporting weather because the result must occupy a three-dimensional space. The action returns control to the client, where the structure request is used in a construction prompt.
Bingham starts the stopped client, Rocky spawns, and the bot joins the running Minecraft game. He sets the game time to noon so the audience can see, then checks the basic loop by asking Rocky to come to him and dig a small hole. With those actions working, he asks the audience what Rocky should build.
The first suggestion is the Coliseum. Bingham passes it over with a joke about spelling and accepts a double-decker couch instead. He mentions having built a rocket ship the previous day, calls Rocky out of the hole, and gives the bot space. The live request is to build one double-decker couch.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Making a structure description executable
Once the request is sent, Bingham opens the builder prompt. It casts Claude as an expert Minecraft builder created by Anthropic, takes the structure description as input, and asks for valid JSON. The prompt supplies the representation that Rocky’s construction code consumes, giving the model a concrete output shape rather than asking for a prose explanation of how to build a couch.
The prompt also constrains how blocks are placed relative to one another and which block types are allowed. Bingham says that without these rules, the generated structures become nonsense. Valid JSON is only the first constraint; the contents must also describe a buildable arrangement. The structure description determines what to make, while the output format and placement rules determine how that intention reaches the game.
Bingham describes the generated positions as X and Y coordinates, although Minecraft construction is three-dimensional. The Mineflayer API uses x, y, and z positions; the walkthrough does not establish Rocky’s complete JSON schema. Its generated JSON is best understood as input to the application’s construction path, rather than a verified native Mineflayer structure format. The client begins building from that generated arrangement.
Back in Minecraft, Rocky has completed the double-decker couch. Bingham moves to sit on it and emphasizes that the shape is Claude’s interpretation of the request in three-dimensional space. The result is not a fixed couch design selected from a menu: the description has become a constrained representation, then physical blocks in the game. That completed structure makes the whole agent loop visible, from a player’s words to a changed world.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Continuing with Rocky
The demonstration closes with a QR code displayed inside Minecraft. Bingham asks attendees to complete the session feedback form, offering AWS credit codes and a GitHub link to those who do, and invites further questions at the AWS booth. The invitation follows a working example: an agent choosing actions remotely, a client maintaining its connection to the world, and a generated structure made visible one block at a time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
JavaScript framework for creating Minecraft bots, with setup instructions and examples.
Explains how an application receives an agent's action request, executes it, and returns the result.
Further reading
Documents bot actions, three-dimensional positions, digging, and block placement.
Updates since the talk
- InvokeAgent API and current availabilityDocumentation
Current invocation reference, including a notice that Bedrock Agents Classic is no longer open to new customers.
Read the complete timestamped transcript
- 0:00
[upbeat music] Today I'm going to, um, show you how I built an agent to play Minecraft, right?
- 0:21
So this talk, in a nutshell, if I could, um, sum it up, is all about agentic workflow. So I'm sure some of you here have, have built agents. You're in this track.
- 0:32
Maybe some of you have not. Um, so I'm gonna start with a bit of a, bit of a, um, level set on agent workflow and agentic workflow, and then I'll show you how that maps to the agent, uh, we built with, with Minecraft.
- 0:48
So this flow diagram is probably familiar to some of you, where an agent will take inputs, uh, like chat or some ins- ins-- unstructured data. Uh, it'll consume that, and then it will use a set of tools, one of many tools, to satisfy the request, uh, from the chat, so it can fulfill its destiny and its, um,
- 1:11
its actions. And also, it uses an LLM, and it may then orchestrate one or many times through the LLM, the tools, and then return, uh, a response.
- 1:23
Pretty well understood now, but very cool. So if we take that into a Minecraft world, where does the chat come from? Well, if anybody, anybody here played Minecraft? A few people, a few people.
- 1:38
Great. So if you've played Minecraft, you'll know there's a chat function where you can interact, uh, with the world and other people in the world. And so that's where the chat comes for this agent.
- 1:51
The tools in Minecraft are many, and that's why we built this. In Minecraft, you can do many things. You can build. You can dig. You can farm. You can slay a pig if you so desire.
- 2:07
Um, but you can do a few things.
- 2:12
And so the agent itself becomes the bot. So we tell it via chat what we want it to do. It uses the tools and hopefully fulfills our requests. Now, remembering that our requests are probably indeterminant, and the output is also indeterminant.
- 2:31
So today I'm gonna live demo, uh, what you should never do. So they say you should never live demo with kids, animals, and an LLM, so I'm gonna do that today. [laughing]
- 2:42
The LLM itself, of course, is magic. It is. It's magic.
- 2:49
And then the responses, the agent will, um, do one or many, um, thoughts and then return the response to Minecraft. So that's, that's the flow in a nutshell.
- 3:01
And of course, we have our friendly bot. And so our bot's name i- is Rocky, uh, in short for, uh, Bedrock. And, um, we'll see what Rocky-- if Rocky behaves today.
- 3:14
And the tools we're going to use with Rocky and the actions that those tools will, will, um, u- uh, jump on is jump, move to position, locate a player, and a few more, more actions that we, we've built.
- 3:28
And at this stage, I'll also mention that this demo that I'm going to show is open source, uh, and you can all have a play with it at the end.
- 3:36
All you need is a Minecraft client. So this is Rocky. I'm gonna give you a quick recording that I did yesterday at the booth. We're at the AWS booth if you wanna come and chat to me.
- 3:48
I am, I am not scary. Uh, I am from Australia, but I'm not scary. Um, so Rocky does a number of things. So this is what Rocky will look like.
- 3:56
At the minute, it's raining in Rocky land. Rocky is a very friendly bot. And so you can ask Rocky questions in the chat. I hope you can all see that.
- 4:05
So what's the weather? So, uh, Rocky being friendly will know my name, uh, and then also tell you what the weath-weather's like. So it's raining in your area. You might want to take some shelter.
- 4:18
Rocky can also do some actions. So this is how we send the actions to the agent via chat. So Rocky jumps for us.
- 4:26
Also, Rocky can do other things, so I'm gonna hide behind this Rocky-constructed building, and I'm going to ask Rocky, can he come and find me? Or can they come and find me, actually.
- 4:37
Rocky is neither male nor female. Um, Rocky says, "On my way," and there, there comes Rocky.
- 4:46
And Rocky's found us. But what else can Rocky do? So what else did I do here? [laughs] Oh, it can find things in the world. So Rocky here, I've asked it to find a pig.
- 4:58
So Rocky knows, based on, um, the plugin that we're using for Minecraft, where everything is, and it says, "Yep, I found a pig. This is the location." And I then ask it, "Go find-- Go, go to that pig."
- 5:10
So off Rocky runs to the pig and, uh, finds the pig. So we've made-- we've demonstrated this at a lot of conferences, and the biggest thing we've seen is people try to hit the pig.
- 5:23
So I asked it to hit the pig. And, uh, there it goes. Rocky hits the pig. Uh, you can do that multiple times, and lots of people I've observed do.
- 5:31
Don't know why, but hey, human behavior is even more fascinating than the LLMs.
- 5:37
Um, so now I am gonna get Rocky to come here, and I'll probably finish with the dig action. So I'll ask Rocky to dig a hole. Notice, um, when I say dig a hole, there's a parameter that I r- that I add.
- 5:52
So I think in this instance I... Yeah, a two-by-two hole. So this is a parameter that I'm sending to, to the action. I'll show you how all that works when I do a bit of a deep dive into the, into the, um, the back end.
- 6:04
So Rocky's, um, digging a hole.
- 6:07
And also, Rocky can come find us out of that hole and dig their way out of the hole. Uh, this is behavior that we didn't expect, and it just, just works, which is, um, really fascinating.
- 6:21
So that's, that's a bit of a three-minute, uh, demo of Rocky. I'm going to live demo Rocky once I go through how I built it. So the architecture. So we built this for a serverless conference.
- 6:32
So w- our, um, constraints were it needs to run in a managed environment, in a service environment, uh, and be cool. So we started off with running it in a container.
- 6:44
So Minecraft you can run on, in a container, and you can also run Mineflayer, which is the bot framework that works with Minecraft. And those two run side by side nicely on the container.
- 6:56
We can't run Mineflayer on Lambda, which is serverless, because we need state, right? So it's, the state is stored in Mineflayer.
- 7:05
We started off with LangChain on La- um, on Lambda to build this. Like most people, when they do start building agents, they probably start with LangChain. Uh, I'm not a big Python developer.
- 7:15
Uh, I am an engineer, but not really with Python. Um, but we, we got it working to a state. But then as we added complexity and more and more actions and, and tools for Rocky to use, uh, it, it got really, really, um, complex.
- 7:28
And then we also used Amazon SageMaker to host the LLM, which in this c- in this case, uh, we were using the Cohere LLMs.
- 7:37
And then we decided, okay, so that's not serverless enough. Uh, let's use, uh, Agents for Amazon Bedrock, keep the Minecraft server and client, which is, um, Mineflayer, on, uh, Amazon ECS, and build that agent up on Amazon Bedrock for...
- 7:57
And then for architecture, so what we're doing and what I'm demonstrating today, um, it is, it's running on my local machine, so both Minecraft and the Mineflayer client, and then it's into, uh, calling out to Agents for Amazon Bedrock.
- 8:12
And may the internet always be favorable over the next couple of minutes.
- 8:18
So if you haven't come across Agents for Amazon Bedrock, let me do a quick, uh, overview. Again, AWS booth is there. Come talk to me. I'll go a deep dive for you.
- 8:27
But in a nutshell, uh, Amazon Bedrock is all about façading, uh, one or more models. So what it does is it produces a common API that then you can use, uh, as an engineer to build out your application and add, add features.
- 8:42
So we host, as well as Amazon models, we host Anthropic models, we host Cohere models, as I mentioned, Llama models, um, and a whole host of other models. So when building out Rocky now using, uh, Agents for Amazon Bedrock, you'll see that I used the Claude, uh, models.
- 9:01
And I'll go through that as well, why I done, did that. Cla- in, in particular, Claude Haiku, because it's, it's fast.
- 9:09
And so as well as hosting models, not hosting models, but providing a façade into models, um, everything, you can also build, uh, additions to that. And one of those additions is agents, as well as knowledge bases where you can, uh, do RAG, and also guardrails and evals.
- 9:26
So all of these things are baked into Amazon, uh, Bedrock, and it's... Think of it more as a sort of a, a managed agentic workflow, right? So you can m-manage, um, RAG and manage a-agents as well.
- 9:40
So it, it's all in one spot. So it also helps you with prompt creation. It obviously, as it's an agent, orchestrates multiple tasks, and also allows you to trace through the chain of thought of the agent.
- 9:52
So you can either do that in the console or it'll spit it out to logs.
- 9:56
And of course, it also, and what was very key for this demo, it has return of control.
- 10:02
Okay, so that's the slides. Let's, uh, let's jump into what we've got.
- 10:09
So let me see if I can just instead of doing that,
- 10:14
uh... The mirror. No, not that. Oh, dear.
- 10:30
Stop mirroring. Okay. There's always... So main display.
- 10:44
Extend display. Main display. I want a mirror, and it's not, it's not letting me. Let me see.
- 10:54
Apologies. You got this. You got this. I got this. Let's do this re-refresh right here.
- 11:01
Refresh right here. This should do it. And mirror.
- 11:09
Mirror. There we go. Yay. Okay. So, okay, so this is, that's Mineflayer. So it's op- all open source. Let's first of all start with the good stuff. So open source, uh, Mineflayer you can use, and then this is, uh, the, um, what we've built.
- 11:29
Now I'll share this at the end, QR code, everything. Don't worry about it. So this is Amazon Bedrock. So this is agents, and this is the, um, the console page.
- 11:38
So what it'll do, um, you can go down here, uh, into agents. If you haven't seen Bedrock before, it'll show you the model access and what models you've got access to.
- 11:48
But I'm really talking about agents, which is down here on the left. So here I've got a Minecraft agent. Apologies if you can't see this too well. Let me see if I can make it a little bit better.
- 11:57
Um, you've got Minecraft, the Minecraft agent. So I've defined this in here. Now, for all the real engineers in the room, you don't have to build it via ClickOps.
- 12:06
So you can build it all as infrastructure as code, and if you check out the GitHub repo, you will see it all in CloudFormation and CDK. Um, so once I've gone into the agent, you can see that here I can select my model.
- 12:17
So at the minute, this agent is using Claude, Claude 3 Haiku, and here is the prompt. This is the system prompt for that agent. "You're a playful, friendly, and creative Minecraft agent called Rocky.
- 12:28
Uh, and your goal is to entertain players and collaborate with them in a fun gaming experience." And it goes on. So this prompt we, uh, built over, over time.
- 12:35
There's a bit of prompt engineering going on in this demo. Um, and then if I go into the actions, you can see all of the actions. So you notice all of the actions I've defined all have return control 'cause we, we want them to return, uh, the, the, their output.
- 12:52
So we've got jump, uh, we've got other actions. Dig. Dig, you'll see this is where the parameters come in. So I've got a depth and a width. So if you remember when I specified a hole, I said two by two.
- 13:04
Also, if I say small hole, it will go as a one by one hole. So it'll infer that, uh, using the model.
- 13:11
Uh, also, there's the action, is it raining? So all of these actions are defined in here.
- 13:17
And also, I've got another action set that I'm gonna share with you and gonna demo. It's called...
- 13:26
Do-do-do-do-do. It's called Minecraft experimental, right? And what this does, this has one action called build, which is a very complex thing to do in Minecraft, uh, believe it or not, because it's a 3D space.
- 13:40
And so the action for build, all I do is build a structure and, um, that returns back to the client, um, and I'll show you how that prompt's actually created, um, when I do the build.
- 13:52
So that's, that's how it's built, and the actual
- 13:55
source code itself is here, and the client is stopped, so I'm gonna...
- 14:01
So this is Rocky itself. So I'm gonna start Rocky.
- 14:06
Gonna start and bo-bot has spawned. Gonna go over to Minecraft we've got running here. Back to game. Rocky has joined the game. There's Rocky. Let me just go set
- 14:21
time. Time. It's time set. Time, set, uh, noon, so you can see it better.
- 14:29
Okay. Okay, so there's Rocky, so you can... Let's just test that Rocky's working. So T, please come here.
- 14:41
Uh, Rocky will make the first request, and there we go. So Rocky, Rocky is working for us. Um, so let's make sure that Rocky can do something. So we go, "Rocky, please, please dig a small, small
- 14:58
hole." Okay, so it'll-- Rocky will dig a small hole. So now I am going to use the experimental feature, which is build. What do you want Rocky to build?
- 15:10
Coliseum.
- 15:11
A, a, a what?
- 15:12
Coliseum.
- 15:13
A coloss- the Coliseum? Okay. I couldn't even spell that. [laughs] Um, some-something that I can spell, please. [laughs]
- 15:25
A double-decker couch.
- 15:25
A double-decker couch. Okay. The, uh, I like that. Um,
- 15:30
please come here. So yesterday I, I built, um, a rocket ship, which was quite interesting, but a double-decker couch. So let's, let's, um, get Rocky out of the hole.
- 15:39
So let's give it some space. Okay, Rocky.
- 15:45
Please build... Can I just say couch? [laughs] Double... Let's try it. [laughs]
- 15:53
Uh, never do live. Double-decker couch. Please build double-decker couch. A one, just one.
- 16:00
I'm just gonna, I'm just gonna make it as... So what happens, Rocky, then, that prompt goes off. And so if I look at the code, which I've lost...
- 16:10
Yeah. So there's the prompt. So what we've done, we've said to Rocky, "You're a Claude, an expert Minecraft builder creating by, by Ro- Anthropic when given a structured description," which will be the input, uh, "then output valid JSON."
- 16:24
So this JSON is how Mineflayer builds objects. So you give that as part of the prompt so that the model will understand how to build Minecraft objects. It, uh, then you'll go strictly adhere to the following rules because if you, we didn't do that, it goes bananas, uh, and builds just nonsense.
- 16:41
Uh, and so all blocks are placed to each other. These are the blocks it can use, and it responds with the, what it thinks a couch looks like. So this is the X, Y coordinates.
- 16:50
And so it's started the build. So let's see what Rocky's doing.
- 16:55
And there Rocky has built- [laughs] -a double-decker couch. Thank you very much. Yeah. [laughs] [clapping]
- 17:03
Um, I'm gonna sit on the couch. Um, so this obviously is the interpretation of what a double-decker couch looks like, uh, to Claude and in the 3D space, and then it's been interpreted into X, Y coordinates, and Rocky has built it.
- 17:21
So that, that is, that is Rocky. Obviously, we are here, uh, at the AWS booth all day. Could you please... I also put a QR code I promised before.
- 17:31
Let's fly to that. [laughs] Wee. So scan the QR code. Um, please fill in the session, um, feedback form. Also, if you fill it in, I'll give you AWS credit codes for everybody who fills it in and a link to the GitHub website.
- 17:47
So please do that. And, uh, hopefully you enjoyed the session, hope, hanging out with Rocky, and come ask me any questions at the booth when you can. Thank you very much.
- 17:56
Thank you. [upbeat music]