Claude plays Minecraft: Introducing a real world serverless AI to a virtual world

Talk transcript

1. 0:00

   [upbeat music] Today I'm going to, um, show you how I built an agent to play Minecraft, right?

2. 0:21

   So this talk, in a nutshell, if I could, um, sum it up, is all about agentic workflow. So I'm sure some of you here have, have built agents. You're in this track.

3. 0:32

   Maybe some of you have not. Um, so I'm gonna start with a bit of a, bit of a, um, level set on agent workflow and agentic workflow, and then I'll show you how that maps to the agent, uh, we built with, with Minecraft.

4. 0:48

   So this flow diagram is probably familiar to some of you, where an agent will take inputs, uh, like chat or some ins- ins-- unstructured data. Uh, it'll consume that, and then it will use a set of tools, one of many tools, to satisfy the request, uh, from the chat, so it can fulfill its destiny and its, um,

5. 1:11

   its actions. And also, it uses an LLM, and it may then orchestrate one or many times through the LLM, the tools, and then return, uh, a response.

6. 1:23

   Pretty well understood now, but very cool. So if we take that into a Minecraft world, where does the chat come from? Well, if anybody, anybody here played Minecraft? A few people, a few people.

7. 1:38

   Great. So if you've played Minecraft, you'll know there's a chat function where you can interact, uh, with the world and other people in the world. And so that's where the chat comes for this agent.

8. 1:51

   The tools in Minecraft are many, and that's why we built this. In Minecraft, you can do many things. You can build. You can dig. You can farm. You can slay a pig if you so desire.

9. 2:07

   Um, but you can do a few things.

10. 2:12

    And so the agent itself becomes the bot. So we tell it via chat what we want it to do. It uses the tools and hopefully fulfills our requests. Now, remembering that our requests are probably indeterminant, and the output is also indeterminant.

11. 2:31

    So today I'm gonna live demo, uh, what you should never do. So they say you should never live demo with kids, animals, and an LLM, so I'm gonna do that today. [laughing]

12. 2:42

    The LLM itself, of course, is magic. It is. It's magic.

13. 2:49

    And then the responses, the agent will, um, do one or many, um, thoughts and then return the response to Minecraft. So that's, that's the flow in a nutshell.

14. 3:01

    And of course, we have our friendly bot. And so our bot's name i- is Rocky, uh, in short for, uh, Bedrock. And, um, we'll see what Rocky-- if Rocky behaves today.

15. 3:14

    And the tools we're going to use with Rocky and the actions that those tools will, will, um, u- uh, jump on is jump, move to position, locate a player, and a few more, more actions that we, we've built.

16. 3:28

    And at this stage, I'll also mention that this demo that I'm going to show is open source, uh, and you can all have a play with it at the end.

17. 3:36

    All you need is a Minecraft client. So this is Rocky. I'm gonna give you a quick recording that I did yesterday at the booth. We're at the AWS booth if you wanna come and chat to me.

18. 3:48

    I am, I am not scary. Uh, I am from Australia, but I'm not scary. Um, so Rocky does a number of things. So this is what Rocky will look like.

19. 3:56

    At the minute, it's raining in Rocky land. Rocky is a very friendly bot. And so you can ask Rocky questions in the chat. I hope you can all see that.

20. 4:05

    So what's the weather? So, uh, Rocky being friendly will know my name, uh, and then also tell you what the weath-weather's like. So it's raining in your area. You might want to take some shelter.

21. 4:18

    Rocky can also do some actions. So this is how we send the actions to the agent via chat. So Rocky jumps for us.

22. 4:26

    Also, Rocky can do other things, so I'm gonna hide behind this Rocky-constructed building, and I'm going to ask Rocky, can he come and find me? Or can they come and find me, actually.

23. 4:37

    Rocky is neither male nor female. Um, Rocky says, "On my way," and there, there comes Rocky.

24. 4:46

    And Rocky's found us. But what else can Rocky do? So what else did I do here? [laughs] Oh, it can find things in the world. So Rocky here, I've asked it to find a pig.

25. 4:58

    So Rocky knows, based on, um, the plugin that we're using for Minecraft, where everything is, and it says, "Yep, I found a pig. This is the location." And I then ask it, "Go find-- Go, go to that pig."

26. 5:10

    So off Rocky runs to the pig and, uh, finds the pig. So we've made-- we've demonstrated this at a lot of conferences, and the biggest thing we've seen is people try to hit the pig.

27. 5:23

    So I asked it to hit the pig. And, uh, there it goes. Rocky hits the pig. Uh, you can do that multiple times, and lots of people I've observed do.

28. 5:31

    Don't know why, but hey, human behavior is even more fascinating than the LLMs.

29. 5:37

    Um, so now I am gonna get Rocky to come here, and I'll probably finish with the dig action. So I'll ask Rocky to dig a hole. Notice, um, when I say dig a hole, there's a parameter that I r- that I add.

30. 5:52

    So I think in this instance I... Yeah, a two-by-two hole. So this is a parameter that I'm sending to, to the action. I'll show you how all that works when I do a bit of a deep dive into the, into the, um, the back end.

31. 6:04

    So Rocky's, um, digging a hole.

32. 6:07

    And also, Rocky can come find us out of that hole and dig their way out of the hole. Uh, this is behavior that we didn't expect, and it just, just works, which is, um, really fascinating.

33. 6:21

    So that's, that's a bit of a three-minute, uh, demo of Rocky. I'm going to live demo Rocky once I go through how I built it. So the architecture. So we built this for a serverless conference.

34. 6:32

    So w- our, um, constraints were it needs to run in a managed environment, in a service environment, uh, and be cool. So we started off with running it in a container.

35. 6:44

    So Minecraft you can run on, in a container, and you can also run Mineflayer, which is the bot framework that works with Minecraft. And those two run side by side nicely on the container.

36. 6:56

    We can't run Mineflayer on Lambda, which is serverless, because we need state, right? So it's, the state is stored in Mineflayer.

37. 7:05

    We started off with LangChain on La- um, on Lambda to build this. Like most people, when they do start building agents, they probably start with LangChain. Uh, I'm not a big Python developer.

38. 7:15

    Uh, I am an engineer, but not really with Python. Um, but we, we got it working to a state. But then as we added complexity and more and more actions and, and tools for Rocky to use, uh, it, it got really, really, um, complex.

39. 7:28

    And then we also used Amazon SageMaker to host the LLM, which in this c- in this case, uh, we were using the Cohere LLMs.

40. 7:37

    And then we decided, okay, so that's not serverless enough. Uh, let's use, uh, Agents for Amazon Bedrock, keep the Minecraft server and client, which is, um, Mineflayer, on, uh, Amazon ECS, and build that agent up on Amazon Bedrock for...

41. 7:57

    And then for architecture, so what we're doing and what I'm demonstrating today, um, it is, it's running on my local machine, so both Minecraft and the Mineflayer client, and then it's into, uh, calling out to Agents for Amazon Bedrock.

42. 8:12

    And may the internet always be favorable over the next couple of minutes.

43. 8:18

    So if you haven't come across Agents for Amazon Bedrock, let me do a quick, uh, overview. Again, AWS booth is there. Come talk to me. I'll go a deep dive for you.

44. 8:27

    But in a nutshell, uh, Amazon Bedrock is all about façading, uh, one or more models. So what it does is it produces a common API that then you can use, uh, as an engineer to build out your application and add, add features.

45. 8:42

    So we host, as well as Amazon models, we host Anthropic models, we host Cohere models, as I mentioned, Llama models, um, and a whole host of other models. So when building out Rocky now using, uh, Agents for Amazon Bedrock, you'll see that I used the Claude, uh, models.

46. 9:01

    And I'll go through that as well, why I done, did that. Cla- in, in particular, Claude Haiku, because it's, it's fast.

47. 9:09

    And so as well as hosting models, not hosting models, but providing a façade into models, um, everything, you can also build, uh, additions to that. And one of those additions is agents, as well as knowledge bases where you can, uh, do RAG, and also guardrails and evals.

48. 9:26

    So all of these things are baked into Amazon, uh, Bedrock, and it's... Think of it more as a sort of a, a managed agentic workflow, right? So you can m-manage, um, RAG and manage a-agents as well.

49. 9:40

    So it, it's all in one spot. So it also helps you with prompt creation. It obviously, as it's an agent, orchestrates multiple tasks, and also allows you to trace through the chain of thought of the agent.

50. 9:52

    So you can either do that in the console or it'll spit it out to logs.

51. 9:56

    And of course, it also, and what was very key for this demo, it has return of control.

52. 10:02

    Okay, so that's the slides. Let's, uh, let's jump into what we've got.

53. 10:09

    So let me see if I can just instead of doing that,

54. 10:14

    uh... The mirror. No, not that. Oh, dear.

55. 10:30

    Stop mirroring. Okay. There's always... So main display.

56. 10:44

    Extend display. Main display. I want a mirror, and it's not, it's not letting me. Let me see.

57. 10:54

    Apologies. You got this. You got this. I got this. Let's do this re-refresh right here.

58. 11:01

    Refresh right here. This should do it. And mirror.

59. 11:09

    Mirror. There we go. Yay. Okay. So, okay, so this is, that's Mineflayer. So it's op- all open source. Let's first of all start with the good stuff. So open source, uh, Mineflayer you can use, and then this is, uh, the, um, what we've built.

60. 11:29

    Now I'll share this at the end, QR code, everything. Don't worry about it. So this is Amazon Bedrock. So this is agents, and this is the, um, the console page.

61. 11:38

    So what it'll do, um, you can go down here, uh, into agents. If you haven't seen Bedrock before, it'll show you the model access and what models you've got access to.

62. 11:48

    But I'm really talking about agents, which is down here on the left. So here I've got a Minecraft agent. Apologies if you can't see this too well. Let me see if I can make it a little bit better.

63. 11:57

    Um, you've got Minecraft, the Minecraft agent. So I've defined this in here. Now, for all the real engineers in the room, you don't have to build it via ClickOps.

64. 12:06

    So you can build it all as infrastructure as code, and if you check out the GitHub repo, you will see it all in CloudFormation and CDK. Um, so once I've gone into the agent, you can see that here I can select my model.

65. 12:17

    So at the minute, this agent is using Claude, Claude 3 Haiku, and here is the prompt. This is the system prompt for that agent. "You're a playful, friendly, and creative Minecraft agent called Rocky.

66. 12:28

    Uh, and your goal is to entertain players and collaborate with them in a fun gaming experience." And it goes on. So this prompt we, uh, built over, over time.

67. 12:35

    There's a bit of prompt engineering going on in this demo. Um, and then if I go into the actions, you can see all of the actions. So you notice all of the actions I've defined all have return control 'cause we, we want them to return, uh, the, the, their output.

68. 12:52

    So we've got jump, uh, we've got other actions. Dig. Dig, you'll see this is where the parameters come in. So I've got a depth and a width. So if you remember when I specified a hole, I said two by two.

69. 13:04

    Also, if I say small hole, it will go as a one by one hole. So it'll infer that, uh, using the model.

70. 13:11

    Uh, also, there's the action, is it raining? So all of these actions are defined in here.

71. 13:17

    And also, I've got another action set that I'm gonna share with you and gonna demo. It's called...

72. 13:26

    Do-do-do-do-do. It's called Minecraft experimental, right? And what this does, this has one action called build, which is a very complex thing to do in Minecraft, uh, believe it or not, because it's a 3D space.

73. 13:40

    And so the action for build, all I do is build a structure and, um, that returns back to the client, um, and I'll show you how that prompt's actually created, um, when I do the build.

74. 13:52

    So that's, that's how it's built, and the actual

75. 13:55

    source code itself is here, and the client is stopped, so I'm gonna...

76. 14:01

    So this is Rocky itself. So I'm gonna start Rocky.

77. 14:06

    Gonna start and bo-bot has spawned. Gonna go over to Minecraft we've got running here. Back to game. Rocky has joined the game. There's Rocky. Let me just go set

78. 14:21

    time. Time. It's time set. Time, set, uh, noon, so you can see it better.

79. 14:29

    Okay. Okay, so there's Rocky, so you can... Let's just test that Rocky's working. So T, please come here.

80. 14:41

    Uh, Rocky will make the first request, and there we go. So Rocky, Rocky is working for us. Um, so let's make sure that Rocky can do something. So we go, "Rocky, please, please dig a small, small

81. 14:58

    hole." Okay, so it'll-- Rocky will dig a small hole. So now I am going to use the experimental feature, which is build. What do you want Rocky to build?

82. 15:10

    Coliseum.

83. 15:11

    A, a, a what?

84. 15:12

    Coliseum.

85. 15:13

    A coloss- the Coliseum? Okay. I couldn't even spell that. [laughs] Um, some-something that I can spell, please. [laughs]

86. 15:25

    A double-decker couch.

87. 15:25

    A double-decker couch. Okay. The, uh, I like that. Um,

88. 15:30

    please come here. So yesterday I, I built, um, a rocket ship, which was quite interesting, but a double-decker couch. So let's, let's, um, get Rocky out of the hole.

89. 15:39

    So let's give it some space. Okay, Rocky.

90. 15:45

    Please build... Can I just say couch? [laughs] Double... Let's try it. [laughs]

91. 15:53

    Uh, never do live. Double-decker couch. Please build double-decker couch. A one, just one.

92. 16:00

    I'm just gonna, I'm just gonna make it as... So what happens, Rocky, then, that prompt goes off. And so if I look at the code, which I've lost...

93. 16:10

    Yeah. So there's the prompt. So what we've done, we've said to Rocky, "You're a Claude, an expert Minecraft builder creating by, by Ro- Anthropic when given a structured description," which will be the input, uh, "then output valid JSON."

94. 16:24

    So this JSON is how Mineflayer builds objects. So you give that as part of the prompt so that the model will understand how to build Minecraft objects. It, uh, then you'll go strictly adhere to the following rules because if you, we didn't do that, it goes bananas, uh, and builds just nonsense.

95. 16:41

    Uh, and so all blocks are placed to each other. These are the blocks it can use, and it responds with the, what it thinks a couch looks like. So this is the X, Y coordinates.

96. 16:50

    And so it's started the build. So let's see what Rocky's doing.

97. 16:55

    And there Rocky has built- [laughs] -a double-decker couch. Thank you very much. Yeah. [laughs] [clapping]

98. 17:03

    Um, I'm gonna sit on the couch. Um, so this obviously is the interpretation of what a double-decker couch looks like, uh, to Claude and in the 3D space, and then it's been interpreted into X, Y coordinates, and Rocky has built it.

99. 17:21

    So that, that is, that is Rocky. Obviously, we are here, uh, at the AWS booth all day. Could you please... I also put a QR code I promised before.

100. 17:31

     Let's fly to that. [laughs] Wee. So scan the QR code. Um, please fill in the session, um, feedback form. Also, if you fill it in, I'll give you AWS credit codes for everybody who fills it in and a link to the GitHub website.

101. 17:47

     So please do that. And, uh, hopefully you enjoyed the session, hope, hanging out with Rocky, and come ask me any questions at the booth when you can. Thank you very much.

102. 17:56

     Thank you. [upbeat music]