The Next Game Engine Won't Have a Manual

Talk transcript

1. 0:00

   [upbeat music] Hello. Hi, everyone. Thank you for being here.

2. 0:16

   Um, let's start the presentation. Uh, I really appreciate you having interest in, in learning more about, uh, game development. So today, I'm going to talk about how I think the next game engine won't have a manual, and, uh, at the end, you'll see what I mean by, by this.

3. 0:33

   My name is Arturo. I'm, uh, working on this tool called Nereu, and, uh, first, I want to show you kind of, like, how it works today, uh, so you get a glimpse of what I mean by all this.

4. 0:47

   Um, so we, we, we, we can start asking for what we want and describing something like, "Okay, I want to add a robot." Um, okay, so I ask my, uh, assistant to help me add a robot.

5. 1:06

   It found some assets that are robots or have been tagged as robots or the description are, uh, a robot. In this case, I want to have a character. It's going to be the character that I will be controlling.

6. 1:19

   Um, if I try to play right now, it won't do anything. I still need to tell what I want. I need to describe, "Oh, okay, I, I want this to be, uh, to move with WASD, and I want it to be animated, uh, have some sort of animation."

7. 1:34

   So I will describe that to my, um, assistant, uh, which is called Bibi, by the way, and I'm saying, "Make this robot move with WASD and animate it," as I, as I mentioned.

8. 1:47

   These descriptions are pretty common if you know or if you've played games. You just describe what you want the, the thing to, to do, and it's language that, uh, people who play games kind of understand.

9. 2:01

   Uh, you don't need to, to code or you don't need to, uh, learn about importing models or anything like that. The focus is on making the game. So now we have our character moving.

10. 2:11

    I want to continue, um, uh, describing my game, building my game, so I want to add some buildings. Uh, in this case, we have, like, a library of, of assets that might match the description of a building.

11. 2:24

    Mm, we could have, like, futuristic buildings, something like this, uh, warehouse or, or store. Uh, maybe we can use a, a castle. The idea here is that we can le- uh, let our imagination go wild.

12. 2:39

    It doesn't need to be, uh, you know, like a, m- the, the, the, the, the perfect game or anything. It's kind of like playing a game, playing with toys, playing with, with Legos or, or something like that.

13. 2:51

    I'm saying I want it to, to, to, to have some rain, so I just describe the rain, and it will spawn a particle system that's set up or configured as, as, as rain.

14. 3:03

    Uh, the last thing is I want a camera to follow my character because if I move around, it will go out of the, of the, of the view, and I won't be able to, to see it.

15. 3:14

    So it added a camera. It configured it. It's looking at my main character, and I have a game without knowing how to code or import assets or anything besides just knowing, uh, what I want because I, I know the, the language of, of building, building games.

16. 3:32

    That's the idea. Uh, so making games or getting to this point with a, a, a regular tool, it's very difficult. Um, I, I recorded this demo, but you could do this in just a couple of minutes without knowing anything, just describing.

17. 3:49

    All the assets are there for you. Uh, but if you were to follow the traditional workflow, you will need to know, uh, other things. Sorry, I, before jump- continuing, I, I want to mention, uh, why am I speaking here, and you can decide if, uh, you, you want to, to pay more attention or less attention to, to

18. 4:09

    myself. I worked at Unity, a game engine company, for almost 10 years, so I saw a lot of people building games and struggling with the same things over and over.

19. 4:19

    At first it was fun. The challenges were fun, but after seeing them thousands of times, it was like, "Okay, this is not fun. I don't think people should spend their energy and, and, and time, uh, re- uh, reinventing the same wheel."

20. 4:33

    So then I, I, I was part of MongoDB. Then I, I was learning about, uh, more AI topics and how to, uh, you know, manage data and all that stuff, and I was helping a startup to manage, uh, or, and handle, uh, version control of assets of, of games.

21. 4:49

    So as I said, I've been... I've seen people make games. I've made games, and, uh, I think

22. 4:57

    I want to, to, to, to, to let more people have fun and, and, and experience the joy of making games without getting frustrated and without, uh, you know, spending two, three years releasing something, realizing that it's difficult to, to, to, to sell g- a game.

23. 5:14

    There's a lot of competition, and of course, if you want to do that, I'm not preventing you from doing that. But a lot of people, I think it's, it's just an, a creative outlet, making games, and, uh, the game itself should be the, the, the, the thing, right?

24. 5:28

    Like, um, uh, enjoying the process rather than the end product. Okay, so as I was, I, I was saying before, making a game, uh, requires a lot of skills, a lot of knowledge.

25. 5:40

    It's very, it's very difficult. You need to know programming, 3D modeling, rendering, music, animation, and so many other things. You either have a huge team that, uh, complement each other, or if it's just you or you have a small team, uh, people need to wear multiple hats.

26. 5:59

    And honestly, it's very difficult to find someone that's great at game design and also great at, uh, rendering and great at composition of cameras in real time. So, um, on top of that, the game has to be fun, right?

27. 6:14

    If, if it works and technically works, if it's not fun, uh-

28. 6:20

    Honestly, people won't play it. And I, I think fun should be for the player, but also for the developers. In recent years, I think, uh, it's become more about producing more and trying to sell more, uh, rather than, than, than the, the craft of making games.

29. 6:36

    And I don't think you need to be a, a, a professional game developer to be able to, to make games. The same as with, with, um, uh, AI assistants to write code.

30. 6:47

    Now you don't need to be a programmer to, to build whatever you want. The same for, for games. There are other challenges, but, um, that's my, my thought. Now, there are powerful engines, Unreal, Unity, and there are powerful LLMs and agents, but I think still it's hard.

31. 7:04

    It's hard because I think we're just, uh, building a bridge between two worlds, and it's not optimal, and you still need to know kind of like what to ask in the vocabulary of an engine, in the voc-vocabulary of, of code.

32. 7:19

    Otherwise, uh, the, the, the, the, the LLM goes, uh, rogue and does stuff and reinvents the wheel over and, and over. Uh, this is the, the point, right? Like if I say, "I want a camera that follows this character," I've seen the, the demos, and L- the LLM reinvents the wheel every single time where when the result

33. 7:39

    is going to be essentially the, the same. Uh, by default, the context is on the game engine, uh, rather than on the game design part, and I think we should flip that idea.

34. 7:51

    So in a current engine, if you want to control a character, you need to have a mesh, uh, uh, import a mesh, and think about, uh, so many things, a renderer, an animator, a rigid body, and collider for physics, an audio source, and then you add your movement logic and your game rules.

35. 8:09

    Uh, most of that is just boilerplate that every single game out there has, or every character in every single game out there has. But somehow, developers need to understand and read the descriptions of those components and what the hundreds of, of, of, of sliders do, uh, in, in, in a game.

36. 8:28

    So I think the, the, the, the, the goal should be just to think, okay, everything is just an asset. Everything has to be rendered on screen. Everything has physics most of the time.

37. 8:41

    Uh, and you just add tags to describe the intent of those, those assets in your game. So I want this to be a character, to be animated, to double jump, and this is the language of...

38. 8:53

    that we use when we play games, right? Like, uh, the tutorial tells you, "Oh, press A to jump and press A again while you're in the air to do a double jump."

39. 9:03

    That's, that's the language that we should be using. And, uh, also defining what happens when, when even- an event occurs. Like in this case, oh, when you collect a coin, I want you to increase the score.

40. 9:14

    And, uh, of course, the physics and the rendering still happen, but, uh, we're, we're, we're building, uh, on a layer on top to make it more accessible for more people to, to build, uh, games.

41. 9:28

    Um, and the engine knows what to do with these tags because these, these systems, and this system comes from something that we're calling the ATS or asset tag system.

42. 9:41

    Comes from the idea from game development called a Entity component system, uh, data-oriented design, which means we just describe objects, uh, with, with, uh, in, uh, components or in this case with tags.

43. 9:55

    And then there are systems that query for all the assets in the, in the world and say, "Okay, this, uh... I'm, I'm going to move all the objects that have the vehicle, the player, and the drivable tag."

44. 10:10

    So that's how, how it works, and all the games can recycle this. In the... on the, on the right, you see that there's a building. It doesn't have tags, so it's not going to move.

45. 10:20

    But nothing prevents you from adding the, the vehicle and drivable tag to your building, and then you have a, a building that you can put in a Mario Kart style of game.

46. 10:30

    Uh, nothing prevents you from, from doing that and as I said, like letting your imagination go, go wild. Uh, the assistant, the AI assistant essentially helps you get unstuck.

47. 10:40

    Like if I don't know how to make the car moves, uh, I just ask it, and it knows, understands, like what are the tools available, what are the tags available, and then it, uh, a- applies them to, to the, to the, to the asset that I'm talking about.

48. 10:54

    Uh, how it is driven, and this is, this is pretty common, um, concept from, from, um, agents. You just type your query. You, uh... well, we build the, the, the prompt using context from the scene, uh, but also some extra context, um, based on, for example, what type of assets are on your game.

49. 11:16

    If you're using robots, if you're using something like medieval, et cetera, we fe- um, we, we append that to the, to the, to the context. We also, uh, allow the, the user to describe the type of game that they want to, to make, so that's also part of what, um, uh, is, is appended into the, into the,

50. 11:34

    the context. And then the agent just performs calls and appends or removes the, the tags. All these, uh, tags and all these systems are built into the, the, the, the engine.

51. 11:47

    Um, we don't have a scripting, uh, system in there. That's on purpose. But it's just JavaScript and runs on the, on the browser, so whoever wants to extend, they can.

52. 11:58

    Nothing prevents them from, from doing that. But for most users, uh, that shouldn't be the, the case. And, uh, yeah, all the updates to the scene happen, and that's how, how it works.

53. 12:09

    Honestly is... the, the, the challenging part is the composing- Games, uh, into these tags and into these systems because, uh, there are a lot of genres, there are a lot of, uh, ways to describe the same game.

54. 12:24

    There are things like the mood of the game, like I want to make a, a platformer, but that feels, uh, like scary. Well, that also touches on things like, um, like, uh, the post-processing effects and things like the, uh, lighting, et cetera.

55. 12:38

    So those, those things are the things that we're decomposing to, to, to, um, drive the, the engine. How I'm building this, well, it's,

56. 12:49

    it's the first time that I build something of this scale at this speed and using, uh, you know, mostly working with, with an AI, um, daily, and I'm sure everyone uses, uh, Claude Code or something similar.

57. 13:02

    Uh, the way it, it works is I bring all this stuff that is on my mind, and I talk to the, to, to Claude, and we discuss that part.

58. 13:12

    Then I go to... Let's say, if I'm focusing on something related to, to, to, um, lighting, I talk to some friend on the industry who focuses on that, and I ask questions, and then I bring that in, and then we build those tools.

59. 13:25

    And I kind of like say, "I don't think we need this setting. I don't think we need this feature. I haven't seen a lot of users using those things, so let's get rid of them in order to simplify the, the, the engine."

60. 13:37

    Uh, we build it. Then as we build, we give, uh, the definition of the tools to the, to the assistant. So every time, uh, we add new tools, the assistant knows the, the tools that we're, we're adding.

61. 13:50

    And, um, yeah, uh, right now it's, it's kind of like still in closed alpha. I don't even know how to call it at this point, but some people are using it, giving feedback, and, uh, then we go back to square one.

62. 14:04

    Um, one thing that I, I think it's worth sharing also in, in, in, in this presentation is the, the way of assembling this context because, yeah, we're using an LLM mostly.

63. 14:16

    Uh, I've used, uh, also vision models mostly to tag the, the assets that we have because it's like six or 7,000 assets. I could not manually tag them all and explain like, "Oh, this is a, a, a, an astronaut, and this is a, a, a knight, and this is a castle, and this is whatever."

64. 14:34

    I just have the names and the, and the, and the 3D file. So I took a screenshot and ran, uh, a vision model to describe the, those. But it's mostly an, an LLM, what's, uh, what we're using.

65. 14:48

    And if we feed the entire scene to the LLM, the context grows a lot. Let's say in this, in this scene, I think I had like 100 assets, 100 things, uh, being there, but most of them are like grass that we could ignore.

66. 15:04

    It doesn't really make sense. But, uh, this is simple, like, like scene. But what we're, we're using here to, to assemble the context is something that's very well known in, in the, in the game dev world, which is called level of details, uh, which means if I'm close to an object, uh, or, or it's close in, in

67. 15:23

    sight to, to myself, I'm going to render it with a higher quality texture, with a higher quality material, um, with a higher quality model, in many cases. Something that's too far away from the camera, I'm just going to maybe just put a, a, a cube, and the user won't be able to, to tell because it's so far

68. 15:41

    away. So we're using something similar to assemble the context. In this case, if the user is editing the, the, the game, you can see that maybe, uh, well, uh, it's, it's, it's clicking on the, on the knight, right?

69. 15:54

    So the things that are around it might have a higher priority, so we feed them, uh, information about the tags that they, they have and the, the values of the, of the settings on, on those tags.

70. 16:06

    Things that are nearby, we just say, "Okay, there's a something, uh, that's a player, uh, here at this position, uh, but I'm not going to send you the entire, entire, uh, context in there."

71. 16:19

    Okay? And as a, as a user keeps moving around and modifies things, then we update that, uh, and feed the, the, the, the assistant with more relevant, um, information about, about the game.

72. 16:34

    Okay. So just to start wrapping up, I, I, I, I think this assistant called Bibi should be kind of like the, the...

73. 16:44

    Something that ons- gets people on stock from making the game. Uh, I don't want us to one-shot games that nobody is going to play, and, uh, I don't see the point in, in that.

74. 16:56

    There's, in the industry, of course, uh, the need for, for games that are one-shotted, but here the idea is that we allow people to make games and experience that and have fun and, uh, share that, those games with their families and friends and, and, and that.

75. 17:11

    And of course, that they learn along the way the language of making games, the language of game design, not necessarily the coding or programming. Al- although, uh, people have used it, this and say like, "Oh yeah, I can understand now concept that I didn't understand before of programming," but, uh, that was not the, the, the initial goal.

76. 17:32

    Um, there's another kind of like branch of, of engines or tools that are, are being used called, uh, world models, which are being generated on the, on the fly and, and some people...

77. 17:44

    I think there's another session later talking about these and how games can be achieved with, uh, world models. I think that's going to be a different medium, even if we call them video games.

78. 17:54

    Uh, there are many challenges. Uh, games, uh, for instance, in these days, they have to render 60 frames per second, and doing that at 4K resolutions in real time, uh, with a, with a world model, I think it's ta- it's still far away.

79. 18:10

    And on top of that, uh, rendering physics and on, and, well, t- sorry, simulating physics and stuff, it's very difficult, but I think it's exciting what's going on. Um, so that's it.

80. 18:23

    If you want to play around, if you want to share this with your kids or friends or, or whoever, uh, I, I can add you to the, to the wait list.

81. 18:33

    Um, it's... Honestly, it's been very fun to make, but also whenever I want to test something and I just play around and, and, and bring in, um, an astronaut and bring in a dinosaur and make them just walk around, it's, it's so much fun.

82. 18:49

    It's, uh, it's, it's kind of like what got me into game development I think like 20 years ago. And I progressed throughout my career. I started getting like kind of disappointed, like, "Oh, I don't...

83. 19:01

    This is not fun anymore," so I'm having fun again, and I want to share that with, uh, more people. So thank you so much, and I'll be outside if you have questions, want to chat more about games or whatever.

84. 19:12

    Thank you so much for your time. [audience applauding] [outro music]