AI Engineer Summit 2023
Using AI to Build an Infinite Game
Read the talk
Using AI to Build an Infinite Game
Jeff Schomay turns a finite forest adventure into a stream of generated encounters by combining scene JSON, a custom image model, validation, and a replenished asset cache.
From a talk by Jeff Schomay
Before you start: Familiarity with JSON and basic API request flows will help; no game-development or model-training experience is required.
A forest the player can exhaust
You are lost in a forest. Each encounter changes your vigor or courage, and you need to find home before your courage runs out. In Jeff Schomay’s Lost in an Infinite Forest, the original content was entirely AI-generated, but the world was still finite: sixteen scenes arranged in a four-by-four grid. After a few playthroughs, a player could see everything.
The most enjoyable part of building that world had been generating a scene and discovering what the AI would produce. Why reserve that surprise for the developer? If new scenes could be generated for the player, each game could offer different encounters. Infinite exploration begins by moving generation from content creation into the game’s asset supply.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate a scene the game can consume
The first requirement is a consistent scene definition. A scene is a JSON object describing what happens when the player discovers it, what happens on a return visit, and how those encounters affect player stats. That distinction gives a location behavior beyond its first appearance: returning to a scene can produce different text and consequences. The visible “Intimidating Bear” example includes discovery and return descriptions with courage changes.
Schomay starts with OpenAI’s completion endpoint and a long, detailed prompt. Most of the time, it produces the expected JSON structure and content that fits the forest while remaining varied and interesting. He then tries fine-tuning to make that output more reliable. This is the historical completion-and-fine-tuning workflow; the current OpenAI guide says its fine-tuning platform is winding down and unavailable to new users.
Schomay generates fifty examples for fine-tuning, citing OpenAI’s recommendation at the time of fifty to a hundred. He shortens the training prompt, replacing the detailed formatting instructions with a general description of the desired scene so the examples carry more of the output contract. His spoken account says he removes all mention of JSON; the published asset-server example still requests a specific JSON map. The supported distinction is between supplying the full structure in the prompt and teaching that structure through examples.
Schomay reports spending roughly $1–$2 on generating the examples and fine-tuning combined. In his trial, he reports perfect JSON output even without mentioning JSON in the prompt, although he gives no evaluation sample size or measured failure rate. The shorter prompt also uses fewer tokens, which he describes as faster, cheaper, and easier to work with. Fine-tuning has shifted repeated instructions into the model’s learned behavior; the later asset pipeline will still validate its output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the style consistent and the encounters varied
Next comes the artwork. Leonardo offers both image generation and custom image models, allowing Schomay to pursue a consistent visual style across the game. He experiments with prompts and generation parameters until he finds a look he likes. Surprisingly, the scene description itself works as the image prompt, despite containing second-person narration and details beyond visible objects. That lets the text generator’s output feed directly into image generation.
Training the image model requires separating what should stay consistent from what should vary. If every training image repeats the same subjects, the model can overfit and make every scene look alike. If the images share too little, the desired style becomes unclear.
| Keep consistent | Allow to vary |
|---|---|
| Perspective and scale | People and animals |
| Forest setting | Buildings |
| Overall tone and texture | Scenes without those subjects |
This is more demanding than collecting attractive forest pictures: the training set has to demonstrate both the visual rules and the range of encounters those rules should accommodate. Schomay tries multiple models with different parameters and image sets before finding a satisfactory result.
He tests the chosen model by generating a large collection of images. Recurring trees, a zigzag path down the middle, and a shared surface treatment make the scenes feel related, while their contents remain distinct. With structured encounters and cohesive artwork in place, the next task is to connect them into a working asset pipeline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Validate, illustrate, and assemble
The asset server joins the two generators in a dependent sequence:
- Request a scene from OpenAI using the custom model.
- Validate the returned JSON, checking that the required keys are present.
- Send the valid scene’s description to Leonardo’s custom image model.
- Combine the returned image with the scene data, then save and deliver the asset.
Validation sits before image generation, so a scene missing required fields does not proceed to the next generation step. The completed diagram shows the whole path, ending with “Merge and save.”
The same control flow can be expressed in Python with provider calls supplied as functions. Here, required_keys and description_key come from the game’s scene format:
python
import json
def build_asset(request_scene, generate_image, save_asset,
required_keys, description_key):
scene = json.loads(request_scene())
if not isinstance(scene, dict):
raise ValueError("Scene must be a JSON object")
missing = set(required_keys) - scene.keys()
if missing:
raise ValueError(f"Missing scene keys: {sorted(missing)}")
description = scene[description_key]
if not isinstance(description, str) or not description.strip():
raise ValueError("Scene description must be nonempty text")
image = generate_image(description)
asset = {"scene": scene, "image": image}
save_asset(asset)
return asset
The scene description is the connection between narrative generation and artwork; the final asset keeps the image and the encounter data together.
Schomay also builds a simple preview server that lets him scroll through generated scenes. The pipeline’s structural checks establish that data has the necessary fields; previewing lets him inspect the assembled text and artwork before integrating the output into play.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prepare encounters before the player needs them
The game can now request generated assets when the player enters a new scene, but that exposes the pipeline’s waiting time. Schomay reports scene-and-image generation taking 10, 20, sometimes 30 seconds. Waiting that long at each transition would interrupt exploration.
His solution is a prefilled scene cache. Generate a stock of complete scenes before they are needed, take one when the player enters a new location, and replenish the stock whenever it falls below a threshold. The player receives a prepared encounter while generation supplies future encounters. This does not make either model faster; it moves their work ahead of consumption. Keeping scenes ready depends on replenishment keeping pace with play.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Walk through the generated forest
Schomay introduces the live demonstration as content that has never been seen before and will never be seen again. The game still has a fixed starting point: a lamppost from which the player sets out to find home. Stats appear in the bottom-left corner, but their effects also change how the world feels. Lower vigor slows movement; lower courage makes the viewport smaller. An encounter therefore affects both the player’s resources and their ability to navigate.
Moving down reveals the first generated encounter: soft blue, pulsating light coming from organic formations scattered around a glade. The text describes fear and tiredness lifting. It offers a vigor increase, but the player is already at full vigor. The next location resembles a campfire scene; continuing down leads to a large, dark cave at the end of a path. That encounter is daunting, and courage falls.
The player changes direction and enters foggy trees where visibility is poor, then backtracks through a winding road. Another move brings Schomay back to where he started. The demonstration stops there rather than showing a win: exploration would continue until the player found home, and another playthrough would provide different encounters. Fresh content is being supplied to a navigable world in which the player can still revisit locations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Higher resolution and a more personal forest
The working pipeline leaves room for additional processing. Schomay describes the existing images as 512 pixels and proposes adding an AI upscaler to increase their resolution. That would also add generation time: improved artwork would make each prepared asset more expensive in time to produce.
He also proposes making the scene prompt responsive to the player’s context:
- Selected theme: Let the player choose the kind of forest experience to generate.
- Time of day: Incorporate the player’s current time into the scene request.
- Local weather: Use current weather at the player’s location to influence the generated world.
These are proposed extensions, with the aim of making scenes feel more closely connected to where the player is. The same generation process could also supply assets for other projects: the forest is one application of a pipeline that turns a scene request into structured content and matching artwork.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The forest exploration game, with macOS and Python source downloads and instructions for navigating while managing courage and vigor.
Backend repository covering scene fine-tuning, image generation, asset previewing and a replenished scene cache.
Python game client with installation and executable build instructions.
Further reading
The speaker's written walkthrough of scene prompts, custom artwork and the asset pipeline, with links to the project.
Updates since the talk
API reference for training a custom image model from a dataset, including prompt and resolution parameters.
- OpenAI supervised fine-tuning guidanceDocumentation
Dataset sizing and evaluation guidance, with a current notice about the fine-tuning platform winding down.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi everyone.
- 0:15
I'm Jeff Schomay, and I wanna share with you an interesting generative AI project that I recently did. Not too long ago, I made a game with a hundred percent AI-generated content.
- 0:30
It's a simple game where you're wandering around, lost in the forest, and you go from scene to scene, having encounters that impact your vigor and your courage. And the idea is that you want to find your home before you run out of courage.
- 0:48
There's sixteen scenes in a four-by-four grid, and so if you play a few times, you will have seen them all. Now, my favorite part of making this game was generating each scene and just seeing what AI would come up with.
- 1:04
And I thought, "Wouldn't it be cool to share that experience with the player? What if every time they went to a new scene it was generated fresh for them, and every game would be unique and different this way?
- 1:19
It would be a game of infinite exploration."
- 1:24
That sounded so cool that I wanted to try to do it. Now, the first thing that I would need to do is to generate each scene and have a consistent way of doing that.
- 1:35
My scene definitions are JSON objects that describe what the scene is when you first find it, as well as when you come back to it later, and how that impacts your stats.
- 1:48
So I started out by using OpenAI's completion endpoint and doing some prompt engineering.
- 1:57
This is the prompt that I used. This is a very detailed prompt. It's rather long, but it worked really well. Most of the time, I would get scenes that had the right JSON format and the content was good.
- 2:15
It was fitting, it was varied, it was interesting, so I was happy with this. But I wanted to make it even more reliable, and I decided to fine-tune a model.
- 2:29
I used OpenAI's fine-tuning endpoint, and they recommend fifty to a hundred examples. I generated fifty examples just like these and used them to fine-tune.
- 2:47
Now, the key is I shortened the prompt. I simplified it, I took out any of the JSON, and just generally described what I wanted, hoping that that information would be embedded in the training data.
- 3:07
And I tried this out. I wasn't sure if it would work, and I tried it. It only cost about a dollar or two. That includes generating all the examples and doing the fine-tuning.
- 3:21
And when I tried it, I was very happy to find that it worked perfectly. Even though I didn't mention the JSON at all, it came out perfect because of what was in the examples, and that meant I had less tokens in the prompt, which is faster and cheaper and just easier to work with.
- 3:44
So I was really pleased with how this worked.
- 3:48
The next step was to make the images. Now, I used a tool called Leonardo. Leonardo not only lets you generate images, they also let you create your own image models.
- 4:03
And this is great for a game because it means that you can have stylistically consistent images, which is exactly what I needed. So I spent a while using all the different parameters that Leonardo offers and working with the prompt to try and find an image that looked right and that I liked.
- 4:24
It turned out that using the description directly from the scene as the prompt made nice pictures, which I was surprised about since it had, like, second person and said things other than what was in there, but it worked out great.
- 4:40
Now, the tricky part with fine-tuning an image model is that you need consistent images that have... Like, the parts that should be the same are the same in all of your training data, but the parts that you want to vary need to be varied, otherwise it will overfit and all of your images will look the same.
- 5:01
But if you don't have that consistency between them, then it won't really know what you want, and you won't get that good stylistic consistency. This was really tricky, especially in my case.
- 5:14
I needed the perspective and the scale to be consistent from scene to scene. Obviously, I needed them all to be set in the forest, and I wanted to have this overall tone and texture that looked the same.
- 5:29
Some of my scenes have people in them, some have animals, some have buildings, some have nothing, and so it was hard to get that variety. I ended up having to train a couple of models with different parameters, different sets of images, but I eventually found one that worked out.
- 5:48
And to test it out, I generated a lot of images. I mean, a whole bunch. And you can see they all have similar features, like the zigzag path down the middle, obviously the trees, and the look, and everything looks the same, and yet...
- 6:09
There's plenty of variety. Each one is unique and different, but still feels cohesive, which I am very pleased about. So now I had everything I needed to put it together and make the game.
- 6:23
I made a simple asset server that had an AI pipeline starting by requesting a new scene from OpenAI's endpoint using my custom model. Once I get that, I validate the JSON to make sure that it's got all the keys it needs.
- 6:42
If it's good, I take the description and I send that to Leonardo. Leonardo makes an image from my custom model, gives it back to me, I put it all together, and send it off.
- 6:54
Now, did this work? Well, let me show you. Here is an example scene that was created,
- 7:03
and I'm very happy with it. I made a simple preview server so that I could scroll through a bunch of these scenes that I generated to make sure they worked, and it looked good.
- 7:18
So I made some changes to the game to request images each time the player went to a new scene. Now, there was a problem here. It takes 10, 20, sometimes 30 seconds to do this, and that wouldn't be good for the play experience.
- 7:40
So what I did is I added some caching. I pre-fill a bunch of these scenes, and then as scenes are taken out of it, I fill it back up again once it gets below a certain threshold.
- 7:54
And that way, there's always a scene that's ready to go.
- 8:00
With that, the game was ready, and I'm gonna share it with you right now. Now, keep in mind,
- 8:07
everything that we see has never been seen before and will never be seen again.
- 8:15
So this is the game. You always start out at this lamppost, and you have to wander around and find your way home. Your stats are in the bottom left corner.
- 8:26
As your vigor goes down, your speed goes down as well, and as the courage goes down, the viewport will get smaller and smaller. Let's look around and explore. We're gonna move down.
- 8:39
And here's the first generated scene. This looks really cool. This is like a, uh... You encounter a soft blue pulsating light coming from the organic formation scattered around the glade.
- 8:51
Your fear and tiredness lift, and you feel rejuvenated, and the vigor goes up, but I'm already at full. So that's really cool. Let's head off in this direction now.
- 9:01
I won't read all of these, but this looks like a cool, like, campfire scene, which is really neat, and I'm gonna head down.
- 9:12
And what have we got here? There's a, a large dark cave over here at the end of the path somewhere, and it's, it's daunting, so my courage is going down.
- 9:21
Let's head this way instead. And now we've gotten into some fog, foggy trees, and
- 9:32
hard to see. Let's go back. Uh, this is, like, a really windy road that we're going through.
- 9:41
Let's head down. Oh, I'm back where I started. Well, this is the game, and it would continue on and on and on until you found your way home, and then you can just play again, and it would be different every time.
- 9:55
That's great. I just have a few closing thoughts. One thing is that these images are low resolution. They're 512 pixels, and I could make them a higher resolution by adding an AI upscaler to my pipeline.
- 10:14
It would add more time, so it's a trade-off. Also, I could get more creative with adding something to the prompt to make a scene. For example, I could let the user select a theme or maybe even get the time of day or the current weather at the location of where the p- user is set, and then the
- 10:37
scenes could be generated to match where they are for a very immersive experience.
- 10:44
And of course, I can use this same process on other projects.
- 10:51
That's all. I hope that you found this interesting and enjoyed watching it as much as I enjoyed putting it all together. Thank you so much. [outro music]