AI Engineer World's Fair 2026
OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents
Read the talk
Vault: A Handheld Terminal for Local AI Agents
A keyboard, OLED and e-paper display make a quiet interface to OpenClaw, while a separate DGX Spark handles inference, agent actions and generated adventures.
From a talk by Lech Kalinowski
Before you start: Basic familiarity with microcontrollers, LLM inference and agents that can act on a computer is helpful.
A terminal that answers back
During a basement cleanup, a dusty device appears in a corner: an old-looking keyboard terminal with unusual displays. Its shell is still running. Type “Who am I?” and it answers in natural language. Lech Kalinowski opens with this imagined discovery, then reveals that it is a possible marketing story. The real ambition is straightforward: build a physical, AI-native device that feels as though it came from the future.
Kalinowski, who holds a PhD in physics, spent several months building the device, its backend and the surrounding system. The starting requirement was narrower than the eventual product: a remote controller for an OpenClaw instance on his DGX Spark. That controller became Vault.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate typing from reading
For this interface, the useful starting point is text: a place to type requests and read model responses. Electronic paper suits sustained reading, but its refresh behavior makes it less suitable for updating the screen with every keystroke. Kalinowski pairs it with a small monochrome OLED, dividing the work between two surfaces instead of asking one display to serve both purposes. The intended benefit is a responsive terminal with economical display power use.
The remote controller also becomes an experiment in an AI-native operating interface built around a microcontroller. Its four modes are shell, assist, control and RPG, as enumerated in the recording’s description. The internal shell handles the terminal’s own settings, basic configuration and Wi-Fi connection; the other modes extend the interface toward assistance, remote actions and play.
In the prototype, the OLED is the live typing surface. Entering text updates that display; pressing Enter triggers a render on the bistable e-paper display. This gives the interaction a clear boundary between composing an input and committing a page for reading.
| Surface | Job | Update behavior |
|---|---|---|
| OLED | Live text entry | Updates while typing |
| E-paper | Committed page | Renders after Enter |
The same interface carries requests and responses for LLM interaction and operations on the DGX Spark through OpenClaw.
Making that division practical requires a deliberately small rendering contract. Pages live as one-bit images in fixed, pre-allocated buffers. The microcontroller displays those images without a Markdown engine or calls to malloc. The page representation is therefore bounded in memory, and the terminal does not need to interpret a rich document format to put a response on screen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Power delivery is part of the interface
The interior is more complicated than a keyboard attached to two screens. Much of that complexity comes from power management: Kalinowski reports destroying two displays during prototyping and needing much more stable current and voltage. A reliable interface depends on keeping its fragile display electronics alive.
The hardware centers on a dual-core ESP32 microcontroller, with the OLED, keyboard, rotary encoder and power-supply components around it. The opened enclosure shows how much wiring and supporting circuitry sits behind the restrained exterior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep inference off the handheld
Local inference happens on the backend, not inside the terminal. The compute and power requirements of the LLM would overwhelm the handheld design. Vault firmware runs on the device, while the backend handles both OpenClaw’s agent work and model inference. The terminal is a physical entry point into that system.
For the demonstration, Kalinowski describes an open-source GPT model in the 120-billion-parameter class; the recording’s description identifies it as gpt-oss:120b. He serves it using TensorRT, exposing an OpenAI-style interface through an LLM proxy. That adapter matters because other open-source models and serving setups had not matched the API behavior he needed. Current deployment guides describe available serving paths, but do not establish the exact proxy or server configuration used here.
A concrete example is asking the agent to write a Java example and save it on the local machine. Kalinowski reports that OpenClaw, with the model’s help, completed the request. The important transition is from generating an answer to creating a file on the backend host. A small Java artifact for that kind of request could be Hello.java:
java
public class Hello {
public static void main(String[] args) {
String name = args.length > 0 ? args[0] : "Vault";
System.out.println("Hello, " + name + "!");
}
}
The keyboard supplies the request and the displays present the interaction; the agent performs the filesystem action on the machine it controls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What failed on the workbench
The prototype’s field notes are mostly electrical rather than model-related:
- Software I2C: The initial wiring did not work correctly. Kalinowski reports eventually controlling the bus without adding physical pull-ups. Treat that as a property of this build, not a general wiring recipe: Espressif’s I2C guidance generally recommends external pull-ups because the internal ones are weak.
- GPIO 13: A silent failure led him to move the connection to another port.
- Power regulation: A regulator damaged the OLED and other fragile display parts. Reworking the power-supply unit was necessary, and replacement parts took a couple of weeks to arrive.
- Encoder noise: A cheap rotary encoder produced noisy rotational input. He added pull-ups and capacitors to address it.
These failures explain why the supporting circuitry occupies so much of the hardware design: a clean text interface still depends on clean electrical signals.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The remote controller becomes an RPG console
The unexpected fit is a text-based role-playing game. Kalinowski had never played a tabletop RPG with friends gathered around a book and a game master. Building Vault gave him another route into that experience: a dedicated console where an LLM supplies the unfolding fiction.
NPCs, memory and the mood of the world give the game continuity beyond a single generated response. He creates four worlds, drawing on cyberpunk, Witcher-like fantasy with dragons, and a favorite deep-space setting. The text-focused hardware suits an experience in which reading, choosing and imagining are the main activities.
Generation extends to characters, worlds, maps and skills. Pictures then pass through the same rendering constraint introduced earlier: they are converted into one-bit matrices or image maps for the display. The generated fantasy imagery and its physical e-paper output show the two sides of the pipeline—rich material produced upstream, reduced to a representation the terminal can hold and render.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preserve a usable path—and a quiet environment
The paired components also provide alternative interaction paths. Kalinowski describes the device as resilient because an individual failure need not make it completely unusable.
| Failure | Remaining path |
|---|---|
| OLED stops working | Read through e-paper |
| Keyboard stops working | Use the encoder |
| Wi-Fi connection fails | Use the local shell |
These paths preserve parts of the experience; the local shell, for example, remains useful without implying that disconnected backend inference continues.
That restrained interaction model is also the product idea. In his intellectual-property research, Kalinowski found much of the attention going toward audio interfaces and video capture. Vault instead makes room for quiet reading, writing, agent work and RPG play without bright moving graphics, advertisements or web browsing competing for attention. He reports pursuing a provisional patent, with the possibility of a commercial device in the future.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the build adds up to
Kalinowski reports 130 commits over roughly three months. His project inventory includes two displays, four modes, four gaming worlds and sixteen classes associated with the RPG system. He describes the firmware as small and fast, without giving a measured size or latency. The handheld uses a single battery cell; the spoken recap calls it lithium-polymer, while the earlier hardware slide labels it an 18650.
The resulting design principles follow directly from those choices: match each display to its job, keep heavyweight LLM inference off the tiny microcontroller, and preserve narrative context in the game rather than tracking only numbers. The display architecture controls the physical experience; the backend carries the computation and the remembered story.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From boot to a remote disk-space query
The recorded demonstration starts with the ordinary operations that make a dedicated device approachable:
- Boot the terminal and connect to the local Wi-Fi network.
- Move through the welcome screen to the main interface.
- Type
helpon the keyboard to display the available commands.
The command reference fills the e-paper screen in multiple columns, showing how the larger reading surface complements the small live display.
There are separate tutorials for the device and for gaming. The first-use tutorial is optional, so a returning or confident user can proceed directly to the interface. Kalinowski then uses OpenClaw to ask about disk space on the DGX Spark. A response appears in the recording, completing the path from handheld input to inspection of the remote machine and back to the display.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing an adventure, finding an LLM Game Boy
The demonstration then switches to gaming. Four worlds are offered—named in the narration as Alderan, Neon Nabas, The Hollows and Voidreach—followed by character selection. The LLM generates the adventure’s material, including pictures and character personalities. The deep-space scene introduces a breach and a distinctive mood: the terminal is now a window into a generated setting rather than a remote system console.
After the video, Kalinowski points out the amusing contrast: this quiet text game has no three-dimensional graphics, yet the demonstration runs against substantial NVIDIA hardware in a DGX Spark. That is the backend he chose for the demonstration, not a minimum hardware requirement for text RPGs. The computation has moved out of sight, leaving a small device organized around reading and writing.
What began as a handheld controller for OpenClaw and local models has become something closer to a Game Boy for playing with LLMs. Kalinowski closes by crediting Callstack, whose technological incubator supported the work. The change in purpose is the revealing part of the build: once model interaction has a dedicated physical form, operating an agent and inhabiting a generated world can share the same keyboard and paper-like screen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Hardware specifications for the compact GB10 Grace Blackwell system used as the terminal's inference backend.
Further reading
Espressif's versioned guide explains I2C configuration, pull-up requirements, and error handling for the ESP32-S3.
Updates since the talk
NVIDIA's current deployment playbook includes gpt-oss-120b support and OpenAI-compatible serving on DGX Spark.
Current configuration guidance for connecting OpenClaw to local inference servers and OpenAI-compatible proxies.
Read the complete timestamped transcript
- 0:02
Probably all of you once planned the cleanup activities, and you can imagine I also planned one weekend to
- 0:11
cleanup of my basement, and I just, uh, went there, uh, and want to clean up all of the stuff there, you know, throw up, uh, old washing machine. And I just look at the corner there and realize there's something weird there lying on the floor.
- 0:34
So I just grab it, look at the, the strange device, looks a bit familiar, but a bit old, like from '80s, with keyboard, but with a weird displays, of course, covered with dust.
- 0:51
I just take it to my hand, look at it,
- 0:56
and I was completely surprised that, um, device have a shell and it's on.
- 1:07
So, um, I used my superpowers skills in a terminal, and just I provided there a simple question,
- 1:20
"Who am I?" And the device responds in the weirdest way I could even imagine,
- 1:28
in the natural we- language. So the story I just described to you, it can be a great story for a marketing.
- 1:40
But honestly, mmm, I just wanted to build a device which is physical and AI native, like the device which comes from the future.
- 1:57
Hello. My name is Lech Kalinowski. I hold a PhD in physics, and today I want to present you a device and a backend and a system I built over last couple of months.
- 2:14
And the whole story begins because I just simply wanted to build
- 2:20
a remote controller to my OpenClaw instance on my DGX Spark.
- 2:31
And, um, the story is simple because when you are just first think about, uh, LLMs, then you're not thinking about the audio, about the generative, uh, generative, uh, videos or pictures.
- 2:50
First, you're thinking about the text. And then I just get a bright idea that maybe I can use some kind of better display to read and type with my LLM.
- 3:06
F- of course, on the market, there is a electronic paper, but, um, I realized electronic paper can be a bit slow. Of course, it's perfect for reading,
- 3:18
but if you just want to provide dynamically text to it, you probab- probably need a faster display. And then I just, uh, used and, uh, in my design,
- 3:33
uh, a simple connection of a OLED display, the small one, one color,
- 3:43
with the electronic paper display. And then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to
- 4:01
play with my claw. And, um, of course, behind it, uh, there is, uh, some kind of lot of complications because from one side, I just wanted to build a remote control, but from other side, there is such a niche on the market,
- 4:26
and this is the AI-native operational systems. And when I just started to think more clearly about it, I just realized I can build such a system for my purposes, but not with the super powerful co- powerful computers, but with a small microcontrollers.
- 4:48
And I just design a system built in, let's say, with a, uh, with a, say, uh, four classes. One is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection.
- 5:09
And within my idea to build an AI-native device, I build an assist control and, uh, one thing which really surprised me, and I will explain you later, RPG mode.
- 5:28
And, uh, here it is. I started to prototype DGX Spark at the,
- 5:37
um, at the background here on the picture, two displays. One, it's a live surface. That means the dynamic part. You can type the whole text provided is written there.
- 5:54
And when you are just trigger action, hit Enter, then the whole render also happens on the second bistable electronical paper
- 6:09
But, um, there is a lot of complications and lot of tech and lot of steps you need to do to get an clear user experience and clear interface. So I just found out a lot of the, uh, different approaches how to render properly on the two displays, properly, uh, render the,
- 6:34
the, the, the interface. I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX Spark through the, through the OpenClaw.
- 6:45
And this is the... one of the approaches that you can just
- 6:50
have fixed static buffers and render one-bit images stored in the,
- 6:59
in the memory. That means the pages live in a pre-allocated memory. There's, uh, no markdown engine, uh, and no malloc on the MCU side.
- 7:17
I just wanted really to show you how the system is really complicated here, and this is, uh, not because I just,
- 7:27
uh, wanted to use that much components. It's because there is a huge need to build a, to build a power, um, management system,
- 7:39
uh, because I just blow up two displays over my, uh, prototype build, and, um,
- 7:48
I just, uh, needed to be sure that the current and the voltage is stable as possible. So here inside, you have an MCU, which is ESP32 dual-core microcontroller.
- 8:03
Here you have an OLED display, keyboard, and a encoder, and of course, the mentioned, um, power supply.
- 8:16
But the A Native end-to-end device is not, uh, let's say that simple because of the, you know, high power consumption and the, and the hunger for the compute capabilities of your device.
- 8:32
So here is a full backend created. That means I just created a full backend, and here's the presented architecture of it, that first we have a Vault firmware deployed on the, uh, on the terminal, and there's a huge backend which, um, handle all of the, um, all of the agentic work with the OpenClaw and handle the LLM.
- 8:57
I just want to point out here, um, within my today's presentation, I just, uh, want to demonstrate you, um, the device connected to, um, uh, open source GPT 120 billion parameters model.
- 9:15
Um, and I just serve that model with the, let's say, most possible optimal way using, uh... And I, I, I use their TensorRT, uh, serving system. And I just exposed OpenAI style, uh, to, uh, also like LLM proxy because I just hit a lot of walls with the other open source models, um,
- 9:40
because that not all models match the style of the OpenAI API.
- 9:50
Um, as I said, uh, the first idea was to build and, mm, play with the OpenClaw through that terminal, and this is the stuff I just do with this.
- 10:04
Here you can look at the picture and see an example. I was po- uh, I was possible, uh, to hit a command, like write a Java example, uh, and store it on my local machine, and OpenClaw with the, with the LLM support just did work and make it happened.
- 10:28
Okay, and here's my field notes, the things that doesn't go as expected. Uh, lessons learned. Uh, first of all, the, uh, software I2C, uh, because I just wired up, uh, things there, and it doesn't work correctly.
- 10:48
Um, and there's a way to, um, to do proper control over, over I2C without any additional physical pull-ups mounted.
- 11:01
From other side, GPIO thirteen, um, there's a silent failure. Uh, there was a need to move to the other port, uh, to, um, to be sure that everything works correctly.
- 11:17
From other side, a need to build up the regulator, the power, mm, supply, um, the whole unit of the power supply, uh, because the, the regulator, uh, kills the OLED and, and fragile parts of the display, and it cost me a lot of time and, and, uh, getting the replacement parts
- 11:42
from the market took so long, a couple of weeks. And the last part, uh, quality of your parts. That means encoder cheap and low quality give me a lot of rotational noise, and there was a need to build up the, the pull-ups and to,
- 12:03
and to, uh, wire it up additional condens-condensators there.
- 12:08
But, uh, this is the funnest stuff. Um, my favorite one
- 12:14
Um, because the, this part really surprised me. Um, you already know there's a huge part called, um, RPG, and behind it is a bit sad maybe story. Um, because, um, I just never played RPG's game, like, on the paper.
- 12:32
That means with a, with a lot of friends, you, you can go to a, uh, to apartment, you know, open a book and make an, uh, someone a g- a game master, uh, and play in a real text-based RPG.
- 12:48
And after a lot of those years, I just never tried it, but I built, um, a RPG game and a console,
- 13:00
um, which give me a pure experience of, uh, text-based, um, RPG games, role-playing games.
- 13:09
And, um, actually, I- it's a bit funny, um, because those device,
- 13:17
it's really, like, perfect fit for that kind of gaming. Uh, uh, I build out an NPC and memory around that. I just created the mood of the world, the contents, omnis, and I just used all of the, you know, the LLM, um, advantages to build, uh, the best, uh, role-play game experience ever.
- 13:42
Um, I just created four different worlds. From one side, I just wanted to build something cyberpunk related. From other side, I just wanted something, uh, The Witcher part related.
- 13:53
That means some kind of fantasy world with dragons. But also my, uh, favorite one is the, is the void in, in a deep space somewhere, um, in the cosmos.
- 14:06
And, uh, this is really, uh, um, really good example how we can just use, uh, generative AI to build, uh, computer games. From one side, I just, uh, generate the characters, their m- worlds, maps, skills, uh, and all of those converted with the mechanics I just described at the beginning of that, uh, presentation with the one
- 14:31
bit, uh, memory allocation. That means all of the pictures transform to the, uh, to the m- matrices, to the, uh, to the, to the maps.
- 14:42
And, um, one, um, important, uh, important thing, actually, uh, the device is really bulletproof. Um, if the OLED doesn't work, you... the E-paper works. If the cardboard doesn't work, that means the keyboard, um, then you, you have an encoder.
- 15:04
Wi-Fi falls, you always can use your, uh, local, uh, shell. Um, and, um,
- 15:15
funny stuff, the area of the intellectual property there. I just, uh, do some kind of research in the field, and everyone just want to build the devices around the audio interfaces, the video capture, you know.
- 15:32
For... But for the quiet places, when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, LEDs, you know, popping up with the colors and all of the, uh, distractions, commercials, web pages, and other things.
- 15:54
Sometimes you just want to sit down and, and, you know, do, do the right job with a, with a, with a OpenClaw or, or, or, you know, just chat with your LLM, read and write, uh, in a quiet environment without any of the distractions.
- 16:10
And there's a niche on the market. Uh, so I also decided to build a- the provisional patent here in that field, and maybe it will brings, uh, commercial device in the future to a,
- 16:25
to a market. Some numbers. Mm, in my project, uh,
- 16:33
I just pushed one hundred and th- thirty commits, around three months of work.
- 16:41
Um, two dip- displays. Each of the display do other job. Four modes, four worlds for gaming. In general, sixteen, um, classes. That means you can generate RPG worlds inside of it.
- 16:58
Firmware is, is really tiny, really small, and works extremely fast. And, um, everything is powered by one single, uh,
- 17:09
um, polymer cell, lithium polymer cell. Um, but, uh, maybe
- 17:21
some remarks, some takeaways. Match the displays to the job.
- 17:26
Uh, there's a plenty of them on the market. Keep the model of the metal because it's really heavy. That means for now we don't have such a models which runs really on the super tiny MCUs.
- 17:39
In, it means LLMs. And track a narnative, narrative. The context really matters in LLMs, and in general, not
- 17:51
numbers. So here I also record, uh, the demo, the usage, how the device work, and some kind of,
- 18:03
uh, some kind of, um, last, um, last session. Um, please enjoy, and I will place a couple of comments there.
- 18:41
Device is booting, connecting to a local Wi-Fi network. Here's the welcome screen
- 18:50
and the main screen. Small display. [keyboard clicking] Keyboard for simple
- 19:09
help command. Here you have all of the things you can do with the device.
- 19:22
There's also a tutorial for gaming and tutorial for the device itself. Uh, and when you just switch it on for the first time, then you need to or can go, you, you don't have to, but you can go through the tutorial itself to learn how to use it. [keyboard clicking]
- 19:49
Usage of the OpenClaw agent. That means right now I'm just checking my DGX Spark
- 19:55
and want to look at the disk space.
- 20:12
Here is it. [keyboard clicking]
- 20:40
Let's maybe now look on the-- my favorite
- 20:45
part with, uh, gaming.
- 21:17
Farworld, Alderan, Neon Nabas, The Hollows, and Voidreach.
- 21:44
You can select your, mm, your character. And the LLM just generates the stuff.
- 22:08
Everything is generated there, um, pictures, uh, uh, personalities of the, of the people, and here there's a mood, there's a breach, you're in the deep space and can't
- 22:24
deep dive into the other world.
- 22:50
Yeah. So that's it.
- 23:12
Mm, that was the, the video. Funny part is that, uh, the game like this without the three-dimensional graphics use the most advanced NVIDIA, uh, processor, um, mounted in the DGX park.
- 23:30
That means the, the, the this is the backend currently. I just, uh, prepare for this demonstration. So the quiet game with the, you know, text interface requires the, the most powerful computer, uh, computer graphics card in the, in the world.
- 23:50
And maybe another thing is that, uh, the presentation was about the handle device to control, uh, agent and OpenClaw with the local LLMs. But in general, it was about the Game Boy, the device to, to play, uh, with LLMs.
- 24:12
Thank you for the attention, and thank you very much for the, mm, for the Callstack support because I'm just working at Callstack in the technological incubator, uh, and, uh, the whole work was supported by, by, uh, Callstack company.
- 24:32
Thank you very much.