OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents

Talk transcript

1. 0:02

   Probably all of you once planned the cleanup activities, and you can imagine I also planned one weekend to

2. 0:11

   cleanup of my basement, and I just, uh, went there, uh, and want to clean up all of the stuff there, you know, throw up, uh, old washing machine. And I just look at the corner there and realize there's something weird there lying on the floor.

3. 0:34

   So I just grab it, look at the, the strange device, looks a bit familiar, but a bit old, like from '80s, with keyboard, but with a weird displays, of course, covered with dust.

4. 0:51

   I just take it to my hand, look at it,

5. 0:56

   and I was completely surprised that, um, device have a shell and it's on.

6. 1:07

   So, um, I used my superpowers skills in a terminal, and just I provided there a simple question,

7. 1:20

   "Who am I?" And the device responds in the weirdest way I could even imagine,

8. 1:28

   in the natural we- language. So the story I just described to you, it can be a great story for a marketing.

9. 1:40

   But honestly, mmm, I just wanted to build a device which is physical and AI native, like the device which comes from the future.

10. 1:57

    Hello. My name is Lech Kalinowski. I hold a PhD in physics, and today I want to present you a device and a backend and a system I built over last couple of months.

11. 2:14

    And the whole story begins because I just simply wanted to build

12. 2:20

    a remote controller to my OpenClaw instance on my DGX Spark.

13. 2:31

    And, um, the story is simple because when you are just first think about, uh, LLMs, then you're not thinking about the audio, about the generative, uh, generative, uh, videos or pictures.

14. 2:50

    First, you're thinking about the text. And then I just get a bright idea that maybe I can use some kind of better display to read and type with my LLM.

15. 3:06

    F- of course, on the market, there is a electronic paper, but, um, I realized electronic paper can be a bit slow. Of course, it's perfect for reading,

16. 3:18

    but if you just want to provide dynamically text to it, you probab- probably need a faster display. And then I just, uh, used and, uh, in my design,

17. 3:33

    uh, a simple connection of a OLED display, the small one, one color,

18. 3:43

    with the electronic paper display. And then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to

19. 4:01

    play with my claw. And, um, of course, behind it, uh, there is, uh, some kind of lot of complications because from one side, I just wanted to build a remote control, but from other side, there is such a niche on the market,

20. 4:26

    and this is the AI-native operational systems. And when I just started to think more clearly about it, I just realized I can build such a system for my purposes, but not with the super powerful co- powerful computers, but with a small microcontrollers.

21. 4:48

    And I just design a system built in, let's say, with a, uh, with a, say, uh, four classes. One is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection.

22. 5:09

    And within my idea to build an AI-native device, I build an assist control and, uh, one thing which really surprised me, and I will explain you later, RPG mode.

23. 5:28

    And, uh, here it is. I started to prototype DGX Spark at the,

24. 5:37

    um, at the background here on the picture, two displays. One, it's a live surface. That means the dynamic part. You can type the whole text provided is written there.

25. 5:54

    And when you are just trigger action, hit Enter, then the whole render also happens on the second bistable electronical paper

26. 6:09

    But, um, there is a lot of complications and lot of tech and lot of steps you need to do to get an clear user experience and clear interface. So I just found out a lot of the, uh, different approaches how to render properly on the two displays, properly, uh, render the,

27. 6:34

    the, the, the interface. I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX Spark through the, through the OpenClaw.

28. 6:45

    And this is the... one of the approaches that you can just

29. 6:50

    have fixed static buffers and render one-bit images stored in the,

30. 6:59

    in the memory. That means the pages live in a pre-allocated memory. There's, uh, no markdown engine, uh, and no malloc on the MCU side.

31. 7:17

    I just wanted really to show you how the system is really complicated here, and this is, uh, not because I just,

32. 7:27

    uh, wanted to use that much components. It's because there is a huge need to build a, to build a power, um, management system,

33. 7:39

    uh, because I just blow up two displays over my, uh, prototype build, and, um,

34. 7:48

    I just, uh, needed to be sure that the current and the voltage is stable as possible. So here inside, you have an MCU, which is ESP32 dual-core microcontroller.

35. 8:03

    Here you have an OLED display, keyboard, and a encoder, and of course, the mentioned, um, power supply.

36. 8:16

    But the A Native end-to-end device is not, uh, let's say that simple because of the, you know, high power consumption and the, and the hunger for the compute capabilities of your device.

37. 8:32

    So here is a full backend created. That means I just created a full backend, and here's the presented architecture of it, that first we have a Vault firmware deployed on the, uh, on the terminal, and there's a huge backend which, um, handle all of the, um, all of the agentic work with the OpenClaw and handle the LLM.

38. 8:57

    I just want to point out here, um, within my today's presentation, I just, uh, want to demonstrate you, um, the device connected to, um, uh, open source GPT 120 billion parameters model.

39. 9:15

    Um, and I just serve that model with the, let's say, most possible optimal way using, uh... And I, I, I use their TensorRT, uh, serving system. And I just exposed OpenAI style, uh, to, uh, also like LLM proxy because I just hit a lot of walls with the other open source models, um,

40. 9:40

    because that not all models match the style of the OpenAI API.

41. 9:50

    Um, as I said, uh, the first idea was to build and, mm, play with the OpenClaw through that terminal, and this is the stuff I just do with this.

42. 10:04

    Here you can look at the picture and see an example. I was po- uh, I was possible, uh, to hit a command, like write a Java example, uh, and store it on my local machine, and OpenClaw with the, with the LLM support just did work and make it happened.

43. 10:28

    Okay, and here's my field notes, the things that doesn't go as expected. Uh, lessons learned. Uh, first of all, the, uh, software I2C, uh, because I just wired up, uh, things there, and it doesn't work correctly.

44. 10:48

    Um, and there's a way to, um, to do proper control over, over I2C without any additional physical pull-ups mounted.

45. 11:01

    From other side, GPIO thirteen, um, there's a silent failure. Uh, there was a need to move to the other port, uh, to, um, to be sure that everything works correctly.

46. 11:17

    From other side, a need to build up the regulator, the power, mm, supply, um, the whole unit of the power supply, uh, because the, the regulator, uh, kills the OLED and, and fragile parts of the display, and it cost me a lot of time and, and, uh, getting the replacement parts

47. 11:42

    from the market took so long, a couple of weeks. And the last part, uh, quality of your parts. That means encoder cheap and low quality give me a lot of rotational noise, and there was a need to build up the, the pull-ups and to,

48. 12:03

    and to, uh, wire it up additional condens-condensators there.

49. 12:08

    But, uh, this is the funnest stuff. Um, my favorite one

50. 12:14

    Um, because the, this part really surprised me. Um, you already know there's a huge part called, um, RPG, and behind it is a bit sad maybe story. Um, because, um, I just never played RPG's game, like, on the paper.

51. 12:32

    That means with a, with a lot of friends, you, you can go to a, uh, to apartment, you know, open a book and make an, uh, someone a g- a game master, uh, and play in a real text-based RPG.

52. 12:48

    And after a lot of those years, I just never tried it, but I built, um, a RPG game and a console,

53. 13:00

    um, which give me a pure experience of, uh, text-based, um, RPG games, role-playing games.

54. 13:09

    And, um, actually, I- it's a bit funny, um, because those device,

55. 13:17

    it's really, like, perfect fit for that kind of gaming. Uh, uh, I build out an NPC and memory around that. I just created the mood of the world, the contents, omnis, and I just used all of the, you know, the LLM, um, advantages to build, uh, the best, uh, role-play game experience ever.

56. 13:42

    Um, I just created four different worlds. From one side, I just wanted to build something cyberpunk related. From other side, I just wanted something, uh, The Witcher part related.

57. 13:53

    That means some kind of fantasy world with dragons. But also my, uh, favorite one is the, is the void in, in a deep space somewhere, um, in the cosmos.

58. 14:06

    And, uh, this is really, uh, um, really good example how we can just use, uh, generative AI to build, uh, computer games. From one side, I just, uh, generate the characters, their m- worlds, maps, skills, uh, and all of those converted with the mechanics I just described at the beginning of that, uh, presentation with the one

59. 14:31

    bit, uh, memory allocation. That means all of the pictures transform to the, uh, to the m- matrices, to the, uh, to the, to the maps.

60. 14:42

    And, um, one, um, important, uh, important thing, actually, uh, the device is really bulletproof. Um, if the OLED doesn't work, you... the E-paper works. If the cardboard doesn't work, that means the keyboard, um, then you, you have an encoder.

61. 15:04

    Wi-Fi falls, you always can use your, uh, local, uh, shell. Um, and, um,

62. 15:15

    funny stuff, the area of the intellectual property there. I just, uh, do some kind of research in the field, and everyone just want to build the devices around the audio interfaces, the video capture, you know.

63. 15:32

    For... But for the quiet places, when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, LEDs, you know, popping up with the colors and all of the, uh, distractions, commercials, web pages, and other things.

64. 15:54

    Sometimes you just want to sit down and, and, you know, do, do the right job with a, with a, with a OpenClaw or, or, or, you know, just chat with your LLM, read and write, uh, in a quiet environment without any of the distractions.

65. 16:10

    And there's a niche on the market. Uh, so I also decided to build a- the provisional patent here in that field, and maybe it will brings, uh, commercial device in the future to a,

66. 16:25

    to a market. Some numbers. Mm, in my project, uh,

67. 16:33

    I just pushed one hundred and th- thirty commits, around three months of work.

68. 16:41

    Um, two dip- displays. Each of the display do other job. Four modes, four worlds for gaming. In general, sixteen, um, classes. That means you can generate RPG worlds inside of it.

69. 16:58

    Firmware is, is really tiny, really small, and works extremely fast. And, um, everything is powered by one single, uh,

70. 17:09

    um, polymer cell, lithium polymer cell. Um, but, uh, maybe

71. 17:21

    some remarks, some takeaways. Match the displays to the job.

72. 17:26

    Uh, there's a plenty of them on the market. Keep the model of the metal because it's really heavy. That means for now we don't have such a models which runs really on the super tiny MCUs.

73. 17:39

    In, it means LLMs. And track a narnative, narrative. The context really matters in LLMs, and in general, not

74. 17:51

    numbers. So here I also record, uh, the demo, the usage, how the device work, and some kind of,

75. 18:03

    uh, some kind of, um, last, um, last session. Um, please enjoy, and I will place a couple of comments there.

76. 18:41

    Device is booting, connecting to a local Wi-Fi network. Here's the welcome screen

77. 18:50

    and the main screen. Small display. [keyboard clicking] Keyboard for simple

78. 19:09

    help command. Here you have all of the things you can do with the device.

79. 19:22

    There's also a tutorial for gaming and tutorial for the device itself. Uh, and when you just switch it on for the first time, then you need to or can go, you, you don't have to, but you can go through the tutorial itself to learn how to use it. [keyboard clicking]

80. 19:49

    Usage of the OpenClaw agent. That means right now I'm just checking my DGX Spark

81. 19:55

    and want to look at the disk space.

82. 20:12

    Here is it. [keyboard clicking]

83. 20:40

    Let's maybe now look on the-- my favorite

84. 20:45

    part with, uh, gaming.

85. 21:17

    Farworld, Alderan, Neon Nabas, The Hollows, and Voidreach.

86. 21:44

    You can select your, mm, your character. And the LLM just generates the stuff.

87. 22:08

    Everything is generated there, um, pictures, uh, uh, personalities of the, of the people, and here there's a mood, there's a breach, you're in the deep space and can't

88. 22:24

    deep dive into the other world.

89. 22:50

    Yeah. So that's it.

90. 23:12

    Mm, that was the, the video. Funny part is that, uh, the game like this without the three-dimensional graphics use the most advanced NVIDIA, uh, processor, um, mounted in the DGX park.

91. 23:30

    That means the, the, the this is the backend currently. I just, uh, prepare for this demonstration. So the quiet game with the, you know, text interface requires the, the most powerful computer, uh, computer graphics card in the, in the world.

92. 23:50

    And maybe another thing is that, uh, the presentation was about the handle device to control, uh, agent and OpenClaw with the local LLMs. But in general, it was about the Game Boy, the device to, to play, uh, with LLMs.

93. 24:12

    Thank you for the attention, and thank you very much for the, mm, for the Callstack support because I'm just working at Callstack in the technological incubator, uh, and, uh, the whole work was supported by, by, uh, Callstack company.

94. 24:32

    Thank you very much.