A Piece of PI – Embedding The OpenClaw Coding Agent In Your Product

Talk transcript

1. 0:00

   [on-hold electronic music] All right.

2. 0:16

   I was introduced to Pi by, uh, um, looking into Open, OpenClaw. There was a conference, uh, a meetup, and I said like, "Okay, we're doing OpenClaw." And I wasn't so much interested into, like, all the craziness things that people are doing, but I was more interested in understanding, uh, of how these things work.

3. 0:36

   So I was looking into, uh, Pi and, you know, uh, understand the, the whole world of what Pi is able to do. [lips smack] Um, this is the one picture you need to take.

4. 0:46

   Please feel free to take more pictures, uh, but all the slides and the examples are there, uh, so that's the one slide. [lips smack] All right. Very quick, uh, about myself.

5. 0:57

   Uh, we're creating a small company, uh, TAVON.ai. We're building agents for, uh, organizations, small out of Europe, uh, but getting started. And, uh, what I really like, um, about...

6. 1:11

   Uh, sorry. Uh, what I really like about, um, uh, uh, Mario's talk is this, this quote, uh, you probably have seen, uh, this this morning. "We are on the," uh, "we are in the fuck around and find our own phase for coding agents."

7. 1:26

   Right? So everything that I'm gonna show you is what I know today, right? And, uh, I'm gonna do the talk again in a couple of weeks, and it's gonna be l- most likely be different.

8. 1:37

   Uh, but, um, as, as Mario was showing this morning, um, he has created this minimal set, right? This, this coding agent that is available, um, uh, for, for you, uh, for you guys to, uh, to fool around with, and that's what I'd like to encourage you. [sniffs]

9. 1:56

   So coding agents and why is it so exciting for us to build more products? This is Ken Thompson, um, inventor of, uh, Unix, and this is the famous quote by him, uh, one of the quotes, "Write programs that do one thing and, uh, one thing well."

10. 2:12

    And, um, I really like that because that's, that kind of, like, works, uh, to our advantage with agents. And, um, the best part where I show this is with CoWork.

11. 2:21

    So this is CoWork, uh, Claude's desktop. Um, and they're basically are bundling their coding agent into something where they feel is more applicable. Um, and to be honest, I've seen very good receptions around this.

12. 2:35

    And when you use it, uh, with, uh, financing tools, with their finance tools, you always need to work with Excel, right? So, uh, they have this Excel skill now and there, um, and it talks to, uh, Excel, right?

13. 2:49

    Well, it doesn't. Uh, instead, it uses a c- uh, a set of small tools, small CLIs, um, uh, pandas, uh, OpenPyXL, uh, stuff from LibreOffice, and package this into their own skill, uh, to make it, uh, up and running.

14. 3:05

    And I think this is a great example to kind of get your going, get your thoughts going of what, what is doable. [lips smack]

15. 3:12

    Um, I haven't written a book, and nobody can write a book about this, right? Because there are no patterns, right? We need to figure this out. We're seeing some emerging patterns in the coding space, right?

16. 3:25

    There's obviously tons of different coding agents, and we're seeing this, but there's no authoritative resource around this, right? So get going. One thing, uh, when I was talking to Ivan yesterday, uh, we realized is, like, one architectural pattern that we're seeing is that make it easy for coding agents, right?

17. 3:44

    Now, that is very broad, but think about it, right? Like, like, make not-- Don't try to be, you know, very, um, complex and things, but think about the, the coding agent, uh, what is it good at, and how do I build my system so that the, um, agent is easy, make it accessible.

18. 4:03

    And I, I have some examples. All right. This is the rough agenda, uh, for the next ten minutes or so. Um, I'm not gonna talk too much about Pi in OpenClaw.

19. 4:14

    Uh, I have a two slides. Slides are online, so we'll take it from there. So again, very brief, uh, introduction of Pi. Um, uh, Mario, uh, uh, great work.

20. 4:24

    Something he didn't mentioned is that he's joining Earendel, uh, which I think is awesome. Uh, it seems like, uh, uh, great, great folks working together and, uh, yeah. It's open source, it's minimal, so it's, it's just perfect to get started.

21. 4:41

    And the other part that I do want to re-empha- emphasize on is, is give it a try, right? We're gonna talk about a little bit different, but open up Pi and ask it to build what you want, right?

22. 4:53

    It's amazing of what it, what it actually is able to do by the system prompt, uh, that Mario has shown.

23. 5:01

    All right. These are the extensions. Um, so again, uh, uh, all the extensions you can download, uh, uh, build yourself or download and, uh, yeah, ton, tons to explore.

24. 5:11

    All right. So let's going. This talk is not about the coding agent itself, so using it for your daily dev works, but what can we potentially do with this?

25. 5:20

    And the starting point are actually not coding agents, right? The starting point is, um, and I encourage you to do the, the same, is looking at the, uh, core agent itself.

26. 5:31

    And there's other SDKs, but, you know, we're, we're talking about Pi, so let's, let's, let's use Pi. And what is an agent? An agent is actually just an LLM agent that runs tools in a loop, right?

27. 5:42

    So you have some goals, you have some context information, AGENTS.md of, uh, uh, in many cases, and then you do, do co- tool calls, right? And you get some results, and, you know, you basically do it in a, do it in a loop.

28. 5:55

    Right? That's it, right? There's not, not much more. The rest is magic, trying to put it in your use case a little bit more, in the other use case a little, you know, in that direction.

29. 6:06

    So that's really it.

30. 6:07

    Right? So pretty please, uh, uh, don't like open the curtain, uh, play around with it. Now, with agent, um, uh, agent core, this looks a little bit something like this.

31. 6:18

    You have an agent class. This is all TypeScript. Um, you can, uh, you know, address all, all sorts of informati-information. You can prompt it, um, uh, with different information.

32. 6:29

    Uh, and, um, also, you know, you have an event system, so you know a lot of things that, that, that are going on. So, um, small example, uh, this is a CRM lead qualifier.

33. 6:41

    I don't know. I've started the CRM use case, uh, for my personally, and it, it just sticks around. So, um, terminal int-interface, obviously, uh, small, uh, TypeScript application, three, uh, uh, um, three files, really easy.

34. 6:56

    And you can see this, right? You have a couple of commands that you can execute and, you know, show me all leads and score them, right? So that's what we do, uh, show all leads and score them.

35. 7:05

    And here you see all these, you know, things that are going on under the hood, right? You see that, that the assistant is calling, uh, tools, that you get some results, and eventually, you know, you get some input.

36. 7:17

    Now, obviously, there's tons of things to do, but, you know, I've just vibe coded this away. Uh, uh, uh, and it's a good, again, good, uh, learning exercise. The system prompt, um, uh, you know, um, as you could imagine, right, you know, calling out the different tools at what you do, right?

37. 7:35

    So all pretty straightforward if you are, uh, building an agent. This is an example of how you inject here, right? So, um, we said we want-- we, we do call-- tool calling, right?

38. 7:47

    We reach out to this, uh, uh, uh, and call a specific tool. But for the agent for, for steering it more, right, you know, a typical hook would be before the tool call do something, right?

39. 7:58

    And in this case, we don't want to update a contact, uh, without, you know, checking something or, I don't know. You can imagine any types of authoritative, uh, uh, role-based access, whatever enterprise feature in here.

40. 8:11

    But basically, you know, uh, just before the tool call. There's another one, events. So we've seen these, you know, uh, uh, the stream, and you might have seen a little check mark there.

41. 8:21

    Okay, the tool call was, was fine and res- uh, returned some result. So again, we're subscribing to events. All pretty straightforward. And again, please give it a try. All right.

42. 8:34

    So this is simple agents. Others a-agents SDK, uh, are, are available. Um, and now we're moving through the coding agent. Now, what's, what's a coding agent? At the end of the day, it's really the same thing as we've seen, uh, before.

43. 8:48

    It's a, a, you know, normal agent, right? It runs tools in a loop. But now we have a runtime and some type of shell, right? Bash is-- it seems to be the, uh, the shell that, that everyone is using.

44. 9:00

    But we have a shell and a runtime to c- to start executing.

45. 9:04

    And now things are getting interesting. And now the, the, the magic of, of what you've seen with OpenClaw, uh, suddenly shines. Uh, um, Peter, uh, shared this, this example, uh, on some presentation where, uh, he, uh, sent a message to his OpenClaw and, uh, sent a voice message.

46. 9:26

    Now, at that time, OpenClaw, um, and I still don't know if there's any like, uh, special plugin, but at that time, OpenClaw didn't know anything about voice, about voice messages.

47. 9:36

    So what, what it did is it, it, uh, created and used different tools. Um, in the end, one of the tools was, uh, FFmpeg, right, on the local, local machine, and it started this.

48. 9:48

    And this was one of the tools, right? So from the outside, it, it looks like learning,

49. 9:55

    but in the inside, it's actually just another tool call that is available to the agent, and that's why these things make it so interesting. So, um, a-again, uh, the example here, um, now this is a little bit more sophisticated, but the, uh, important part, and, and this is the extension API, and, you know, please look it up

50. 10:16

    online. We'll-- We're gonna do two things or the, the things that I'm most is-- mostly interested is in, in session events and UI interaction, and yeah, uh, uh, look it up online.

51. 10:27

    But here's, here's the, the actual extension. Now, again, this is what you would... In a coding agent, you probably just generate by asking it. But here if, if we have a look, um, this is a CRM, uh, TypeScript, uh, a small snippet of it.

52. 10:42

    And basically, what we're now doing is we're doing the same example as before, right? And we have a new command called pipeline, right? So if you have the slash commands, and you have a n-new command called pipeline, and now we are able to-- we're loading all the context.

53. 10:58

    Um, and, uh, you see this little in, um, um, don't have the lines. Just below step one, uh, you can see a context UI select, right? So all of a sudden, we're not only interacting with the backend systems and, and sessions and the-- and of those sorts, but we're also interacting with the UI, and we're able to

54. 11:19

    select, right? And that, that's got, got me thinking. Um, so, right? So you have this, th-this command, and again, this is now just the coding agent, right? We're not talking about the core agent class, but, but this is how you would load up Pi if you just don't download the, the coding agent.

55. 11:36

    And now with this new extension, we have Pi, right? And we can start selecting things, right? So this is a simple, simple select here. Um, and, you know, you, you easily even have dropdowns.

56. 11:49

    Now, the important part here is these are extensions and the framework, uh, that currently Pi, um, has included is catered towards the use cases of a coding agent, right?

57. 12:02

    So we-- You know, there's lots of work and other things to do to make this ready for others, for other types of applications. But I hope you can see and understand the vision, where, where this is heading.

58. 12:16

    And, um, yeah, you know, this is all terminal, right? So you wonder how would this look like in the web. Um, it currently is not possible if you ask Pi to build something.

59. 12:26

    So I ask Pi to build something, right? And this is the web UI-- it would be a web UI, same command, same selection, all based on the same extension mechanism.

60. 12:38

    Now, um, there's a refactoring going on to make this, uh, better accessible and make it more clean. But I hope, again, it shows you a little bit of, of where the, where the things are going.

61. 12:49

    All right. Now, um, Pi and OpenClaw, um, is, um, is a special, special setup, right? So Pi and OpenClaw, what we have there, um, is that, that now we're not only talking about like, like, um, a, a single agent in a single session in a coding environment, uh, but now we have a multi-channel, uh, environment where, uh,

62. 13:13

    we have, um, you know, multiple threads going on, multiple agents going on. So there's a little bit more to it. Um, this is, um... And, and the interesting part, right?

63. 13:25

    That's, that's where, where I got started is, is like if you look into, um, you know, the, the, the packages, um, uh, the core packages of, of, of Pi, all of them are used in OpenClaw, right?

64. 13:36

    So OpenClaw has this, uh, uh, this function run embed, uh, uh, Pi agent, and it creates a session, right? So sessions, um, uh, Pi itself has a great session support, um, it creates a session agent and streams all the information back.

65. 13:53

    We have, um, the coding agent, which we just talked about. We have agent core as, um, uh, the other part that we talked about, and there's two other, uh, minor, um, or major packages, uh, Pi AI for the unified LLM abstraction and, uh, a terminal UI interface.

66. 14:15

    Um, there is, um, uh, OpenClaw has built its own plugin mechanism, and that's because, um, uh, you know, it's a different use case, right, and has different requirements. So you have a plugin support for a multi-channel routing, different or, uh, uh, um, provider orchestration, sub-agents, gateway support, yada, yada, yada.

67. 14:36

    All those things that you know by OpenClaw, but it's based around the core mechanics of, of Pi and, and, and leverages it. Cool. But, uh, one thing, and that's, that's, that's the like, the, the major gist I would like to bring across is like, okay, what do we do now with this?

68. 14:55

    What are our other options for us to do? And this is one of the applications we've, uh, been building, um, for, um, for a client. Um, and basically, um, uh, uh, the, the, the use case is a sales process.

69. 15:10

    Um, they get, um, uh, requests for proposals, um, of, of an, uh, ordering another, um, another system, right? Um, parts, parts being sold by that company. And we're taking all this coding agent, all, all of that we're taking away, right?

70. 15:28

    We're, we're, we're new, fresh, new thinking, right? And look at the process from the get-go. So, um, an email comes in, right? We, we, we monitor basically that inbox.

71. 15:39

    Then we have some gateway, because what we wanna do is we wanna forward this to different agents, right? So here I have, um, multiple agents, right? Uh, the way it's structured is we have one agent per customer, and that agent has a general harness, right?

72. 15:57

    Agent MD, uh, um, AGENTS.md as an example, but you can obviously also use different ones. And that helps, um, understanding the role of that agent. In this specific case, it, it tells of, of how to use the system and how to react to certain, you know, inputs, outputs, etcetera.

73. 16:17

    Now, um, the other one is Customer MD, where we basically explain the agent like, you know, the specific customer might have, you know, specific perks, right? Specific, um, uh, access, specific, um, um, discounts and all of that sorts.

74. 16:33

    And then, right, and that's what I said like earlier, I, I like using sessions. Then for each case, right, we're, we're, uh, creating and reusing existing sessions so we can back and forth, um, um, know what, what was previously talked about.

75. 16:47

    All right. So email comes in, we're looking at the gateba-- um, inbox, and we route this to these different agents, and now we have tools, right? So we have these different tools, uh, to talk to the CRM, to talk to the A- ERP, um, and get the right information out of the system for this agent to look

76. 17:06

    on-- like, like behave like, you know, maybe it has, you know, new contact information or, or that sort. And again, we make this available. We make it easy for the agents to access, right?

77. 17:17

    And our way currently is doing this with CLIs, right? So CLIs are-- agents are really good at using CLIs, so we make it available as a CLI. We put-- we make sure that the data is secure.

78. 17:29

    Uh, we have our own sandbox, and then we're creating the drafts again, right? So that's the system, and I hope by this point you basically understand like logically where these things, uh, uh, uh, fit together.

79. 17:41

    But how would this look like? Um, oh, one, uh, uh, final thing, right? There is always the question around, okay, sandboxing, etcetera. And, and to be honest, we're on the, uh, just on the, on the steps of, of getting there.

80. 17:54

    But if you've seen, um, NVIDIA's announcement, uh, around, um, OpenClaw, their policy, their open shell is really, really interesting. And, um, um, it's, it's, it's a way of-- it's one ways of securing an, um, an agent.

81. 18:09

    We're looking into this. Please do as well. All right. So how does this look like, um, to, to, to kind of like get you an understanding of, of how these things, right?

82. 18:19

    So here's the dashboard. Um, rather, uh, boring, but here's the in-- uh, the email, the inbox, right? So again, we see the, the email coming in, and yeah, we, um...

83. 18:30

    It's one of, one of many emails. Most of them are ignored, but this one is like the, the, the LLM call said, "Okay, I'm, I'm interested in this." And it is associated to a case, right?

84. 18:40

    We see the case, uh, up there. Now, this case is again, is an agent session, right? Uh, so we find the session and associate it to it.

85. 18:49

    Um, we then create a draft. Uh, so there's tons of calls, which I'm gonna show you in a second. But basically, the output of all that is a draft email that the user will be able to use, right?

86. 19:02

    So our thinking is, uh, let them user stay in, in email, let them stay in the, the inbox and drafts, and they don't even, you know, need to do a lot.

87. 19:11

    So this is more like an admin interface. They can stay in email. But basically, the output would be a, a draft generated. And how does that look, uh, behind, right?

88. 19:20

    We, we had the, the different sessions before, uh, uh, the threads, and this is the same thing, right? The assistant says, uh, well, apologies, this is German, but, uh, uh, now I'm looking at the articles.

89. 19:32

    It does different tool calls, right? It gets, gets results and does this in a loop to resolve, right? The end effect for, for the user is I'm looking at my inbox, there's a new email, it's associated to a case, and I get a new draft, which they can freely edit.

90. 19:49

    But, um, under the hood, we have all these, um, uh, agents working. All right. That's-- That's it for me. Um, again, um, here, here you'll find the slides. Um, key takeaways, please.

91. 20:02

    Coding agents are and will be a core building block, uh, for your software systems. I'm, I'm betting on it. A lot of people are betting on it, so please give it a try.

92. 20:14

    Pi is perfect for tinkering. Whether you like it or not, it's minimal. You can rip things apart and put things together. It's perfect, so please go tinker. All right.

93. 20:24

    Thank you. [audience applauding] [upbeat music]