LLMs for the working programmer. Become a 10x programming centaur today!

Talk transcript

1. 0:00

   [upbeat music] Hello.

2. 0:15

   Hello.

3. 0:15

   Are you all awake and you just have your coffee? That's great.

4. 0:18

   Whoo.

5. 0:19

   Uh, welcome, uh, to my workshop about LLMs for programming or, like, how to become a 10X engineer. [laughing] Which I'm, like, actually pretty serious about. Um, I put some handouts out, so I didn't print like two hundred of them.

6. 0:34

   But if you can, sit next to someone who has some. There's, like, a couple lying around in this part of the room. And in general, I encourage you to have a partner, two partners, because it's a workshop and it's, like, nice to share ideas.

7. 0:47

   Um, there's a URL that's to a GitHub repo which has nothing in it, but [laughs] it has the handouts as PDF and it has, uh, like a list of links or something like that.

8. 0:59

   I encourage you to join the Slack channel because of the size, I won't be able to really interact with individuals, but I will be able to through Slack.

9. 1:09

   Um, all right. The only prerequisites you need are an LLM. So I use OpenAI, just, like, the chat interface or, like, Claude Sonnet is great. But you can use...

10. 1:23

    Like, if you want to have fun and use, like, a small one on your laptop, you're welcome to. Uh, Fine.com is also great. Like, any- anything that... That's all you need.

11. 1:32

    Uh, no code, no Git fork, no nothing.

12. 1:37

    Um, and with that, I'm Manuel. I'm, like, a software engineer by heart. I knew I wanted to be a programmer when I was [REDACTED:age]. And when GPT-3 came out, or, like, Copilot, I was like, "What the..."

13. 1:51

    Uh, and started just, like, doing everything I do usually, um, after twenty-five years of being a professional software developer with LLMs and just, like, keep finding new techniques until today.

14. 2:03

    I kind of am not on social media. I don't really follow what other people are doing, so this is kind of, like, all homegrown kind of stuff. But always with the focus of, like, actually doing, like, e-commerce work, PHP programming, just, like, down-to-earth stuff.

15. 2:17

    Um, and so I'm gonna show first just, like, a few general concrete techniques. Like, I think a lot of programmers don't really,

16. 2:27

    uh, vibe very well with the technology, especially because the hype around it is like it's gonna replace you, um, us, which is kind of true. Um, [laughs] but so I'll give, like, a bunch of techniques to just, like...

17. 2:39

    to, to, to ease in getting into it. So who of you

18. 2:44

    are programmers? Cool. Is there someone who's not a programmer? 'Cause you can still stay, 'cause it's, uh- [laughing] Everything will still apply, actually. Um, who's using LLMs, Copilot...

19. 2:59

    Well, I'll put Copilot a little bit apart, but, like, LLMs code generation, uh, for their work?

20. 3:07

    All right, cool. I guess that's why you're coming to the fair, huh? Um, so after a bunch of general concrete techniques that I use, which I feel are pretty important, I'll go into the main concept of how I approach programming with LLMs, which is, like, treating them as translation engine, not, like, as something that reasons or something

21. 3:23

    that, like, writes code and can figure things out with agents. It's just like, oh, they're good at translating one type of language to another. Which is honestly what we do as programmers as well, right?

22. 3:35

    And we take, uh, GitHub tickets or whatever and we convert them to code. Um, or the other way around. When the code doesn't work, we create GitHub tickets. But, um, then a pretty important...

23. 3:46

    This is, like, a new concept that I haven't really played with, but it's so massive, is treating LLMs as, like, word simulators and what we can do with that, um, in the context of just, like, pragmatic programming.

24. 3:58

    And finally, a bunch of, like, what I try to distill is, like, what are the skills that we need in this new age as software developers to become really effective?

25. 4:06

    'Cause they're very, very different from what they were in the past. Uh, but some of them stay the same, which is also good. Um, so

26. 4:18

    what I'm gonna do is, like, I'm gonna go through these pretty quickly. I wrote up... Uh, I wrote them up on the handout, and the handout, you'll see these gray blocks.

27. 4:27

    They're concrete examples of just prompts. Like, I don't wanna paste twenty pages of GPT transcript that no one cares about. But what you can do and try to maybe it will work, just, like, paste these prompts into ChatGPT and see what happens.

28. 4:40

    And it... Like, there, there are some sequences which might or might not work depending on what the model outputs. But hopefully you can follow your way around. And there's links, which obviously don't work on paper, um, to some of the transcript I just did yesterday.

29. 4:54

    But really it's about you trying them out. Like, don't try to... These are not examples you should follow. This is not the way to do it. It's just, like, an example of applying what I really wanna get at.

30. 5:05

    Um, and I'll give you about ten minutes after showing you these things to basically communicate with your, with your neighbors, try to solve some problems that you have, come up with some, and just use some of the examples I show to get a sense of, um, of, uh, how these things work.

31. 5:25

    Um, so first technique which I see a lot of programmers not doing is not regenerating, right? You'll come in, you'll put some prompt in it, and it'll, like... Either it outputs something great and you're like, "Holy crap," or it outputs something that's buggy and you're like, "Eh, it doesn't work."

32. 5:38

    Uh, but if you re- regenerate ten times, you'll see, like, oh, it maybe works three times, and then five times it outputs something that's, like, almost working, and then two times it's outputting something that's completely random.

33. 5:49

    The API works better for that because you can up the temperature. Like, I realized ChatGPT OpenAI used to be much more wide in its outputs. Now when you regenerate, it often comes out with the same stuff.

34. 6:00

    Um- Second one is, like, instead of trying to correct the models and like, "No, this is wrong. This is not the right method," and stuff like that, just go back and edit either your prompt or if your UI allows it, like LM Studio or Libra Chat, like other front ends allow you to edit the LLM's response.

35. 6:18

    So if the LLM outputs bad code, just go back, fix the code, and then the LLM thinks it output the right code. Um, so [laughs] it's, it's, it's really useful.

36. 6:27

    Um, then a next technique is to often clear the context. Like, I don't think I do more than three, four prompts in a conversation, and then I just take the output at the end and just start a new conversation.

37. 6:39

    That tells me if, you know, my, my technique is, is robust. Uh, 'cause I can't... You know, if I have a 20-page transcript, I, I don't really know what's going on.

38. 6:49

    I don't know what the context has been summarized to. And usually by the end, you're just, like, in, in La La Land. Um, it works much better with the bigger models now, so, you know, maybe this doesn't apply anymore.

39. 7:01

    Um, but it's b- good practice anyway. Experiment with different models 'cause they all kind of have different vibes, and you get to feel these different vibes and how they change over time or, like, how smaller models might just need more coaxing but can actually do the same stuff than the bigger models.

40. 7:16

    And this is all through practice. Um, so like practice, practice, practice, practice. Um, I have the trouble that I just get rabbit holed when I have a task to solve.

41. 7:25

    I try to write the prompt that solves the task, and then I try to write the prompt that generates the prompt that solves the task. Um, but, you know, practice.

42. 7:34

    Um, and then this is some very concrete stuff. Uh, just make, like, custom system prompts that you can put into, like, ChatGPT, OpenAI or whatever. Don't worry about changing them.

43. 7:44

    Like, if you have a session today where you were gonna write PHP, just change the system prompt to be PHP-oriented, like which libraries you use, maybe what code formatting you use, uh, just because it saves, like, a ton of time.

44. 7:56

    Um, that's a very concrete technique. There's a prompt on how to do that. And I want... Once I get to this slide, maybe we can try it out. Uh, generate helpers.

45. 8:07

    Like, every time you have a problem, just ask the model to write a script to solve it, right? Like, often I put stuff in my clipboard, and I have to, like, edit it to put it back into my code, right?

46. 8:16

    Like, maybe the imports. I don't wanna always, like, have to remove the imports when I paste it into my ID. So I can ask, like, "AGPT, write a shell script that just opens the clipboard in an editor," and...

47. 8:28

    Or open the shell script and just, uh, open the clipboard and remove the imports at the beginning. It's like a simple, sad three-liner. Just generate it, and at the end of the day, you can throw it away or save it.

48. 8:40

    And this might be the most important technique, summarizing your transcript, and I want you to do that today to exchange with your neighbor when you try something at the end, just, like, summarize what I did as a README, as a how-to, as a wiki entry, as a RFC, and then share it with your neighbor so that they

49. 8:58

    only have to read, like, one clean document and not, like, 10 pages of whatever. Um, so I really want you to do that today if you're, like, working with your neighbor, which I really encourage.

50. 9:08

    At the end, summarize the transcript and just send them that. Even if you do, like, rabbit holes, I'm fine trying to solve a problem, right? Like, "Oh, what's this TypeScript bug that I have?"

51. 9:17

    At the end, you can summarize your search and just send it over and say, like, "Look, this is, this is how I solved it. Here's all the links if you want to look at it."

52. 9:25

    So that's something you would never do before when you were reading Slack, Overflow, and, like, Google searching. You would never send someone your, your Google search history. Um, but now you can.

53. 9:35

    And a final technique is, like, when I ask it to-- When I try to figure out how to do something, I use ridiculous domain examples. Like, instead of working on my app, which is often going to be kind of a meta app, right?

54. 9:48

    Like, if I do infrastructure, that's gonna be, like, programming jargon, and the model might get confused because I want to, you know, create a script that creates scripts or manages scripts.

55. 9:59

    So instead, I always ask it to do, like, uh, you know, manage dinosaurs or, like, zoo animals or, like, uh, y- you write an operating system about Tolstoy. And, and, and then you see in the output what is Tolstoy relevant is, like, what it fills into the pattern, and then when it says, like, kernel driver, you know,

56. 10:16

    okay, that's kernel driver. Uh, but if it says, like, kernel driver system function interface, you're like-- you're not sure if it's, like, a driver about functions, if it's the function of the driver, like, um...

57. 10:27

    So I have an example here, like, write an Ar- Arduino RTOS for remote-controlled T-Rex. Um,

58. 10:35

    and I'll show that on screen. And while, you know, for the next ten minutes, I want you to try out some of these things, like, you know, s- write some helpers, like, create some system prompts or summarize some transcripts that you have.

59. 10:47

    Research how to-- how bourbon, bourbon is made and summarize, you know, your research to share with your neighbors. Like, just try different things. And while you're doing that, I'm gonna watch the Slack.

60. 11:00

    If you have cool examples, you can also open PRs on the GitHub re-- on the, on the GitHub repo so that everybody can see them later on.

61. 11:08

    And I'll answer questions, and if no one has questions, I'll just do my own stuff in the browser that you can look at for inspiration.

62. 11:16

    Um, so before we get into that, like, maybe I'll just show you, um,

63. 11:22

    one of these, uh, of these prompts. And, you know, the code doesn't need to work, but it, like, gives me an idea of, like, different concepts. Um, can you read this?

64. 11:31

    Is this big enough?

65. 11:33

    Yeah. Yeah.

66. 11:37

    Um, real-time, preemptive, uh, prey hunting. I don't know, just mixing up words.

67. 11:49

    Um... This is amazing. The, the, the artifact stuff is just, uh, is just amazing. And here you can see, right, like, the tasks are prey detection.

68. 12:04

    I know, okay, like, it understood that the task name is something different than the task thing itself. Um, but so here I can tell that it used free RTOS and it's like, uh, no.

69. 12:15

    Implement an RTOS from scratch, and then you can just like rabbit hole, right? So gonna implement it from scratch, and then I'm gonna summarize the transcript, and then you can see how that works and how useful it is.

70. 12:26

    Um, so this doesn't necessarily need to work, right? But it gives me a bunch of stuff to look at and try to Google maybe. You know, I, I can just try it out and run it, um, but then I'm like, "Okay, well, where did it kind of fetch this from?"

71. 12:41

    You can probably Google it 'cause it's in the training corpus, um, and that gives you like a good start of, uh... These are a lot of boilerplate that you wanna change, right?

72. 12:50

    Um, and then maybe you're like, "Okay, well, I know how to do a, a timer interrupt. I will replace that with my own." You can edit the LLM's response to put in your timer interrupt and regenerate, and it'll be like, "Oh, this is how you write interrupts.

73. 13:02

    I'm gonna write interrupts that way as well." Um, and you have a really efficient thing 'cause it's still two, two interactions with the LLM, right? Like fast inference and stuff like that.

74. 13:12

    Summarize this as a README for my colleagues.

75. 13:16

    And then, um, this is like high value when you work on a team, and then you can just be like, "Look, this is how my RTOS works." I would never have written that by hand, uh, documentation of that quality, right?

76. 13:31

    You can say like add examples, list of caveats, only output new text.

77. 13:41

    Um, I will make many typos. The model doesn't care.

78. 13:48

    Um, all right. Your turn. I'm watching the Slack. Um,

79. 13:57

    if you wanna share ideas of stuff to build, like feel free to do it so that people who are,

80. 14:02

    you know, early morning, uh... Yeah. But can you later on ask in the Slack? That way I, I don't have-- I-- more people can do it. But, uh, go ahead.

81. 14:11

    I just wanted to say that it works, uh, nice for Mermaid charts. You can create a chart of your code and give nice shots on the, the artifacts.

82. 14:22

    Yeah. Oh, nice. One more generic technique. When you talk about doing something, just type that into the LLM and then we'll do it. Um, it's like programming at the speed of mouth.

83. 14:41

    Um, all right. And then...

84. 18:15

    Hi.

85. 18:15

    Hey.

86. 18:15

    Could you spend time how to join the Slack channels?

87. 18:19

    Uh, oh, yeah, sorry. The, the-- So the Slack channel, to join it, the Slack channel is called LLM Working Programmer-- Workshop LLM Working Programmer. Uh, sorry about that. Uh, maybe I'll just leave this slide on.

88. 18:31

    You have the handout maybe or, um-

89. 18:34

    Yeah.

90. 18:35

    -and I'll just mess around in a different tab.

91. 23:42

    All right. One, one minute, and then, uh, maybe I'll spend two minutes showing what I did.

92. 25:03

    All right. Uh, I hope you had fun. Feel free to summarize what you did, you know, in like a README or a little message and then share, share the chats in, um,

93. 25:17

    in, uh, in Slack so that everybody can see them or make a pull request to the repo.

94. 25:22

    And for the sake of time, because we have, uh, fifty minutes left and I have a lot to show, let's just, uh, move on. Is there any blocker? You all good?

95. 25:33

    All right, no Wi-Fi issues, no model issues, anything? All right, cool. Um, all right. So this is maybe

96. 25:42

    the main concept I use, um, which is no one knows how LLMs work and we never will. Um, because LLMs are basically these weird computational artifacts that have been trained on like whatever gigantic corpus we've trained it on, which no one has the time to read, let alone understand, right?

97. 26:01

    So no single human will ever understand what is in a model. You can just like kind of poke at it and connect it to what we know. And so in a way, it's not like a, you know, it's not like a computer technology or a math technology or whatever we wanna call it.

98. 26:15

    It's like a cultural technology. It's like based on human culture and if you know your way around human culture, you'll be able to squeeze out more of the model, not just programming, right?

99. 26:26

    So it's all about language. The model really doesn't care if you-- if it's programming language, if like a formal language or if it's like German or, you know, symbols or whatever because the only thing it does is transform tokens into tokens and it looks back at its own tokens doing so.

100. 26:41

     So even if we say like, "Oh, please output JSON," it's like, "I'm gonna output numbers, uh, or probabilities of all the numbers that exist." Um, but so if you make the switch away, right, like you're writing code, but like there's no reason that what is in the code isn't actually also like language.

101. 26:59

     Suddenly it makes a lot of sense of calling a variable like year-over-year revenue and not call it A even though as programmers we think like, "Eh, that doesn't really matter," right?

102. 27:08

     Like, uh, but actually to model it matters a lot because one is like the token A and the other one is the token year of year revenue and then suddenly it's able to access all the like, uh, finance literature to do-- to write your code even though you think that's like just, you know, um, a, a stupid

103. 27:28

     dashboard. So understanding that means that suddenly things like theater, poetry, marketing literature, like textbooks, mathematical formulas, all of that is like fair game to write code. Um, and so for example, theater, I bring that up as the first one.

104. 27:44

     The models are trained to be nice, right? Like they're, they're reinstructed to be like, "Yeah, I, I, I understand. Like please, uh, please tell me what to do. I'm, I'm happy."

105. 27:52

     And you lose a lot of conflict, which is useful. Like when you do a co-code review, I don't want someone to be nice in the code review and give me platitudes or like, you know, like s-shit sandwiches.

106. 28:03

     I want something that just like challenges every line I give it. And so if you say like, "Make a theater play code reviewing my code with like [REDACTED:age] different people just like arguing about it," then suddenly in one inference you have like a really good code review instead of someone telling me to handle my errors.

107. 28:19

     Um, so that's kind of the main thing, right? It's like thinking of them as culture, not as technology, uh, or as cultural technology. Um, and so the way to approach is I think of them as translation engines, right?

108. 28:34

     Like there's many, many different kinds of languages. Like when I talk to my mom I have a different language than when I give a talk. I have a different la-language than when I write Python.

109. 28:42

     I have a different language than when I try to write like a math paper. Um, and so some of them are like from less formal to more formal. Once you step into the formal language world you can use a computer to interpret it or do stuff with it more easily.

110. 28:55

     Um, but you can also, you know, transfer from one domain to another. You can go from the T-Rex domain to the real-time operating system domain or mix both together.

111. 29:04

     Um, it's just tokens, right? Um, but so what that means is writing software now is about decomposing the problem that you wanna solve into language translation steps and then let the model do these language translation steps, understanding which of these steps are too much for a certain model or how to do them, like how to prompt it

112. 29:23

     to say transform. You know, when it outputs JSON it doesn't output the right JSON you might put in like this is important for your career. That's like kind of a cultural technique to get it to output JSON which, which is a little bit weird as a programmer.

113. 29:38

     Um, so here's a bunch of examples and like I'm not gonna go through every graph but like feel free to look at them in the handout. Um, this is for example an example of how to transform meeting transcripts where like you're on Zoom or you have like your software planning meeting.

114. 29:53

     Everybody interrupts each other. It's like a big chaos. No one's gonna go back at that transcript and try to understand what was discussed. Uh, but now you can, right?

115. 30:01

     And so the steps to do that-- because if you just ask like, "Tell me what we discussed last week,"

116. 30:07

     they're getting good at that. Like, like it's a little bit scary but also it used to really work not very well because you, you have like twenty-eight pages of transcript with a lot of like just people interrupting each other and using colloquial language.

117. 30:20

     So the different steps to do it that I like doing is like first I wanna get a summary because Claude has gotten really good at like figuring out actually most of the information that's in there as a bullet point list.

118. 30:30

     It's not good at details. I just want like which topics did we discuss? Bam, five, five bullet points. Then I take this translation step, right, from transcript to five topics we discussed.

119. 30:41

     And then for each topic, I go back and I say, like, "For each of these topic, tell me exactly what we did." And now it has, like, a direction to know what to grep for in this huge transcript that's only relevant to it, so you get this, like, pretty good result.

120. 30:55

     People call that... Like, there's plenty of papers about it, I think, right? Like, with complicated names or whatever. The technique is really just translation steps. It's like from a topic to get all the details to that topic.

121. 31:06

     Um, no need to have, like, really fancy names about it. No need for fancy prompts. You just go back, paste the summary, and then for each point in the summary, like, give me the details.

122. 31:17

     Um, then the next step is once you have these technical details and you take the transcript again, you say like, "Oh, give me action points," like who's gonna do what, which we discussed in the planning meeting, and just give me a list of like, oh, Enrique's gonna do that, and Sarah said that maybe we should do this.

123. 31:31

     Uh, you know, and you get like a clean list of action points for each of these, like, fairly detailed technical summaries of each transcript point. If you do that one shot, there's no way that that will work.

124. 31:42

     Um, but if you decompose in these, like, little translation steps, uh, then you get it. At the end, you can just take the action points and the technical details we discussed, maybe paste some of your code base in there and create the GitHub issues, which already have like a plan laid out.

125. 31:56

     They're assigned to the right person. You can ask to add tags to it the way you do tags, like name them correctly with the whatever ticket naming convention you use.

126. 32:06

     Uh, so that would be an example of like a workflow for tran-transforming transcripts into GitHub issues. No need to really automate that. Like, it's, it's just a list of things you can paste.

127. 32:16

     And then in the conversation, if you're seeing it veering off a little bit, you say like, "Oh, no, please tell me what Enrique said about XYZ. I remember it."

128. 32:25

     I don't wanna script and, like, kind of formalize that. I'll just put in transcript tomorrow, forget about it. Um, so

129. 32:34

     I have a couple of scripts to help me do prompts, and I'll get back to those, but I zero-shot everything. Like, haven't done agents, um, whatever people call agents.

130. 32:44

     Um, so that was an example. Um, you can do other techniques, right? Like take this meeting transcript and the existing docs and code and just say like, "Oh, we talked about refactoring this thing.

131. 32:56

     Now give me a concrete plan. Like, tell me which API should be renamed to what based on what we discussed." I don't wanna do that by hand. It's like I need to look for the functions or whatever.

132. 33:05

     Just paste all the names, say like, "How do you rename them?" Write the script to rename them with said. You know, uh, [laughs]

133. 33:12

     you can add in, like, the technical details we worked out, right, from the summary. We had all these, like, oh, we discussed this way. Uh, you can maybe put in like your RFC guidelines.

134. 33:21

     You know, this is how we write RFCs, and then you get an RFC out of the refactoring planning that we discussed in the meeting. But with the right function names,

135. 33:30

     maybe with a script or an actual refactoring that you can put into a test branch, for example. And then you have an RFC that people can discuss, right? And then you can go back.

136. 33:39

     RFC comments, just put them back with the RFC and the technical details in the summary and just, like, update the RFC. Um,

137. 33:47

     so this one's a little bit more meta. Um, and I have to see how much time we have. Uh, this one is like, once we have the GitHub issues, right, from this transcript, I can formalize them because just, like, output them as YAML.

138. 34:01

     Like, make me title, tags, assignee, ticket number, link to your whatever. Um, and then you have-- now you have a formal representation that a, that, that a computer that we can write programs for, right?

139. 34:16

     So then ask it like, "Oh, make a shell script to just take this YAML and create the issues for real. Like, just use the GitHub API." Um, models are usually pretty good at zero-shotting that.

140. 34:25

     If they mess up the GitHub API like they used to do, just paste the API in there.

141. 34:31

     Uh, then it will usually work. And then you can take the script, which is pretty simple, but iterate on it, right? Like add a help, add a README, add more flags, uh, maybe confirmation, add like colors and little emojis, um, make like an NPM package for it that everybody can install.

142. 34:47

     That's all, like, little translation steps going from a JavaScript to the package JSON. It's like not really reasoning. A model can do it great. Uh, but when I do it, like I'm-- it's really annoying and tiring.

143. 34:59

     And then I'll, like, forget that you have to write, like, ID uppercase and ID lowercase, and it costs me the afternoon. Um,

144. 35:07

     so what you can do in three minutes, right, like, 'cause I was talking about it, but if the model-- If I just paste it into the model, I would be there.

145. 35:13

     I would have the script. It would probably work. Um,

146. 35:18

     now you can actually talk, okay, now that we have the script, how do we change our workflow within the team, within the company, without the open-- within the open source project to just automate it, right?

147. 35:28

     Like, we have a meeting transcript where it's like create GitHub issues. Boom, it's all kind of automated. You maybe want to look at the YAML before you press fire, right?

148. 35:38

     But there's a decent chance that at least you get a lot of boilerplate, and you can replace like a couple of, uh,

149. 35:44

     LLM fringe speak. Um, what you can do now is just like take the script and the Slack bot API and say like, "Make a Slack bot for this. Notify the people that have been assigned the tickets."

150. 35:58

     You take your GitHub transcript, paste it to the bot, and then everybody gets notified of their GitHub tasks that come out of it.

151. 36:06

     Uh, and then I did some meta meta stuff, which I'm gonna skip. Um,

152. 36:11

     if we go back to this workflow now, right, which probably took us like maybe an hour to build all of this, a good chance that, uh, you know, if, if it's annoying to get like the Slack access API token or if you like some OAuth stuff, you just like write me a script to get a Slack API

153. 36:25

     token. [laughs] I really don't wanna look up how to do it. Just give me the URL. Um, and but what you can do now is that as part of these prompts that we give it, every individual developer that you have on your team, they can like put in their preferences, right?

154. 36:41

     They could say like, "Well, I don't know about this library, so explain a little bit more when it goes about this-" ... thing. I like bullet points. You know, my background is this and this and this, but currently I'm busy with this other part, so only tell me about parts that overlap.

155. 36:55

     You can... That's just like a TXT you can have on your, on your, on your GitHub, right? Or somewhere. But every time the Slack bot, for example, is gonna create the GitHub issues assigned to you, it's gonna put this thing into the context and then you will have like tailored documents that explain how to do something.

156. 37:12

     So, you know, if you have the intern and you know the intern doesn't really know much about APIs or like HDP, you can just put in their prompt, like explain HDP as it relates to this ticket, uh, or explain our internal infrastructure deployment script.

157. 37:29

     Onboarding, super easy. Just have like an onboarding TXT that's ready to go explaining like the architecture of your code base. Just put that into the new developer's GitHub issues and make them match, right?

158. 37:42

     Um, easy. It's just language. Um, so for the sake of time, uh, you can go through some of these. Maybe I'll show, uh, one that's really funny is like, uh, do you wanna write API clients?

159. 37:57

     I just don't even look at the docs anymore. It's like I record, say I'm in a browser and I click on things, and I wanna write like an API for, for the Claude UI, which they don't want me to do, but they have.

160. 38:08

     I just record the entire session, clean it up a little bit, and then just say like, "Well, write the documentation for this

161. 38:16

     massive XGP request that I have." And once I have the API documentation, I'm just like, "Well, write me like a proxy." And then I have a proxy for Claude AI's newest thing.

162. 38:28

     Um, and, uh, let me see. This one's great for learning. You have a topic you wanna learn, get the search result, existing knowledge that you have, maybe some code base or preferences that you want, create a wiki entry for your personal knowledge base.

163. 38:45

     So you wanna learn about advanced TypeScript Lambda things, just look for it, take your existing knowledge, saying like, "My background's in Ruby. Give me some exercises, project ideas to build.

164. 38:56

     Um, maybe use the guidelines from my company to format this exercise so I learn two things at once. Create a prototype." And then maybe once you have a prototype, you just like paste your actual API and say like, "Well, just make it real," right?

165. 39:09

     And you get the actual implementation. But those are all... If I just said like [laughs] you know, "Solve the problem and teach me how to do this," it-- it's not gonna work.

166. 39:18

     But if you decompose it to these little steps at each point, not only do you get value of doing it because you see the output, but you also get like artifacts you can share, for example, right?

167. 39:27

     Like once you explain how Lambda TypeScript stuff works and your colleague also wants to learn it, you just give them the exercise, the transcript, you can help them. Um, maybe you put their personal prompt into it and their background's like in Lisp, and then it will just update it.

168. 39:41

     Um, all right. So you can take some of these graphs in the transcript. Um, I'm gonna give you like ten minutes really sharp so that I get to the mind-melting stuff.

169. 39:54

     Um, there is a section about DSLs. If you know about DSLs, if you don't, you really should look it up. LLMs are great at creating their own language, so you can actually create the target step itself.

170. 40:06

     You can say like, "Well, I wanna write reports, just like create a language to write reports." And then you can target that and, you know, inter- implement the interpreter for that language.

171. 40:15

     So I really... There is a example set of prompts in here about DSLs where I create like a text adventure game. I did that, you know, this morning at breakfast.

172. 40:28

     Um, try those out. It's like on page, page no number [laughs]

173. 40:33

     . But you'll find it. Creating adventure games, three lines of prompt, and I create like an alien-inspired adventure game with like, uh, full source code and stuff. So, um, yeah, try stuff like that.

174. 40:44

     Um, a good target is also self-contained HTML plus JS plus whatever your favorite library is.

175. 41:15

     Uh, there's a couple over here, and then if you have neighbors, I recommend just working with your neighbor as well, like sharing the handout and communicating. It's like, um...

176. 41:28

     Yeah, I think that was the last handout.

177. 41:33

     It's also as a PDF on the repo, uh, if you wanna see the repo again. Uh,

178. 41:40

     I guess it's in the Slack channel. So the Slack channel is LLM Workshop Working Programmer. [background chatter]

179. 44:10

     I'm gonna accelerate a little bit, like four minutes, you know. Uh, you should be able to write like five programs in that time. [laughing]

180. 44:20

     I-I'm serious, right? Like try out my prompts and just replace creator YAML DSL for the program you wanna write.

181. 44:42

     Where's the Slack workspace again?

182. 45:06

     It's, uh, called workshop LLM working programmer if you- Where do you find the link to it? Uh, if you're in Slack- That's the link Slack? Are, are you in the AI engineering Slack?

183. 45:14

     Yeah, just look for workshop, and then you should find it. Uh- That's a channel or? Yeah, it's a channel. Channel of the AI. Uh, yeah. Actually, let me put that back.

184. 45:23

     Here's, here's the Slack. Here's the GitHub repo.

185. 45:51

     All

186. 48:53

     right. I'm, um, I'm moving on 'cause I've got some really important stuff to show you. And one is-- So Josh-- Uh, no. Uh, Brenton just asked, like, "Have you found LMS able to work with PCAP files?"

187. 49:05

     I'm like, um- Not really, but, like, create a program that outputs the PCAP format in a format that, you know, in a language type that the model can understand.

188. 49:14

     So I said, uh, "Write a YAML DSL to represent interesting DNS traffic out of the PCAP file." That looks pretty useful, right? Like, this is language that the LLM can do well with 'cause it's called DNS traffic, destination IP.

189. 49:26

     It's not called, like, byte zero X5 or, like, DNS underscore

190. 49:31

     I don't know, whatever. Uh, and then I'm just like, "Write a Python file that takes a PCAP file and outputs .YAML." And now I have a script that I can give every PCAP file, outputs that into something that the LLM can use, and I can put it into my LLM agent loop or whatever.

191. 49:45

     Um, that's what I mean with 10X engineering, right? Like, in the three minutes here, I wrote another one 'cause, uh... So, um,

192. 49:57

     what Josh was asking, right, is like, oh, when I put in code base in my thing, like, what does that mean? It's, like, huge. Um, but usually when you interact with a code base as a human, you either interact with a small certain part, you read some docs, you maybe look at the function APIs.

193. 50:12

     So just create a YAML DSL to, like, grab your code base for interesting stuff, right? So, uh, maybe I wanna get all the functions in Python files as well as the classes.

194. 50:22

     So I said, like, "Invent a YAML DSL to create a program that finds functions and classes in my code base, um, as well as their docstrings." So it outputs this, and it's like, yeah, that's a cool idea, right?

195. 50:32

     If I don't like the idea, just regenerate until I have a y- language that matches what I want. Here, it added, like, recursion, which I like. Um, and I was like, "Well, I also want, like, the Markdown files."

196. 50:44

     So it, like, made up, you know, two things for the Markdown. And then my third prompt is just, like, "Implement it, please." And then now I have a Python file with a YAML that I can tell how to look for stuff in my Python code base.

197. 50:55

     And then, well, now I can run it on my massive code base, and I just have the function titles, the docstrings, the Markdown document titles, and that's a good context to give to my DSL, right?

198. 51:08

     Like, took me 30 seconds. Might maybe not work too much, right? Like, in real life, I would say, like, this is an afternoon project maybe to make it really nice with a README, with unit test, with, like, examples and install package, whatever.

199. 51:22

     Um, there is a graph on how to do this. Like, this is maybe one of my main techniques is, like, wr- writing these little fragments that I know are good context to do a certain task.

200. 51:32

     So some of them could be like, oh, write a new widget to do something. And then I'll actually add, like, grab for these widget names, maybe add the documentation.

201. 51:41

     Out of the Git log of last week, I'm gonna create a how-to, and then that's gonna be the how-to that I'm use for prompting. Um, and now I have, like, a little script that I can...

202. 51:50

     Every time I wanna write a new widget now, I just paste the result of that script in, so that's the live status of the code base. And then usually it does pretty well, right?

203. 51:58

     Like, if you give it, like, first write the props, write this and write this. Um, so this thing-- And I, I do this for third-party libraries, right, which often don't have, like, good documentation.

204. 52:08

     I'll go over their code base and just generate good documentation that I just need for, you know, for the task at hand. And then it takes me, like, a minute usually.

205. 52:18

     Sometimes it'll take me, like, the whole day, I'm like, "What am I doing?" But, um, pretty good technique. There's a graph on how I do it. Um, and now I'm gonna

206. 52:28

     quickly switch to the wild stuff. Um, so I went over DSLs. So word simulation, right? Like, everything you tell the model is what it thinks reality will be or whatever it uses as patterns to grab its training corpus.

207. 52:43

     If I tell it, "You are a wizard," it will pretend to be a wizard. If I tell it to write poetry, it will write poetry. If I tell it, like, time travel exists, it will say, like, "Sure, time travel exists."

208. 52:54

     So anything that's kind of in the training corpus, so things that humans can think up, so that you can think up, the model will, like, usually know what to do with, right?

209. 53:02

     That also applies to, like, formal languages. If you say, like, "Your world is, like, def function paren whatever," it's like, "Yeah, cool. Def function paren, that's my world." Um, so one thing you can do, for example, for code reviews, right, is, like, I tell it, "You're a reality TV show, Code Survivor."

210. 53:20

     And, like, each, each participant in this reality TV show is one of my variables or functions in this piece of code. Now have them, like, fight each other and figure out who's the bad one, right, and then vote them out, like, who's got the most bugs.

211. 53:35

     And, um- [laughs] So this is based on the concept that whatever you tell it it is, I'm gonna use what's in the training corpus, like reality TV drama. Um, so I'm gonna write some bad PHP code, 'cause all the code I write is great, so, um. [laughs]

212. 53:56

     And I have a GPT for that that I shared. Um, but really you c- you can, like...

213. 54:01

     Right, it's the idea, it's not the prompt. Uh, I don't really do any prompt refinement engineering except when I wanna repeat something in a, in a loop with a small model.

214. 54:12

     Um, I make typos and all that stuff. So now I have, I have this, uh, I have this bad PHP, right? I'll just paste it into my Code Survivor thing.

215. 54:19

     And this has multiple [laughs] So the contestants are, are, like, different variables. And I told in the format, in the prompt, you know, just, like, each one gets to say one thing and just, like, attack someone else, identify the conflict, and then there's some kind of format in there.

216. 54:39

     And, and one of the advantages is, like, A, I get conflict, right? Even though the model has been trained to be nice, this is, like, standard red team tactics to just exploit LLMs.

217. 54:48

     This works really well to do this stuff. Just have it pretend to be a different world. It allows me, back when ChatGPT was, like, really annoying and said, like, "Please fill in the rest...," dot, dot, dot, um, which it doesn't do anymore.

218. 55:01

     This would allow me to say, like, "Make [REDACTED:age] rounds," and it would do [REDACTED:age] rounds, right? Like, it was so strong in the prompt. That I could, in one inference, right, like compare that to code review my PHP.

219. 55:13

     This has already a lot of concrete information in it and I can... If I don't like it, I just say like, "Write three more things," and there is the comeback participant.

220. 55:23

     And it doesn't need to be a variable, right? It's like, uh, uh, SQL injection. It, it can be a concept. Like, concept can be a character. You can give them like a personality, um,

221. 55:37

     and then SQL injection will start to be [laughs] like dissing other people, which is pretty funny. Um,

222. 55:44

     but it's useful, right? Like, this is proper engineering. It's not... Like, it's not fun. It will actually tell me like, "You should use MySQLi close when you finish this and you forgot."

223. 55:54

     Like, "I'll vote you out." Um, and so once you have all this garbage, which I don't even really need, it's like funny to look as it scrolls by because they're like, "Hey, we're just...

224. 56:06

     Sanitation isn't our job." Whatever. Um, [laughs] and the... Oh my God, this just goes on for pages and pages, which was really hard for GPT to do back then. Like, it would be like, "Oh yeah, I think this variable's bad.

225. 56:20

     Like, please handle errors." And I'd be like, "What?" [laughs]

226. 56:24

     Uh, then you can do like write a sober code review report at the end, um, just based on everything that's in the context. Like now strip the reality TV part out of your language and just keep,

227. 56:37

     just keep that stuff, right? Um, and actually suggest fix, and I could then turn that into GitHub issues or whatever. Um, you get the idea. Um, but so this is the world simulation thing is that you can really make everything up.

228. 56:49

     Like one technique that I have is time travel because when you correct a model it's like useful information, right? It tells the model what not to do. But if you have 25 messages telling it not what to do, it will confuse it a little bit.

229. 57:02

     So I'll do my 25 messages. I'll get to the right code at the end. I'll say like, "Summarize what you did," and then... Thank you.

230. 57:13

     It will be like, um, "Summarize what you did," then I'll rewind the 25 things and I'll paste in like, "This is you from the future. This is what you tell your past self."

231. 57:22

     And now I have like this pretty good...

232. 57:25

     And it knows what to do with time travel, people coming back from the future telling like your young version what to do, and it, like it's able to generate better code out of that.

233. 57:33

     So [laughs] it's weird engineering concepts. Um, so try it out. Like it- this is not, like this is not fun. This, this actually works, right? Um, presidential debate is great.

234. 57:48

     Uh, Greek drama is pretty funny. Um, but where it gets really wild are prompts like, "You are the application I want to build. Please start." Right? And now suddenly it'll like going to pretend to be what you want to build and so you can tell it like, "Please output your UI as like a concise DSL."

235. 58:10

     And it's like, "Well, I've got like a main widget, I've got like a text area, and I've got this and I've got this." That's formal language it can already like visualize.

236. 58:18

     Um, but also it can be like, "Well, now I want a sidebar." And it's like, "Yeah, sure. You have a sidebar." Uh, so like, "Click the third element of the sidebar."

237. 58:25

     It will pretend that I clicked the third element of the sidebar, right? Um, so just to show you,

238. 58:32

     uh, you are an application to plan flight, travel, and sightseeing for me and my kids. Output your UI as a concise DSL YAML.

239. 58:49

     Start. So you know, regenerate if you don't like it. If you want it to be like React widgets instead of like this high-level description of what it is, uh, feel free to do that.

240. 59:04

     Like you can steer it, right? It can be like, "No, I don't want like my data, I want like actually the UI." Here I don't wanna rewind it. I'm just gonna correct it and then once I have the right thing I can like gaslight it and say like, "This is what you generated." [laughs]

241. 59:17

     Like, uh, "No, I want some HTML markup kind of YAML DSL with the actual UI."

242. 59:27

     Um, zero prompt engineering. Like I don't know what I'm doing, but you know, this is much more closely to what I had in mind. It's like, oh, I want the actual widgets, um, and if I were to, you know, write a JavaScript that parses this and then as callback handlers just like output press...

243. 59:43

     I press the button X, right? It's like, uh, if I wanna do like open the tab for day two, I don't think there's even tabs for it, but it'll just be like, "Oh yeah, I've got tabs like of course." [laughs]

244. 59:56

     Um, and so you can interact with it and it's kind of like is this code? Is it not code? But at the end you can say like, "Oh, write a spec," right?

245. 1:00:06

     It's gonna write like a concise README kind of spec and I can be like, "Implement yourself."

246. 1:00:12

     Um, which this is new. I haven't really played with it too much. I came up with this two weeks ago, but it's kind of wild because I'm into film photography.

247. 1:00:22

     I wanted an app with like three timers telling me what chemicals to mix in what, and I built it in five minutes and I was like, "This is wild."

248. 1:00:28

     Like I just said like, "I want an app to help me develop film with this process," and like filled it in with the right chemicals [laughs] and all. And I was like, "Nah, I want like two parallel timers that, you know, loop."

249. 1:00:38

     And it was like, "Yeah, sure." And then at the end I was like, "Write a concise HTML JS self-contained prototype," and then I can pass that on to the graphic designer, or maybe the graphic designer does that all day, right?

250. 1:00:50

     And he's like, "Nah, nah, I want a sidebar. Wouldn't that be nice?" And then tomorrow he'll just send me the transcript and I'll be like, "Okay, I'll make this work for real."

251. 1:00:59

     Um, so there we go. Um, this is you are an application. I showed you the code survivor.

252. 1:01:07

     This is your turn now. Uh, five, [REDACTED:age] minutes maybe. Just like invent an application, play with it. Like you're an application. Use a concise HTML markup YAML DSL. Start and then see how far you get

253. 1:02:56

     Um, I'll actually interrupt this. Like, I, I think you, you... afterwards you're, you're gonna have fun this, but I had a really interesting question come up is, like, how do I manage all of this, right?

254. 1:03:04

     Like, 'cause I don't know why every LLM company doesn't manage to make, like, a decent history browser. Like, it's really not that hard. Um, they could-- I could generate one in two days probably that, like, has tags and allows me to [laughs] search and, like...

255. 1:03:18

     But no one manages to do that. I don't know, I don't know why. Um, so what I use, and I would recommend not using my tool, right? Like, don't take my tool as the, as the thing.

256. 1:03:28

     I break it off and I tweak it to my things, but just build your own. Like, the tool to manage fragments is like you're managing text files and how to paste text files together, or maybe manage shell scripts.

257. 1:03:39

     So my most used tool is called Prompto. It... I literally generate it once. It's a hundred-line script. Um, it will go over all my repos, right? Like, I have a config files where I say, like, "Look into all these directories."

258. 1:03:51

     If it finds a text file or a script or a YAML template, it will just list these. It's horrible. It has no pagination. [laughs] It's whatever. But so once I have, like, a, a context, right, for, like, my library, when I want to do something with parameters, I have, like, a little script that gives me the current parameter

259. 1:04:10

     API, and I can call it, right? And it's like if I do Prompto get Glaze parameters, it will find... It will look into all these repositories, like, super inefficient, no indexing, no caching.

260. 1:04:23

     It will find this parameters MD file, and it will just paste it. Uh, or actually, it grabs the c- the, the Pyth- the, the Go thing and just outputs.

261. 1:04:30

     It's super ugly, right? Like, there, there's no structure in this except dash, dash, dash. But so now what I can do, I have another tool that just... It's really just like a command line

262. 1:04:42

     LLM prompter. It's like HTTP request with no format. Um, and I can pass it, this as a context, right? Uh, I've got a couple of, like, pseudo things which I wrote two years ago, a year ago.

263. 1:04:56

     They're not great. Um, I can print a prompt that it's gonna output, and it'll say like, "Oh, create a parameter

264. 1:05:04

     for URLs." Um, and so it will just paste [laughs] everything together in, like, a big blob, and at the beginning it says, like, "You're a great, you're a great Go developer."

265. 1:05:14

     Uh, actually, I messed it up. It actually [laughs] says PHP, so whatever. Um, and then it just, like, literally says, like, "Create a parameter for URLs," right? Um, in Go.

266. 1:05:26

     And then if I actually run this, it will have... It will use my API. It will do a good job. Like, this is how I just add stuff to my, to my code.

267. 1:05:35

     It will have the right packages. I can literally just paste this, and it will probably run. And then once I have this prototyped, then I have other scripts to, like, integrate this into the framework, like add the documentation style I want and, and all of that stuff.

268. 1:05:47

     But so this tool, pasting strings together and just, like, finding scripts in your... in a directory, they're, they're like nothing scripts, right? They're, um,

269. 1:05:57

     um... So that's how I manage all my fragments, and we have eight minutes left, so I'm gonna skip to the final part, which is more experimental. It's like what are these...

270. 1:06:07

     Like, you realize now that I'm, like, kind of not joking with this 10X, 'cause during this workshop, I wrote, like, [REDACTED:age], seven decent programs, right? Um, they maybe don't work too, too well, but, like, y-you can guess that they're not far from something that's of decent quality.

271. 1:06:24

     And then if I want to add unit test and whatever, I can, you know, I can take an hour at lunch, uh, while watching Netflix to actually do it.

272. 1:06:31

     Um, sometimes it doesn't work. So this is, like, a big part of it is, like, know when to step away. Don't... Right? If it doesn't work, like, within ten minutes,

273. 1:06:41

     don't force it. Either do it by hand like you're used to. It's fun. Like, we all love programming. Uh, maybe go for a walk and come back when you realize which language step it didn't understand.

274. 1:06:51

     Um, that's a really important one. I still get into full rabbit holes, um, where I just, like, don't get the task done that I actually wanna do. I just, like, generate programs that don't work to solve the task, and it's, uh...

275. 1:07:04

     So knowing when to step away, 'cause it's, like, breathtaking, right? Like, I generated seven programs. Like, what am I gonna do with that? Like, I can't mentally manage seven programs within a time sp- Like, I, I can't do it, and I, I really have this problem.

276. 1:07:20

     It's like I have, like, a, an experimental repo where I put in all my scripts to kind of... I don't even ma- I don't even bother Putting them in, right?

277. 1:07:28

     But I have, like, so many, and some of them are like [laughs] are like 5,000-line code things with, like, crazy features, and then I forget that I wrote them, and I write them again, and I go back, and it's like, "Oh, you already did it."

278. 1:07:40

     Um, it's like shopper-approved customer management things with, like, parallel work queues, and I, I wrote it three times. [microphone feedback] Oops, sorry. Um, so

279. 1:07:54

     what, however, doesn't go away, I think, is, like, fundamental and practical knowledge. Like, I know how a real-time operating system works, so it makes it really easy to see if the model doesn't get it, right?

280. 1:08:03

     Like, I'll be like, "No, let's-- don't do it in Assembly. Like, do it... Use this pattern or use this framework." If I didn't do it by hand or, like, learn about it, um, I wouldn't be able to do it.

281. 1:08:15

     It would go into a big rabbit hole. I wouldn't get an RTOS. But that's the difference between someone who knows RTOS and generates a new one with an LLM and can maybe do it in an hour and someone who will just, like, after a week, just get nowhere.

282. 1:08:29

     Nothing works. Nothing compiles. The Assembly is broken. So fundamentals, super useful. What you can totally forget is APIs, right? Like, I really don't care about Amazon's CloudFormation API to do XYZ.

283. 1:08:42

     Like, I really couldn't care less. It's not knowledge I wanna take with me into my retirement. Uh, and that's gone. But however, the knowledge of how to deploy is, you know, to do Coda's infrastructure for, for functions as a service, that's, like, really useful pattern knowledge, and I can take that knowledge and apply it to, like, DigitalOcean

284. 1:09:01

     and to, like, Amazon and Google. What I don't need to do is, like, figure out after the 10-minute deploys that I forget to put, like, an IM rule somewhere.

285. 1:09:12

     Like, I really, I really don't care. Some people do, like, and, and those are gonna have a hard time, I think, in the next few years. Um, what also-- while practice, I already went over it, um, but as soon as you think about humans in the loop, right?

286. 1:09:27

     Like, 'cause language is always going to be useful when interpreted by human. I don't care what the LLM... It could generate, like, 10,000 programs that don't work and, and, like, what are you gonna do with that?

287. 1:09:39

     So everything you generate with the LLM, either it's like to kind of help the LLM along to finally output something that's useful to humans, uh, or it's directly useful to humans, right?

288. 1:09:48

     Like a README is useful for humans. It so happens that because LLMs are trained on human language, it's also useful for an LLM. But if you make it nice for humans, it will work well with LLMs as well, right?

289. 1:10:00

     Like, if I have like-- if I go on for three paragraphs about how error handling is useful, like, that doesn't matter very much for my senior software colleagues. Uh, so I can strip it out, and now I have a smaller prompt, and it's, like, gonna be more focused.

290. 1:10:14

     So if you always think in this, like, language decomposition things of, like, how would it work for humans, how would it work for the user, what do I actually wanna do, um, that's, like, a good engineering skill.

291. 1:10:25

     It was already before, right? But now it's actually in the small is also pretty good. Um, divergent thinking, I, [laughs] I don't know how to teach creativity or whatever. Uh, smoke weed.

292. 1:10:37

     Um, but, like, thinking about, like, th-these weird things of, like, well, I'm gonna make, like, a reality TV show about my code review. I don't know necessarily how I came up with it, but it's the best code review prompt I ever f-found, right?

293. 1:10:53

     It's, it's-- I don't know. I don't know how to come up with that stuff. And what's useful there is to look at people who are not programmers 'cause they have amazing prompts usually.

294. 1:11:03

     And programmers are like-- we're, like, very focused on, like, oh, we know how to do it, and it's like you should write code. And then someone comes along, and it's like, "You're an alien that writes code," and, and it actually works.

295. 1:11:12

     Um, that's why people who have no programming knowledge are able to build, like, full apps while a programmer comes in and says, like, "Well, I forgot to do the for loop check, and this is worthless."

296. 1:11:22

     And in the meantime, the non-programmer is like, "Well, I've wro- I wrote an Android app that, like, allows me to film my daughter." And, um,

297. 1:11:31

     abstract thinking, I think a lot of it is, like, once you build something, knowing how to build the thing that builds the thing or find, like, the deeper abstraction of it and then focus that abstraction, I think, is pretty useful.

298. 1:11:42

     I-- my background is in Common Lisp a lot, so there's a bunch of languages like Common Lisp, Ruby, Haskell that are very focused on, like, creating abstractions, manipulating the language that actually solves the thing.

299. 1:11:55

     I think that's, like, a pretty interesting, uh, thing. Not that you should write Common Lisp or Ruby, right? But it's like getting familiar with the concepts that make up the community around that language of creating compilers or, like, creating, like, dynamic dispatch DSLs or finding, like, mathematical abstraction of how to do control flow or, or so.

300. 1:12:15

     That, that's really useful, and that kind of goes into language design. It's not just, um...

301. 1:12:22

     Like, programming language design is like any kind of language design. It's like our head-- Like, my favorite format is, like, title, one sentence, code example. That's, like, a language design thing.

302. 1:12:32

     I, it's, uh, it's how I like to have my docs. Um, so that's it. We have two minutes. Maybe

303. 1:12:42

     there's some questions. I can-- I guess I can take live questions and then repeat them maybe. Oh, there's a microphone. Great.

304. 1:12:53

     Thanks so much for the great session. Uh, it's amazing to be here in a room with like-minded people who we've been probably geeking out on all these things for so long, and thank you for sharing your, your findings.

305. 1:13:04

     Couple questions real quick. One of them is, are you finding yourself working with other people in your team kind of spearheading some of the incubation portions of work following these techniques, and then the caval- the cavalry comes behind you to build the actual thing?

306. 1:13:21

     Have you found yourself that kind of role? And then the second question is, I see you can push yourself going really far with all these techniques, but how much are-- have you tried using some of the Auto-GPT kind of metaphors as well?

307. 1:13:34

     I'm curious. Thank you.

308. 1:13:36

     Um, so I just recently joined a team. I was, like, kind of solo for a long time, and it's hard. I don't know. Uh, it, it's-- I, I don't know how to bring people on board with, like, how I do things, where I'm like, "I write 5,000 lines of code an hour, and then I throw them away."

309. 1:13:53

     Um, I don't really know how to bring that into the team currently. And then for the second question, I, I guess that's gonna be the last one 'cause I, I think the, the time is pretty strictly enforced.

310. 1:14:05

     Um, what was the second question? [laughs]

311. 1:14:08

     How much do you rely on Auto-GPT or those kind of-

312. 1:14:11

     Oh, uh, zero. Um, I just like zero-shot stuff, and then I'm in the chat API. So I don't, I don't use-- I, I use Copilot for, you know, like, just as, as Copilot is.

313. 1:14:21

     It's a very different thing. Um, and then that's it. It's a wrap. [audience applauding] [upbeat music]