Don't Build Slop (4 Levels of AI Agent Maturity)

Talk transcript

1. 0:00

   [upbeat music] Hi there.

2. 0:15

   Um, very nice to see you guys. Um, it's, it's, like, it's so interesting to see, like, it's been such a long day, and you guys still showed up. And, uh, I'm always very flattered when, like, I'm trying to present something and, like, people are there.

3. 0:29

   It just, uh ... I- I- It f- it makes you feel like what you do matters. So today, the topic that I wanna talk about is, like, don't build slop: Four levels of AI agent maturity.

4. 0:39

   And what I'm trying to do here is that I wanna ... I'm gonna talk about something that, like, a lot of people have brought this up to me, and there's like a mass psychosis problem.

5. 0:47

   So basically, it's something like ... It's like every person feels like, um, there's, like, this, like, giant set of robots around all of you that are just, like, kind of like breezing through, doing a lot of things.

6. 0:58

   And, and you're, like, in the middle, and you're so confused. You're so confused of, like, what is this? What, what should I do? Should I have, like, 15 agents, like, ripping through all the time and I'm just, like, vibing?

7. 1:10

   Or, or should I just be like a peasant and just, like, read through every line of code? And it's, it's very hard to, like, make up your mind. I feel like you come to a place like this, and you're always ...

8. 1:19

   Like, the formula just, like, gets to you. You get, like, a panic attack or something. But, like, I think, I think ... I wanna ... I kinda wanna pull back out of this, and I wanna be like, "Guys, let's slow down.

9. 1:28

   Let's just slow down, and let's think through, like, what are the problems that we can, like, actually solve that will, like, actually help you build, like, really useful agents?"

10. 1:36

    And I wanna a- take care of, like, every end of the spectrum of, like, your necessity to build things really fast, but at the same time, like, your necessity to, at a certain point, build things slow and take things to production.

11. 1:47

    So that, that's basically, um, the goal. To give you an example of, like, mass psychosis, like there's a lot of things that are very similar. So I'll give, give you example of, like, three UIs.

12. 1:56

    And these are three frontier labs. And I guarantee you, not one of you can predict which one is which. Uh, one of them is Factory, one of them is Codex, and one of them is Cursor.

13. 2:05

    I, I am positive none of you know which one is which. Even I don't know which one [chuckles] is which. But yeah. So that's why I say the point. Like, everything is same.

14. 2:12

    You, you wanna do your own thing. So to build agents in particular, uh, here's what, here's, here's what I would, would present here, that I wanna break down the problem of building agents into, like, four different parts.

15. 2:23

    The first part is just, like, trying to somehow figure out a way to, like, see if this even makes sense, like this even works. That's, like, where you use a framework.

16. 2:31

    The second part is, like, when you actually are serious. Like, okay, I actually wanna do something here. Now you build things by yourself, like, as, like, a state machine, like building an actual agent.

17. 2:40

    The third part is, like, the UX workflow, where you use Kanban, which I suggest is, like, a great, uh, form factor to be able to work with agents. And the fourth part would be, like, shipping to cloud.

18. 2:50

    So this is, like, mostly out- outline read of, like, how we wanna be able to, uh, work with agents, but I think it will give you a good heuristics of, like, how you, how you would, uh, solve this problem.

19. 3:00

    So level one of building agents is, like, literally just, like, just, just use a framework. And I, I think there's, like, a couple different frameworks, uh, like LangChain, LangGraph.

20. 3:08

    Like, I, I wouldn't ... I don't use them. I, I, I work at, like, client. I've, I've ... I'm supposed to, like, write all the agentic experience stuff by myself, so why would I use a framework?

21. 3:18

    So I won't be the best person to give you advice. But I do think that if you're trying to find PMF, if you have a problem that's like, "Hey, I have this problem where I wanna be able to, I don't know, aggregate emails or do something, like, rudimentary, and I think that, like, an AI agent would probably

22. 3:32

    be helpful here. I probably don't care about the best model. I just, I just want something that works." Any of the agent frameworks can give you something that just works in, like, half an hour.

23. 3:39

    You can just vibe code this. It, it, it's, it's a great thing to get started. It's a great thing to see that agents actually do work and that you can build them yourself.

24. 3:47

    There are a lot of pitfalls with, uh, using frameworks, and one of the biggest ones that I think is that, like, if you really want to take things to production, if you really wanna build something serious, very quickly you'll learn that, like, the level of customizability, the level of, like, futuristicness, the level of, um, modularity that you

25. 4:07

    need, you just ... you won't find them in a framework. I know a lot of people will disagree with me, and a lot of those people are wrong. Anyway [chuckles], um, so level two is, like, building, building your agents yourself, right?

26. 4:17

    So here's how you, here's how you build. You, like ... I feel like, I feel like this is, like, a very intricate problem. Like, I can't sum this down.

27. 4:22

    So I'll, I'll just, like, give you, like, five rules of, like, when you actually write code to build your agents, like, there, there are five rules that you could use.

28. 4:29

    Um, and these rules will, like, give you a rough outline of, like, how to, how to build and write, write code for your own agent. So the first one is, like, you always wanna think of every agent as, like, a state machine.

29. 4:40

    Um, state machine, it's, like, a sophomore year or freshman year topic. But, like, state machine is basically, like, every agent is, at, at the end of the day, is like a recursive loop.

30. 4:48

    That's, that's basically ... Like, every, every, every hype cycle, whatever you think, it's at the end of the ... It's all a recursive while loop. It's a while loop with a few conditions.

31. 4:55

    And no matter what your agent's doing, it, it always ... It doesn't matter it's Cursor, Clock, or whatever, it is a while loop with a few conditions and a few end states.

32. 5:02

    And what you wanna be able to do is that at any point of time, you wanna be able to have, like, a mental model of which point in the state it is.

33. 5:11

    So let's say you wanna be able to, like, say, um ... Okay. So let's say you wanna be able to, uh, I guess, read a few files and explain them, uh, through Clock code.

34. 5:22

    So it starts from the user task at the top where you ask it, like, "Read a few files." It goes to the state of, like, reading a few files in, like, the action tool where it reads the file.

35. 5:30

    Then it realize, "Oh, I have read the file. It makes sense." And then it will call the completion tool and just, like, complete. And the, the right, right thing at the bottom, task complete, is when the state machine finishes.

36. 5:39

    Uh, you can take this in a very complex way where you can, like, rip through the whole state machine for, like, eight to 10 minutes or even, like, hours if you wanted to.

37. 5:47

    Uh, but essentially, every agent is a state machine. And if you can visualize that as a mental model, then every time you're building an agent, it will be so much easier for you to think through that.

38. 5:55

    Um, the second rule is that every single thing you add to an agent risks making it worse. I think this is the hardest thing that we've learned, and it has been a very bitter lesson that a lot of agent builders have learned, which is that large system prompts, lots of different edge cases, lots of different, like, fancy

39. 6:12

    if/else logic, all of that for frontier model just, just makes them worse Just like, just, just get out of the way of the model is the, is the lesson that we learned, which is that, like, frontier models are so good at their job that the less instructions you give them, they actually perform better.

40. 6:29

    A classic example is that if you go through the Codex repo, the prompt for GPT-5 versus the prompt for GPT-5.3 is one-third of the size. Uh, part of the reason for that is that the newer models are so good at their job that giving them too many instructions and longer system prompts, it leads to a sensory overload

41. 6:45

    where they get so many instructions that they get overwhelmed and can't figure out what the right thing to do. Um, so I think the simpler it is, it's always better, and it's like you have to think that, like, every single thing I'm adding, I, I'll be very careful that I'm-- I, I hope to God I'm not making

42. 6:59

    it worse, and just always try to prune it down. We, we took it so far where we literally rewrote the entirety of Cline because we realized there was so much, uh, junk from the older, older versions of Cline.

43. 7:10

    Claude code has been written, I think, at, at least seven times from scratch, I think. Um, but again, um, the, the, the people of the team might know better.

44. 7:19

    Um, the third rule is that you wanna be able to make agent an easy part of a pseudo RL pipeline. And this is a tricky one, but basically what this means is that anytime you're building an agent, you wanna be able to have some sort of like a CLI kind of a thing.

45. 7:34

    The reason for that is that as long as you have something that can build and test the agent really well in the form of a CLI, um, you wanna be able to build things that are very easy to build and test with, um, other coding agents.

46. 7:49

    So right now there's this thing, there's this, like, interactive dance that's happening between AI and humans where at-- in the back in the day, humans used to, like, guide AI, like, do this.

47. 7:59

    And I think at this point, we're at a point where humans are being guided by AI. And this is, I think, is a part of that where, like, if you're as a human, you wanna be able to build things in such a way that, like, the AI can work very easily through it.

48. 8:10

    So that might entail writing a- writing, you know, AGENTS.md the right way using the right skills, and building like a CL- CI or, or CD, um, such that like the agent, other coding agents can easily build your agent, test it, make changes to it, and then test it end-to-end is a very critical part.

49. 8:27

    Because that way, if you wanna make changes, you can just let, let a long-running agent run through in a parallel thread. It'll make all those changes, test it, and you have the whole thing running.

50. 8:35

    But if it's harder to build and test, it, it would also be harder for you to, like, use agents to work on your agent. Um, super meta. Um, rule number four, don't build slop.

51. 8:45

    Guys, don't. Please, for the love of God, don't, don't, don't build slop, please. I think that, I think that there's, there's like, there's so much like throughput that you can get and so much, uh, tokens that can go through really, really fast.

52. 8:58

    I think that the best lessons that we've learned as like real engineers is that, like, it's really worth spending some time just like thinking through the architecture, thinking through the design and the outline of what your agent's supposed to do, making sure it actually makes sense and, uh, and just like actually at least spend some time reading

53. 9:16

    the code, even if you don't write everything m-- by hand. I think that that is like, for us, we found that that to be super critical because I think that it, it's at least the architecture point of building an agent has to be done by a human and has to be done like very thoughtfully.

54. 9:30

    Even if you're using like an agent to have a conversation with it, spend a lot of time like trying to think through like what the architecture, what the state machine is going to be.

55. 9:38

    Don't, don't just let like, uh, other models like rip through the code. Um, and then the f-- rule fifth is Frontier labs kinda wanna lock you down, and this is a tricky one.

56. 9:47

    A lot of people will disagree, but basically what's happening is that a lot of times when you are working with the APIs of Frontier labs, the APIs are trying to lock you down, and they make the interchangeability harder.

57. 9:58

    So to give a very precise example, the new set of models that have come out, say, Opus 4.6, uh, Gemini 3.1 Pro, 5.3 Codex, they have this thing called reasoning traces.

58. 10:08

    And reasoning traces are, um, a part of the cache, and they're also the part of the reasoning, uh, uh, test-time compute loop that the model does. And when you have conversations and back-and-forth conversations with those models, you wanna send the reasoning traces in the exact precise format that is expected.

59. 10:27

    If you don't, the response would still work except that the performance would be degraded, and you would have no way of knowing. And a lot of people are kind of missing on the massive performance gains that are coming from the new models because they're just like not using the APIs in the exact precise way that they're supposed

60. 10:42

    to be used. Um, and there are asymmetries in the API. So some would argue that, "Hey, maybe you could use OpenRouter," but I don't think that's enough. I think you really wanna be careful that like if you're using different Frontier labs and different, uh, Frontier lab APIs, like they, they have-- you very carefully thought through, uh, if

61. 10:58

    the API is working correctly and you've actually tested it. Um, so the form factor is like the next step is like the form factor of like how do you like, how do you visualize the agents, right?

62. 11:09

    So I think originally I came back to like in the one of the previous slides, I tried to show you guys like the thing where like Codex and, uh, Cursor and others, they were all looking the same.

63. 11:18

    And I think I have a different claim. So for this, uh, uh, on March 26th, I made a tweet, uh, where I said we-- people should use Kanban. So Kanban boards, I think if anyone has used Linear, whatever, I'm sure all of you are familiar with Kanban boards.

64. 11:32

    So my argument is that Kanban boards are the thing, uh, to use. Um, Henzen in the audience was gracious enough to offer me his thoughts as well. So thank you so much.

65. 11:41

    So Kanban boards are, are, are this idea that like if you're working through an agent, you're always, uh, inference bound. A lot of you are working through like Codex or Opus, and it's working for eight to ten minutes at a time.

66. 11:53

    When one of the agent is working for eight to ten minutes, what do you do? You could doomscroll, but you can only doomscroll, scroll long. So then you run another agent, right?

67. 12:03

    So that way you have at least like two or three agents running in parallel at all times because you're inference bound and they're all like mutating the same source potentially.

68. 12:12

    So you wanna isolate the thing that they're mutating, uh, to take care of that, like the, the, the isolation of state and then the inference bounding. The best UX form factor to me is Kanban board, mainly because it takes care of-- it gives you like-- it makes you like an engineering manager which can look at all your

69. 12:29

    agents. And you ... Gives you, like, a headline-level view of what they are, and it also helps you build, like, flows with them that like, okay, if these two tasks fir- finish first, then I'll do the other task.

70. 12:40

    And that helps you, like, become, like ... You, you basically become like an engineering manager, and all your agents are your ICs and, and you can, uh, w- uh, look at them through this.

71. 12:50

    Um, so I was making this claim on March 26, and 10 hours ago Cloud Code came out with the same thing, so I believe I was right. Um, um, okay.

72. 12:59

    So yeah, you can use that through Cloud Code, Cline, wherever. You can use Cline as well. Whatever, uh, whatever works for you. Um, so yeah. So you wanna think of, like, ca- uh, Kanban as basically like an engineering manager, which y- you would be.

73. 13:11

    Then there's, like, a final step of, like, okay, you have, you have, like, an agent. You've tested it, you made it, and you, you have a good UX form factor to be able to, like, interface with it and look at it.

74. 13:23

    Uh, what do you, what do you do then? Like, how do you, how do you, how do you, how do you have an agent that's just, like, useful, it works well, but it, it scales?

75. 13:32

    It really scales to, like, millions of tasks, millions of users. If you're working for a company which has, say, 8,000 people, how do you, like, how do you make sure that, like, all of them can, like, very easily interface with this?

76. 13:43

    And I think that rather than, rather than, like, making people install and, like, you know, s- having these complex workflows in the machines and stuff, it's just, it's just so much junk and it's so hard.

77. 13:53

    Just, like, take it all, take it all out. Put, put the hard work once. Just take it all on the cloud. Um, so there are many benefits of cloud agents, but the primary one is that, like, you can completely paralyze and have, like, a separate machine for each of them.

78. 14:08

    There are, like, no local dependencies. Like, um, in the cloud, like, the agent can set up the environment, do all the UX tasks. Because right now, one of the most missing pieces, the thing that a lot of people are not using, that I use fairly extensively, is cloud agents, because they can really run, uh, really long.

79. 14:26

    So I, I on my phone often would send tasks that would run on a cloud agent for like 15 to 20 minutes. So let's say it's, like, a UX change of, like, um, go build this, uh, VS Code extension, and this VS Code extension I want you to sign in here, I want you to click on settings,

80. 14:42

    I wanna change the settings to, um, uh, pick this theme, and then I want you to test this thing in the terminal. And the cloud agents are so good that they will actually manually do all the clicks of the Q&A testing that I just described on their own, figure out if they worked, and if they didn't work,

81. 14:59

    it will just keep iterating, keep trying. And this thing could easily take, like, 50 to 60 minutes. But if you send, like, a lot of these tasks in parallel through your phone or through your laptop or whatever, whatever, um, I think you, you have, like, you have this, like, really easy, customizable, extensible thing, and it helps to

82. 15:18

    scale really fast. And then you could send the, all these tasks, like, running in, like, the, like the cloud machine. Like, you know, like a messiah. And you could come back to your laptop, and then it's like, oh, you could just pull down the PR, and then you've got the whole thing going.

83. 15:32

    Um, the other aspect of, like, cloud, cloud agents is that, like, if you're working with a lot of different people, um, I think that it's just, like, it helps you build, like, a common setup that, like, so many other people can share and so many people can mutate.

84. 15:46

    Um, so I think bringing that together, I think would be, like, the final form factor. So my f- my claim is that, like, there exists a future where, um, most of the UX of working with agents would be Kanban, and then most of the actual compute, uh, that's involved with, uh, with agents would be on the, on

85. 16:04

    the cloud. Um, I think these four levels of frameworks are just, like, some ... The last part of, like, you know, Kanban and shipping to cloud, like, those things are very difficult and very intricate problems, and I think that if I were you, I would just, like, use this, like, as, like, rough heuristics and start with, like,

86. 16:19

    okay, let me just, like, let me just do the bare minimum easy thing and then just, like, depending on how much effort I wanna put in, I will, like, slide up and down, um, up and down these levels.

87. 16:29

    So yeah. So that's it for me. And yeah. So, um, I, I made a lot of hot takes. I feel like I think I left a lot of, like, open-ended questions here.

88. 16:38

    If you have any questions for you, any thoughts, this is my Twitter. You're very welcome to reach out to me, send me any questions. And, uh, yeah. Um, it was very, very kind of you guys to give me your time.

89. 16:49

    Uh, thank you so much. [audience applauds] May I take a photo of you? All right.

90. 17:00

    Okay. All right. Thank you. And do you guys have any questions? It's, like, I've got a minute. [singing]

91. 17:13

    All right. This is it. [laughs] Oh. Yeah.

92. 17:16

    Um, how do you do planning inside of a Kanban? Uh, like, uh, I, I find the sort of, like, back and forth requirements gathering, you know, the, the, uh, getting the agent to figure out what it wants from me to be the most useful part of the-

93. 17:30

    Oh, yeah, yeah, yeah. Yeah, yeah.

94. 17:32

    So how do you handle that?

95. 17:32

    Oh, yeah. Let me, let me show you. Let me show you. So basically, like, I think my interpretation of, of, of Kanban is that, like, it's just, like ...

96. 17:40

    If you see the screen, like, you can go into any task, and it will give you, like, the entire trace of the task. Uh, and that point, it's like the ...

97. 17:48

    So this, this interface that you're looking at is basically, like, um, the actual CLI of Codex. And I think I would ... I, what I, what I usually do is just, like, have a conversation here, and then I know at a certain point that, like, "Bro, you can go out on your own, do your thing."

98. 18:02

    At that point I'll just pull out and just focus on other things.

99. 18:05

    And then d- does it transition state when it either needs your review or asks you a question?

100. 18:09

     Yes. Yes. Yes. So, so initially the phase, like ... It's like, let's say I would say, like, uh, if I say read a few files or whatever, right? Um, so it will be, like, initially it's in, in the in progress state, and then when it's, uh, when it's, like, it needs my input, it will go through review.

101. 18:25

     Um, yeah. All right. Thank you. [upbeat music]