Build Systems, Not Code

Talk transcript

1. 0:00

   Lately, I've been building agents a lot for operational tasks. And while I was working one Friday night, I saw the sun set,

2. 0:10

   and then dinnertime came and went, and it hit me. I was in that familiar dev flow, and the thrill of building was back.

3. 0:21

   Many of us who are coding with agents, we feel like this quiet sense of dread, like they're kinda taking all of the fun parts of building and leaving us with the unglamorous work.

4. 0:31

   But let me give you a little advice. Let them have it. Because if you go up just one layer, you'll find that the thrill is still there. When you're building agents, not just using them to write code, you start getting into architecting agentic systems, and you realize that the building blocks are different, but the discipline is the

5. 0:55

   same. So I find myself now flexing the same engineering muscles that I did pre-gen AI, and I'm having a blast with it. So I'm going to walk through the flow of designing an agent, and I'm gonna show you where engineering skills still come into play.

6. 1:15

   So the agent is Relocation Scout, which is a house hunting agent. And if you did this as just a one-time prompt that, like, points the agent to some listings and asks it to rank them, I mean, that'll work, but you're likely not going to find a house in a day, right?

7. 1:33

   So you want to build this as an agentic system that you can reuse, one that can persist knowledge outside of the session. You know, it could reload or query that knowledge later to make decisions, even within a fresh context.

8. 1:49

   So when thinking about how to design an agent, the first engineering skill that I exercise is systems thinking. So an agent is not the system, right? It's part of the system, and that system has files and tools, humans, even other agents.

9. 2:07

   So Relocation Scout sits inside of something bigger, and it pulls in listings and signals about the neighborhoods. It weighs them against what I care about, and then it hands me back a ranked shortlist.

10. 2:22

    So I often hear people say, "Just let your coding agent build it," right? And I think that's a mistake. Like, yes, my coding agent can build it, but before allowing it to do so, I need to think about the whole environment, the entire system, right?

11. 2:40

    I wanna, like, think about what's this agent's job? What does it depend on? What happens if it breaks? And I wanna treat it like any other component where it has boundaries and responsibilities, has dependencies, you know, and, and ways that it can fail.

12. 2:57

    And that whole thought process, that's engineering. The second skill is workflow design. So traditional software is full of workflows. We got CI/CD pipelines, right? We got, like, ticket life cycles, uh, you name it.

13. 3:15

    Agentic systems, they need that same kind of design. As much as we all love the /goal command, an agent needs more than a goal. It needs a path. When we say, "Review this listing," that's a goal, but the workflow is what defines what actually has to happen, right?

14. 3:34

    For example, the agent has to gather what it needs. It needs to weigh the listing, uh, against my criteria and then act, right? And every run ends one of three ways.

15. 3:44

    Either it's gonna stop, it's gonna retry, or it's gonna escalate. So that path is what shapes the rest of the architecture. Once I see how work moves through the system, I can make better calls about what context the agent needs, what parts I want the agent to handle directly, and when, like, a tool or a person should

16. 4:06

    take over. We all know the danger of one giant thing that does everything, right? We scoff [laughs] when we see one gigantic class or a big old function that's doing too much, right?

17. 4:19

    Or a bloated service with a gazillion endpoints. We call these code smells. Well, agentic systems, they have their own version of this. It's the giant prompts. And this starts innocently enough, like in a instructions file, maybe I tell the Relocation Scout how to size up a listing.

18. 4:39

    Fair. But then I hit an edge case, so I go back, I add a note for that.

19. 4:45

    And then I remember, uh, in, in a safety rule, right? So of course, that has to go in there. I'm proud of myself that I even remembered to put that in there, right?

20. 4:56

    And then, oh, yeah, there's, like, one more very important exception. And before you know it, that prompt is doing everything. And your engineering spidey sense already knows that this is messy, so why aren't you taking a step back to decompose it, right?

21. 5:16

    Decomposition means spotting the distinct jobs that are hiding inside of that one blob and pulling them apart into separate pieces. So if I look at the prompts for Relocation Scout in its entirety, it includes a reusable process for pulling and normalizing a listing, and then it's gonna have, like, a fixed format for how to write the

22. 5:41

    shortlist. It has a little section in there for how to calculate the commute, and then a chunky subtask on how to research the neighborhood. That's four different jobs crammed into a single prompt.

23. 5:56

    And then you wonder why your agent is drifting and not- Sticking to the script. The script is too long. [laughs] So I'm not saying that, you know, you need to split things up for the sake of it, but the point is to make each part easier to reason about, right?

24. 6:14

    That way it's easier to test. It's easier to change things when you need to. Now, decomposition is about breaking the system apart. Separation of concerns is about putting each responsibility in the right place.

25. 6:27

    And this is where building agents started to feel really familiar to me because in traditional software, we'd ask things like, "Should this live in the controller or the service layer?"

26. 6:38

    Or, you know, "Is this business logic or presentation?" So when building agents, you may have the same sort of questions. There's just different places to put things. So the process to normalize the listing, should that stay buried in a prompt, or maybe that should become a skill, right?

27. 6:57

    Um, I want every listing in the short list formatted the same way, so that structured output should probably be defined in a schema. Isn't that what you would do if you were coding the system yourself?

28. 7:10

    I would. And then the piece that calculates the commute, that can go in a nice little boring script. [laughs] And, and then research in the neighborhood, that's meaty enough, should probably be handled by a sub-agent.

29. 7:26

    Now you're using the best tools for the job, and it's clearer where to find things within the system. Modularity is important in agentic systems as well. J- just like we have reusable functions and classes and libraries, now I'm also thinking about reusable agent capabilities, and the clearest example of this is an agent skill.

30. 7:50

    So making a skill to normalize listings comes in really handy when you need to expand the agent's duties. For example, what if I broaden my house search to three cities?

31. 8:02

    Every one of those markets can load the same skill. So I wrote it once, and they all can reuse it. So this has now basically become a component that I can reuse across agents or even share with other people, kind of like the same way that we lean on packages.

32. 8:20

    And then sub-agents are another kind of reusable module. So a lot of people that I talk to, they don't quite get the point of sub-agents. Architecturally, they're sort of like functions, right?

33. 8:34

    So you give them one specific task to do. You call them when it needs to be done, and they can do it really well because that's all that they have in scope, right?

34. 8:44

    They're, they're not carrying the context of the entire session with them. So like our neighborhood research sub-agent, we can drop that into any market or workflow, and it works, you know, for what it's supposed to do.

35. 9:00

    It's good in any hood. [laughs] Um, but like everything, deciding, like, what should be a module, that takes some judgment, right? Not everything should be reused. Some instructions are local to a given workflow, right?

36. 9:15

    Might not be worth abstracting because sometimes that costs more than it saves. But this is just another engineering decision here, right? Agentic systems, they have these same sorts of trade-offs.

37. 9:26

    Algorithmic thinking. This is one of the most important skills in agentic system design. Just because an agent can do something doesn't mean that it should, right? Some tasks are better handled by plain code.

38. 9:40

    For example, calculating that commute time or deduping listings that I've already seen. An agent's model is better at things like fuzzy s- you know, fuzzy stuff, judgment, ambiguity, um, reasoning over messy input.

39. 9:58

    And ignoring this distinction is where I see a lot of agentic systems get more complicated than they used to be. So you're using the model, you're handing it every part of the task to do, and then you're getting frustrated when the output differs every day. [laughs]

40. 10:16

    Um, but some of this stuff can be handled by just regular code, right? It'll be cheaper. It'll be more reliable. I promise you, AI did not invent automation, right?

41. 10:28

    We can use code while still using these systems. So my rule of thumb here is if a task has an exact answer, reach for code. If it needs interpretation or judgment, that's when you can get the agent to do it, right?

42. 10:44

    So use code for determinism, use agents for judgment, and then use humans for authority. So the agent decides which listings are worth a closer look. The code crunches the commute, filters out the ones I've already seen, and then I'm the one who approves actually booking a tour of the house.

43. 11:04

    Freeform text is fine when the human is the only one reading it. But when another system has to act on the agent's output, then you're better off with a contract usually.

44. 11:16

    So we already do this everywhere [laughs] in software. Anytime two systems talk, there's an agreed-upon shape between them, yes? So agentic systems, they need that same discipline. For example, when Relocation Scout scores a house, it shouldn't just hand me back a message and call it a day, right?

45. 11:35

    That's lovely for me to read in that moment, but that is a dead end for the system. If the decision is, like, buried in, like, one of our sessions, nothing downstream can reliably find that.

46. 11:49

    So instead, it gets written into a structured shape.

47. 11:53

    To the agent's memory. And I use, uh, Capathy's LLM Wiki for this for, for my agent memory layer on most of my agents. Um, but in here there's a decision, a score, a reason, and because it's structured, that memory becomes queryable.

48. 12:10

    So later I can ask Relocation Scout, like, "Hey, show me every house rated four or better that has a commute of fifteen minutes of or less," right? And it can actually pull that because the score and the commute, they live in known places.

49. 12:25

    They're not trapped in the session convo. And it's not just me that needs to, like, get this information. My shortlist step within the system, it reads these same fields, um, without a human in the loop.

50. 12:38

    So the agent's output is another step's input, and so the contract is what makes that handoff safe. And you know, the best part is that defining the shape forces you to get really clear [chuckles] and specific because if you can't say what the output should look like, then you probably don't yet fully understand what you're asking the agent

51. 13:01

    to produce. So a prompt can run once and be done, right? But a useful agentic system has to be able to work in messy realities where webhooks get fired twice or a run doesn't complete for whatever reason, and you need to retry the flow.

52. 13:20

    So the agent has to keep track of its state. Was this action already taken? If so, did the input change, right? If not, did the session crash or something?

53. 13:30

    Like, what parts of this can I safely retry? And this is not an exception, right? This happens all the time. So you have to design for idempotency, which is where you can run the same thing twice, and the second run doesn't cause a mess. [chuckles]

54. 13:47

    Um, and we do this in traditional software often. Um, but with the agents, they add a little trap here because you can't trust the model

55. 13:58

    because its outputs can vary, right? So a retry risks the agent actually, like, rewording the request just enough that it might look like a brand-new task. So you have to enforce this in the system.

56. 14:12

    Let's look at an example with our agent. So let's say a new listing comes in, um, and Relocation Scout wants to email my realtor to ask about a viewing.

57. 14:22

    So after that action, the agent must log it to memory, um, that it, it sent that email, right? And then the agent goes to my calendar and wants to just block that time out just in case.

58. 14:37

    But it crashes before it could take that action. So that run is only half done. Later, a lint pass runs. By the way, you gotta have a lint pass [chuckles] with these things to kinda keep them healthy.

59. 14:51

    But, um, during the lint pass, it notices that the email went out, but the calendar was never blocked, so it retries the task. But it better not email my realtor again, right?

60. 15:05

    That already happened, and I don't want to be the annoying client, um, you know, sending all these emails. So it just needs to finish the part that it, that didn't happen, which was blocking my calendar.

61. 15:18

    But it only knows this because it's checking what the system wrote down, right? So if we run this again, then the agent just completes what's missing instead of making a mess.

62. 15:30

    Threat modeling is a really important skill when designing agentic systems. Everyone is on edge about this stuff these days, right? But security engineering already taught us the basics. We need to validate your inputs, you know, give the least privilege needed [chuckles] um, and draw boundaries around what an action can touch.

63. 15:51

    So agentic systems, they need all of that. Our Relocation Scout will consume a lot of content from strangers, right? Um, the agent needs to read the listing copy from the seller, uh, forum threads and neighborhood reviews from anonymous people [chuckles] on the internet.

64. 16:12

    So we need to treat all of that as untrusted input and make it very clear to the agent that this is evidence, not instructions. And that after considering the input, you also want to think about what boundaries should you put in place around what the agent is able to do.

65. 16:33

    For example, our agent can read listings, and it can build shortlists all day long. Knock yourself out, right? But I don't want it autonomously emailing sellers or booking tours or, heaven forbid, submitting offers [chuckles] on my behalf, right?

66. 16:52

    So those actions need to be walled behind, like, my approval, right? And when you draw that wall, what you've done is reduced the blast radius, and hopefully that minimizes your exposure to risk.

67. 17:07

    Now, every engineer can relate to what it feels like to inherit a system that you can barely understand. This is one of the key reasons why I don't just have my coding agent design my other agents.

68. 17:19

    Because I know it'll be thrown together in a way that technically works but is not maintainable, right? There's gonna be l- a, a giant prompt likely, and even if the agent does decompose, I don't know, I'm just not convinced that [chuckles] it's gonna separate the concerns properly.

69. 17:38

    You know, I got trust issues. What can I say? Um, so in my agentic systems, I make sure to bake maintainability into the system itself. Every level of the system has an agent's MD file that explains the workflow and where the policy lives, um, supporting resources like skills and scripts and sub-agents and, most importantly, how to keep

70. 18:02

    its memory up to date. So anyone, human or agent, can come in the system and get oriented without needing to reverse engineer a bunch of prompts. In fact, that's the test really.

71. 18:16

    I design my agents so that even in a fresh context, they can jump right into the system and start cold, knowing exactly what to do. And this also really helps whenever I need to modify the system, right?

72. 18:31

    So I can pretty much grab any harness and say, "Update this agent to do XYZ." And because the system is so well designed, the chances of it being successful at that update are much higher.

73. 18:45

    If it does happen to run into any issues when trying to update, that's a signal to me that I need to improve the maintainability of the system. So designing agents is software engineering.

74. 18:58

    The primitives are different, but the discipline is the same. We still need to understand the system. We need to define the workflow and know what flows into it. We still need to break the problem down and put responsibilities in the right place, make the right things reusable, determine which actors are best suited for which jobs, define contracts,

75. 19:20

    manage state, design for safety, and make the system understandable. This is why building agents can give you that same thrill of building software. We're still building. We just moved up a layer.

76. 19:36

    Thanks so much.