Vibes won't cut it

Talk transcript

1. 0:00

   [upbeat music] Thanks for coming, by the way, and, uh, for sticking around for a little while.

2. 0:19

   Um, if youuu aren't prepared, I hate to break it to you, but this time next year, half of us won't even be here anymore. That's basically if you listen to whatever the hype is about AI and AI coding.

3. 0:35

   You know, there's lots of fanfare. No disrespect to very intelligent people, um, that made these quotes, um, but I think they're probably wrong. Not because I don't think AI coding is going somewhere important, but probably because they haven't actually touched a production system in a very, very long time.

4. 0:52

   And so maybe generating code at thirty percent isn't really what they think it is. Because really, AI code is still code. One thing they don't really recognize in this space is that they're working in very, very, very large code bases that have basically every decision that's ever needed to be made about that code, about that architecture, about

5. 1:15

   that infrastructure, has already been made for them. So if I'm generating thirty percent of my code against, you know, let's say they're doing thousands of lines a day against millions of lines of new code or of, of existing code, there's not a lot of wiggle room for what that code can do or should do, right?

6. 1:32

   Like, it's just another button. If you've ever... No offense. Anyone from Meta here before I start ripping on Meta? [laughing]

7. 1:38

   No. No Meta. Great. I'll rip on Meta for a minute. If you've ever talked to a Meta engineer, they will talk about how they built a button in the ads platform for six months.

8. 1:46

   Like, that's, that's what they worked on for six months. That was their job. So, like, there's very little definition about, like, what this can-- needs to do that has wiggle room that AI really stru-- can, can influence.

9. 1:58

   Next, AI is still writing code. This is code we have written for fifty years. Same programming languages, nothing is different there. That code still needs to run in production somewhere.

10. 2:08

    And if you're not familiar with-- If you haven't run a large production system, even if you write a great line of code, in complex systems, things fail. Like, complex systems have emergent behavior that don't show up in just single lines of code.

11. 2:20

    And so we still have to run this stuff. And so who's gonna fix it? Who's gonna examine it? Who's gonna understand those nuances when, uh, when-- if you don't have software engineers?

12. 2:30

    So I think we're still gonna have software engineers. And then also, history repeats itself. This is not the first time I've been told that my career is over. Um, I'm getting in on the, on the years at this point in time.

13. 2:42

    But anyone, uh, around, like, the DevOps transformation from fifteen years ago, cloud, all the, all the sysadmins that I know that were racking boxes and, and booting kernels all got pay raises and all work on much more valuable things now.

14. 2:56

    They're much happier doing the work they're doing. So this isn't new. This is just a different level of abstraction, right? Like, tractors didn't get rid of farms. They just got rid of farmhands and horses.

15. 3:06

    Yeah. Like, yes, there will be change in an industry for certain, but tractors didn't get rid of farming. We still have to farm.

16. 3:14

    So vibes. Anyone here a vibe coder? It's okay if you are. No shame. Anyone-- And who here thinks call-- would call themselves a softwa-- a professional software engineer?

17. 3:25

    Great. So vibe coding, if you're not familiar with the term, I think everyone in this room is, but I'll do a quick summary, is basically letting the AI write all of the code and think through the code and not really examining the code at all.

18. 3:35

    Just, does it do what I need it to do? And then in that case, like, I just keep going and let it keep going, let it keep going. Don't, don't examine the code and edit it that way.

19. 3:45

    But that's not what we're talking about. We're ta-- I'm talking about how can we write code that's for production. And when I say production, I mean you. You have four nines.

20. 3:54

    Do you even know what four nines means? That's production code. You have thousands of users, gigabytes of data. We're talking software that runs the internet today. Vibes don't cut that because there's a lot of, like, nuances on what goes into code.

21. 4:09

    So first, let's clear up one misconception. Code is not the job, in the same way that blueprints are not the job of an architect. That's an artifact of being an architect.

22. 4:20

    The artifact of being a, a software developer is code. Sure, yeah, I have to output some code, but I make thousands of decisions about what my software is supposed to do, what kind of-- what I'm writing, what packages I'm bringing in.

23. 4:32

    So let's stop conflating generating code with the art and the f- the craft of doing software engineering. Those are different things. And so yes, LLMs are great at producing code, but are they-- is that the same as writing production software?

24. 4:47

    So a very smart person that, uh, started Stack Overflow, Jeff Atwood, rest in peace, Stack Overflow, um, said, "The best code is no code at all." And I think that's true because every line of code comes with, comes with a burden, right?

25. 5:00

    I have to maintain that code. I have to debug that code. So every line of code I generate, I have to be responsible for. And so we've been s- we've spent so much time thinking about, like, how much code can AI generate.

26. 5:11

    Who cares how much it can generate? The more it generates, the worse off I end up being, worse off the system ends up being. We want to put as little code as possible in there.

27. 5:20

    Because all code has trade-offs, right? Like, we recognize this package has this performance characteristic, this, this methodology has this. The best example I can give you, which I'll, I'll try to get through so you can all get back to snacks, is the difference between like a monolith, uh, and microservices architecture and an event-driven system.

28. 5:39

    How many decisions go into those kinds of architectures and the choices you have to make to do the same thing? Like, if you've ever-- If you built a flight booking system in all three of these, you have thousands of individual decisions that have to get made.

29. 5:53

    LLMs don't make decisions. They generate text. They generate patterns. And at some scale, at some point, there's no pattern for my software. Anyone here run a piece of production software that's kind of a bit of a snowflake?

30. 6:06

    Has a bunch of like idiosyncrasies in it that like, "Well, only Bob knows how that works, and only Jane can fix that. She wrote it six years ago and hasn't been on the project in..."

31. 6:16

    That is software. So at some scale, pattern matching doesn't work anymore because all of the nuances that go into all that software can't be pattern matched again. So when, when the software's going down, this is my like trauma of like carrying a pager.

32. 6:29

    When, when the software goes down at 2:00 in the morning, vibes aren't gonna fix the bug. Like, someone has to diagnose that problem.

33. 6:37

    So what is the work of software engineering? For me, that's changing software safely. That's been my job for 20 years, is how can I make changes to software, whether that's adding new functionality or, or changing existing code, and how do I do it safely so that software doesn't go down, so my users get their...

34. 6:53

    get the thing that they're getting, the, the widget ships, that, um, you know, data is secure. So how do we d- how have we done that so far in the industry?

35. 7:01

    We've done lots of different things to solve that problem. One, there's just my own knowledge. I have to like learn a shit ton about a code base before I can make changes safely.

36. 7:09

    Uh, we use version control to do that. We know testing. You know, like that's why we write tests, to catch if like, if I change this, did this thing break over here?

37. 7:17

    Um, we use type systems. We use deployment strategies. Can AI start to help us? Can context that an AI has about and can understand more of the code base help us?

38. 7:28

    Probably. You know, at Augment, we believe that context is like the most important part of all AI generation in code. So yeah, we think we can solve that problem.

39. 7:36

    That doesn't change that I still have to care about production. So let's just assume that we're writing code. Like, okay, the world is... the future is here. I have to use AI.

40. 7:49

    Um, the thing I find the most interesting about this space, I've been in it a long ti- in this, in dev tools for a long time, is professional software engineers are the last people I see adopting AI, and I've never seen that before.

41. 8:02

    Like, I've seen version control systems change. I was at GitHub in the early days, and so like massive jumps to Git, uh, the cloud transformation. Uh, pick your, pick your like innovation and developers are like, "Hell yes, give me that new thing."

42. 8:15

    I don't know why. Well, I have some hypotheses, but I don't unders- fully understand why software developers are like, "I'm not touching code. I'm not touching AI." It's like, it can't do what it's gonna do.

43. 8:26

    So I wanna talk about that for just like, just a couple minutes. So a few years ago, AI coding was mostly just a pile of bricks. Like, it kind of worked, but really didn't do much.

44. 8:37

    Um, about a year ago, you know, years and change when Sonic, uh, 3.5 came out, that's really when we saw a massive explosion, um, in, in AI coding because the quality substantially jumped.

45. 8:50

    Um, and then, uh, four weeks ago, if you weren't watching the news, literally every AI coding tool said like, "Agents are the future." Uh, we did. I'm very proud of that.

46. 9:00

    But like, now agents are everywhere, right? And so this transformation's happened, uh, very, very quickly. Uh, and so I wanna talk about how we can talk, do software engineering with this new future that we're seeing.

47. 9:14

    Um, so how do you build software that's easy for AI to write? A couple tips for you. Um, have some documented standards and practices, right? Like what do you use?

48. 9:25

    Every code base I know is in some sort of flux, right? So like, are we using this package or this package? Okay. Well, document that. Let the AI know that this is the next direction of your code base.

49. 9:35

    Um, have reproducible environments, right? Like, can you easily spin up a developer environment? Is your developer environment very bespoke and unique? Um, you wanna have it reproducible. We wanna have easy testing, right?

50. 9:48

    Like, can you run your test locally? Can... Is that fast? Uh, you know, that's pretty s- pretty standard stuff. Um, establish clear boundaries of what you're gonna do. You're never gonna give AI the idea of like, uh, extract this module using the Strangler pattern.

51. 10:03

    What does that even mean to an AI? Like, that's a whole... You have to give clear boundaries of what you're trying to build and how to get the AI to do it.

52. 10:10

    And lastly, like have clearly defined tasks and work, um, because AI is gonna be as-- You know, I wouldn't give any engineer on my team, whether they're senior staff, junior, just a vague task of like, "Could you make this button do something different?"

53. 10:24

    Like, I see too many engineers prompting in that way. And what's interesting about this to me is like, this just sounds like software engineering, right? Like if we don't...

54. 10:35

    If ideally, your software engineering stack looks, has these qualities, and if it doesn't, you're like, "Our productivity sucks because we have, you know, bespoke testing infrastructure. We have bespoke developer environments."

55. 10:46

    You have to give AI the same tools that engineers need because it's doing exactly the same job. It's writing code. I've never one-shotted a piece of code in my life personally.

56. 10:56

    Like, I've always made mistakes in the code I write. I run a test, it fails. I've got a linter, it fixes it. Whatever thing is. But we've had this expectation that AI can like write perfect code, and I don't know why.

57. 11:08

    And so when you're thinking about adopting AI as a software engineer, you have to make sure that your, your systems work like you would expect any other engineer to work, because that's how it's writing code.

58. 11:21

    The next thing is code review is by far the most important skill. Um, I think we've probably forgotten that skill as an industry. Um, we probably should have been interviewing for code review and not like, "Here's this esoteric leet code problem that you can solve," but like, "Can you read somebody else's code and comment on why it's

59. 11:37

    good or bad?" Um, I think this is gonna become far more important as agents are writing more and more code, and our code review tools today frankly suck. Like, I'm getting a list of changed files.

60. 11:48

    It's lexicographically sorted, so like great, file A changed. Let me read what changed in file A. What changed in file B? That's not a way to think about how software changes is in the order of the files.

61. 11:58

    And so I think we're gonna see a pretty big explosion on the way code review can happen, um, and that's the skill we need to be interviewing for, you need to be, um, again, brushing up on.

62. 12:09

    All right. I'm running quick on time, so I do wanna give you a couple highlights of, like, if you are so- one of those software engineers that's like, "I don't, you know, I'm not-- I don't trust AI, I need some tips," I wanna give-- leave you a few of these.

63. 12:21

    Um, feel free to take a screenshot. One, the most important thing I can imagine is AI talks like a human, but is actually a machine. I had this interaction with AI the other day where I was like, uh, you know, I was yelling at it because that's what you do at AI when it doesn't do what you

64. 12:36

    want it to do. And it's like, "Oh, I'm sorry, I just scanned that file, I didn't read it." I'm like, "The fuck does that even mean?" [laughing]

65. 12:43

    What-- How does a so- piece of software scan a file? It just, like, skimmed it? Like, that's not a thing in software. It's because LLMs are trained on all the data in the world, right?

66. 12:53

    There are thousands of emails that it has read that said like, "Oh sorry, I didn't read your document thoroughly. I actually just skimmed it." It's like, "Oh, that's what I should say when someone's yelling at me about not reading a file."

67. 13:02

    Right? And so we have, we have to distrust some of the things that LLMs are saying they're doing because it's not actually doing that. Don't forget, it's just generating text.

68. 13:11

    It's not always doing exactly what you're s- what it's outputting in that text. So keep that in mind when you're writ- when you're reading through LLM output. Um, let's see, what other tip, quick tips can I give you?

69. 13:25

    Sometimes code is just different. It's okay if the LLM outputs code differently than you would. I can't expect it to produce the code that I would exactly. In the same way that, like, the person that sits next to me also writes code a little differently than I do.

70. 13:38

    And so just accept that that's okay. And if you wanna force it to write code like you, you can spend that energy. But know the difference of is the code better or is it just different?

71. 13:48

    And so let go of some of that, like, this is how... You know, we-- this is why we have linters. This is why we have rule systems. This is why we have style guides, so we can stop arguing about, like, is that how you define a function or is this how you define a function?

72. 14:02

    Let some of that go if you can.

73. 14:04

    Um, write a rules file. You know, give it-- tell it what you want it to do. Have a file. I always start all of my projects with, like, here's the stack I'm using.

74. 14:14

    Here's the, like, guidelines I want you to use. And that's always ends up being part of the context I send with the LLM. Um, and then lastly, I like the define, create, refine loop.

75. 14:24

    Expect... Create a, a document of some kind. Have the LLM help you generate it. Like, I'm doing this thing. Write a markdown file that would, like, lay out the plan.

76. 14:33

    Here's my plan. Save that as a markdown file. Then use that again in your context. Have it create the thing. Have the agent run against that file. You know, you can make your edits to it and then refine it.

77. 14:44

    Then you're going in with, you know, code completions or whatever to, like, just tweak the things that you wanna tweak. And just do that loop over and over and over again.

78. 14:51

    Make a plan, have it create it, and then just make your, make your tweaks. And you'll get a lot more comfortable, h- one, how to prompt the LLM to get the code you want to do, as well as, um, that's just a very efficient way of coding.

79. 15:03

    And if you've letting go-- if you let go of that code has to be how I would write it, not just good functional code, then you can get a lot more productive in that way.

80. 15:13

    Um, thanks again. Uh, I blasted through that. I'm happy to talk. Augment is in the expo hall. I'll be around there to chat or over here. Um, I hope to see you here next year.

81. 15:22

    Hope we're all here next year. Uh, I think the jobs aren't going anywhere. Um, but happy coding. Really appreciate it. [outro music]