Agents Don't Do Standups: Building the Post-Engineer Engineering Org

Talk transcript

1. 0:00

   [upbeat music] All right, so I'm in the slot before the snacks.

2. 0:18

   Um, but I'm here to speak about the post-engineering org and a case study we've been doing at, uh, PFF. Uh, started in, uh, January, and we finished it in March.

3. 0:29

   Uh, PFF is a sports data company. We help NFL and NCAA teams figure out what they should be, uh, doing. Um, and we also have a consis- We also have a, a consumer arm which does fantasy football, sport betting, and for those of you who know American football, there's a draft happening end of the month, and so

4. 0:48

   allows people to basically play as one of these, uh, teams. We're a fully distributed engineering team, so we got engineers in India, we got engineers in Spain, and all the states in America.

5. 1:01

   Um, but first, let's get into some kinda stats. So we have a hundred million, uh, page views annually. We have nine million, uh, drafts happen on an annual basis.

6. 1:10

   It's a fairly, uh, popular tool. Um, and the issue that we had was we were, uh, two hundred employees around, uh, uh, twenty engineers, and we were falling behind, uh, competitors.

7. 1:24

   We were focusing on sport betting stuff, but a lot of our consumers were actually interested in this. And then with, uh, Claude Opus coming out, I started experimenting around November on a personal level,

8. 1:36

   and then it got spun out to, uh, two engineers. Um, one of them was our strongest frontend engineer, and the other one was one of our strongest full-stack engineer.

9. 1:47

   Uh, so really the question I was asking was, instead of figuring out how we can help engineers go and output more, how do we help make the-- how do you help make-- How do you help make, uh, the agents quicker, right?

10. 2:00

    So I think if we all think historically, software engineering, you've got the Agile Manifesto, software craftsmanship. Um, you've got a lot of good, uh, perks and benefits for engineers from foosball tables to sleeping pods to pretty much every perk that most other industries really don't have, and it's 'cause we're the bottleneck.

11. 2:19

    And if, like, companies are able to go and optimize for that, it gives them a lot of benefit. But now things are changing. So I just wanna go straight into how the case study ended up.

12. 2:30

    So we had, uh, uh, twenty-five times more the-- uh, twenty-five times more deploys. So the two engineers were deploying five times, five times every-- five-- were deploying five times every day.

13. 2:44

    Uh, and the other team was a team of around, uh, ten engineers. They're doing it pretty much one deploy every five days. There is a big obvious caveat here, right?

14. 2:52

    Small engineering teams are always gonna be quicker than the big ones. So there is an element of that multiplier that is just from it being a small engineering team.

15. 3:02

    However, that smaller, uh, tiger team still had to coordinate all of those daily dep-- still had to go and coordinate all of those, uh, deploys with the bigger team.

16. 3:12

    So there was still that-- there was still that kind of issue that was happening.

17. 3:16

    And this one, this is always hard. How do you validate, like, the output is actually helpful and the number of PRs isn't helpful, the amount of code isn't helpful?

18. 3:24

    So we basically blended the number of, uh, tickets with the code complexity, and we found that their-- we found that their, um, that their output was at ten X.

19. 3:35

    Got another slide which might look a little bit hectic, but just bear with me. Um,

20. 3:40

    so these were the features that we went and, uh, constructed. With those two engineers, it took them under, uh, two month. If we're gonna do this before, we were estimating it's gonna take four months.

21. 3:52

    Uh, the big thing you can probably have a look in the, in the top half is one of the engineers gets unblocked, uh, in under a month and can then start building other stuff.

22. 4:01

    Whereas in the old way, they're both blocked for three months. And so you get this thing where you now have a, a compounding increase, where it's not just faster for that stuff, but you're now able to do a lot more stuff than you were able to do before.

23. 4:17

    And the one thing that I really wanna go and basic-basic go and highlight is it doesn't matter if the output's more, it doesn't matter if the number of the deployments are higher.

24. 4:27

    What really matters is, really matters is basically if our customers, if the customers are happy. And so we did statistically signi- we did statistically significant, uh, surveys. Uh, and the average quality score was eight point six out of ten.

25. 4:42

    What was interesting was before AI, we would probably average, uh, seven, seven and half. So we weren't really delivering what the c- what the customers, uh, had been interested in.

26. 4:53

    So Scrum did not survive. We were l- we were basically having a look not just on an engineering front and all of the delivery gains from that, but also from a process standpoint.

27. 5:08

    And so no need for a project manager. Uh, we don't need to play multiple, uh, games of the telephone, and everything we were doing was optimizing to be as, uh, quick as possible.

28. 5:20

    Engineers aren't the bottleneck, so we don't need to have all the old, uh, ceremonies that we had before. So what did we have? It looks pretty basic. We basically had a-- We, we basically had huddles.

29. 5:33

    So those huddles were basically every other day. They'd be like half an hour, maybe an hour. You'd have the engineers, someone from product, someone from the des-science team, and you would speak about the things you've been building the last, uh, couple days.

30. 5:47

    You get instant feedback. We were trying to deploy to production as fast as possible in an MVP state and getting as much feedback as, uh, possible also.

31. 5:58

    And that development flow is we have a spec. I think most of us are probably fairly familiar around this, but we get the agent to go and interview us.

32. 6:07

    We get feedback on that spec, and then we make a lightweight design document that's done by, that's done by a-the agent. So we have like a skill.

33. 6:18

    And the, uh, and the nice thing about that is it analyzes how we've done all the LDDs before. And so anything that we're building is in the same kind of ethos as everything else that's been built.

34. 6:30

    Uh, so it's not a, a Claude code spec-- it's not like a Claude code specific, um, kind of thing. And then those LDDs, they get distributed, and we get feedback from all of the, the engineers.

35. 6:41

    And then at that stage, uh, we automatically create all, we automatically create all the, the tickets and then the PRs after that.

36. 6:51

    So sprint planning, we don't have sprint planning anymore 'cause we don't need to have an hour going and basically estimating, uh, tickets 'cause those estimations don't really make any difference.

37. 7:02

    Uh, something we're not doing at the moment, but we will probably have to do once the s-s-once the subsidiz-once the subsidization of, uh, tokens ends, is we'll probably have to go and estimate the token expenditure to go see if we wanna actually spend this amount of money on it.

38. 7:18

    Uh, the daily stand-ups we don't have to do 'cause all these tickets get auto-updated. That is, uh, specific to the status of the PR. So, so we've got the PR is open, it goes auto in progress.

39. 7:30

    If it goes into review, it updates it. If it gets merged, it gets closed. These things were obviously manageable to do before, but it's all been made a little bit easier.

40. 7:38

    Uh, sprint refinement we don't need to do 'cause that happens in the spec and the LDD, um, flow. And then when we automatically create all of the, the tickets, we structure it in a manner so the tickets are made, so none of them are bl-blocking each other.

41. 7:50

    And if there are ones that are blocking each other, that information is, um, flagged up and highlighted. And retrospective, th-this one might be a little, like, controversial, but we rely on the customer satisfaction, the customer survey.

42. 8:03

    That's the main thing really. And then you got the normal development me-metrics like the deployment frequency. We ask all of our engineers to really flag when there is an issue, to flag it immediately until instead of hanging on till the end of the sprint to go and go and f-flag that issue.

43. 8:19

    Where sometimes I think all of us have probably felt that our feedback hasn't really been heard 'cause it's been a sprint before.

44. 8:28

    Oops, sorry. So how do we start? You gotta pick the engineers which have the best, uh, system and the knowledge, and I think every engineering, uh, team probably has one or multiple engineers where if anyone ever gets kinda stuck or hung up on anything, they're like, "You should speak to them.

45. 8:44

    They'll be able to sort it out." The other one is you should s-- you should go slowly. So, um, there is an appetite to give everyone all the coding assistance and open it all up.

46. 8:55

    But I think, um, with this, you want a slow-phased approach,

47. 9:00

    and then you should experiment in non-critical systems. So that's what I did in, in the November and the s-- in like the two months before the case study. I was making small proof of concept features going, going and pushing it to production.

48. 9:16

    Didn't really get much traction, so if there was a bug or a mistake, it didn't really matter. And then we moved it to the thing which gets a hundred million page views, um, after that.

49. 9:25

    Um, the bi-big thing here though is not everyone can drive a sports car, and that's all right. There's n-- that's not a big issue. But I think w-- I think everyone needs to be a little bit honest around the engineering org and engineering team.

50. 9:38

    And this new era is gonna be hard for a few engineers. And I think the, the type of engineer where they will really strive is one who is, like, curious.

51. 9:49

    It's the engineer who, if they haven't figured it out, they're gonna spend a b-bit of time just figuring out how something's been built. They'll be able to, they'll be able to, they'll be able to smash this easily.

52. 10:00

    The old style of engineer, which needs something really, uh, prescriptive as a spec, I think they're gonna struggle.

53. 10:09

    So in the engineering side of things, you need verifi-verifiable, verifiable, deterministic tasks. I think a bunch of people have spoken about this today. This ranges from everything from all the different types of tests, but there's also things which are specific for your product or your org.

54. 10:29

    So for us, it's feature flag 'cause we do a Trunk-based development. Um, or it could be generating the interactive elements, um,

55. 10:38

    generating the analytics for the interactive elements. Um, next thing is the agentic code reviews. So with us, we didn't really l-like relying on agents to do code reviews for system design and what engineers would usually do code reviews for.

56. 10:56

    We use agents to do the code reviews that engineers hate getting any feedback from. So that's like, uh, the variable names, this doesn't fit the style or like those kind of opinionated manners is much easier to offload.

57. 11:11

    Uh, and then you can remove that whole kind of emotional aspect out of it, and then you just allow the engineers to focus on the big picture.

58. 11:19

    Um, we still need people involved, but it's really heavy on the spec, really heavy on the LDD, the Lightweight design document, where we figure out how we're gonna b-build this.

59. 11:29

    And the thing I haven't touched on at the moment is the product. We're in an era where everyone's able to create anything in an hour. But a lot of these tools have the brand feel and the product feel of something that's been created by a Claude code.

60. 11:47

    And so if you really wanna get the best out of it, you really need to make sure that the engineering team is spending time making sure it still feels like, um, basically every other product from the company.

61. 12:00

    So how would I recommend tackling this? You should go see your engineering development life cycle like a, uh, like a kind of a, like a kind of a f-f-f-factory.

62. 12:12

    So you should think you're in a factory. How do you break that up into each small composable element? So in a car factory, there's one thing which is building a, building a, building a- A door, another one fitting a steering wheel.

63. 12:24

    The exact same thing happens for engineering, right? So you have the branch name, you've got, uh, creating feature flags with Trunk-based development. You got the-- If you build APIs with a specific software design, a pattern, you should abstract that into a composable skill.

64. 12:41

    The one thing I would flag is I'm not a big fan of, um, consuming other people's skills if they have strong software kind of opinions that are in contrast to, to the engineering org, `cause you're just gonna kind of end up in issues.

65. 12:57

    So you just need to make sure that any skills that are getting composed are still kind of matching.

66. 13:04

    Uh, and yeah, this is just at the moment, pretty much everything is fully autonomous from the spec LDD, a ticket from the PR. Uh, we have a QA process also.

67. 13:14

    So what happens is, um, whenever we merge a PR, it automatically deploys onto staging. When the deployment on staging has happened, we will spin up a, a QA agent, which has a look at all of the tickets that have happened, has a look at the acceptance criteria, and it will go and then a QA against that.

68. 13:34

    And then if everything's passed, I mean, great. And if-- and then if everything hasn't, it will flag what those items are. And the bit we haven't gone and done at the moment, but I'm, um, um, but I'm aiming in the next, um, a couple of months, is to then have an agent have a look at those tickets,

69. 13:52

    find out where the acceptance criteria hasn't happened, and then automatically create the PRs. And so then you get into this flow where agents can basically, uh, self-heal. And the cool thing about that is it lets us do multiple things in parallel because we're now in a position where we basically trust agents as well.

70. 14:09

    Uh, where do we still use people? I mean, I've experienced this, but they do like to use, uh, sh-- they, they, they do like to use a sh-- do like to use, uh, shortcuts.

71. 14:20

    Um, so we got to make sure the sec-- the, the security side of things is behaving. At product feel, I spoke about that a little bit earlier. And then the scale and engineering complexity for the task.

72. 14:33

    The LDD is really meant to help on this. So I think we've all experienced where it'll make a thousand lines of code or over-engineer something. If you're really prescriptive on that engineering document at the start, you can really prevent that from happening.

73. 14:49

    So what things should you go and aim for? You should start with boring, repetitive tasks. Ideally, things engineers hate because you're going to get the most amount of buy-in from that.

74. 14:59

    Remove as much redundant process as is possible. And I know it can be strange when you've been doing something for, like, uh, two decades plus to just chuck it all out.

75. 15:09

    But I would ask you, what is the purpose of this meeting? What's the purpose of this process? Is it just `cause everyone else has been going in and doing it before, or is it `cause it actually helps out?

76. 15:20

    Uh, make sure your team's personal kind of engineering culture and the patterns are encoded in skills. So if there are software design patterns that you really do. So for us, whenever we build an API, we focus on the Service repository pattern.

77. 15:34

    Uh, make sure your guardrails are fully functional before you get into, before you get into an autonomous flow. And you should start out with the best engineers.

78. 15:44

    Things you should not do is to try and onboard everyone at the exact same time. Uh, I think a big reason why a lot of these things haven't really panned out is `cause we've given everyone a Claude Code Codex and given them a hackathon of like, sweet, we've done everything.

79. 15:59

    They should now be able to sort it all out. It's just not that easy. Every engineering org is completely different, uh, and every engineering org's style is really different.

80. 16:09

    And you just need to go and have a slow-phased approach to make sure, uh, that that transition is happening smoothly. And that's why I think those small companies are at a huge advantage over the big enterprise companies `cause it's really easy for me to scale this out when I've got twenty engineers.

81. 16:27

    It's really hard for me to scale this out if I was in charge of a hundred or if I was in charge of a thousand engineers or if I was in charge of ten thousand engineers.

82. 16:34

    So, um, but you don't want to be too shy. You don't want to be too cons-- you don't want to be too conservative

83. 16:40

    because the fact is that there's a lot of other companies that are going full speed. And, um,

84. 16:49

    for us, even from our perspective, I felt a few months behind, and I was kind of feeling it. And I think if you think about the compounding impact I was talking about at the start, that's just gonna carry on happening.

85. 17:00

    So, like a few months behind at the moment might be six months behind in a few months, might be twelve months behind, uh, a little bit afterwards. And then again, just, just the, just take it slow.

86. 17:11

    Uh, it needs a phased, it needs a phased kind of approach and rely on your engineers to let you know whether you can scale things faster or if it needs a bit more, more time.

87. 17:20

    I know I spoke to you about a lot of features that we built, but I didn't show you any of the features. Uh, if anyone does want to have a play around, feel free to scan the QR code, and then you can have a look at all this stuff.

88. 17:31

    But cool. Thank you, everyone. [audience clapping] [outro jingle]