Hiring & Building an AI Engineering Team

Talk transcript

1. 0:00

   [upbeat music] Hex is a data science platform company.

2. 0:16

   Um, what you should think about is sort of like a really powerful Jupyter Notebook. Um, I've been building AI capabilities at Hex for about a year and a half.

3. 0:23

   I've been in data science and machine learning for about 12 years. So let's, let's get one thing out of the way. Um, why might you care about my opinion?

4. 0:32

   Well, um, let's take a look at this tweet of mine. This is when I was hi-hiring my first AI engineer. I'll draw your attention to the date. I'll also draw your attention to the date on this very famous blog post.

5. 0:44

   Um, so, uh, my, my tweet was one of the [laughs] inspirations for the rise of the AI engineer. Um, that does not mean that I'm fully aligned with that blog post, but it does mean that I have some opinions.

6. 0:56

   Um, what I expect many of you are expecting is for me to start talking about

7. 1:05

   what AI engineering is. And so, um, to, to kind of try to help with this, I, I made a little simple chatbot. I call it AI Leader GPT, and I figured let's have it answer your questions.

8. 1:17

   Um, I'm pretty lazy. I have a lot of stuff on my plate. I figured just hand it over to AI. That's kind of our goal anyway. So, um, I asked my little chatbot, um, that I wanna ask some questions.

9. 1:27

   I asked it, uh, "What does AI engineer mean?" Um, unfortunately, um, this is one of those moments where clearly the intelligence really isn't there yet. Um, despite a lot of improvement on MMLU, we yet can't define this term.

10. 1:40

    So unfortunately, I'll have to do this myself. Um, okay, so, um, let's start with the question, what? Uh, building an AI product requires a team, um, but what does this role really look like?

11. 1:54

    I'm going to tell you what my very first job posting looked like, and I'm gonna talk you through it to try to give you some sense of what this actually should be.

12. 2:02

    Um, I was looking for a senior engineer. That senior engineer could come from SWE or MLE, and we wanted them to rapidly expand our capabilities for greenfield applications. That should sound very normal, should sound very expected.

13. 2:19

    Unfortunately, while we respect ML researchers, we are explicitly not looking for this. This was in the job posting. For those of you that are putting up job postings, I highly recommend that if this is not what you're looking for, that you tell them.

14. 2:37

    It is both a waste of your time and their time for them to apply hoping to go to ICLR based on the work they're gonna do at your company.

15. 2:49

    Once again, this is not for lack of a, a deep interest in their work. This is just not the stage we're at.

16. 2:57

    I would love to hear from you if you have experience getting ML or AI capabilities into production and serving real users,

17. 3:06

    if you have a lot of a- enthusiasm for applications of AI to business problems, and if you have a core understanding of the architectural things. Maybe you've read one of the books on MLOps.

18. 3:19

    Maybe you've read previous discussions about MLOps. Um, maybe you've worked in some infra-adjacent things as a backend engineer. That all sounds wonderful. And here is where we start to get a little controversial.

19. 3:33

    You should be comfortable working in both Python and TypeScript.

20. 3:38

    Ooh.

21. 3:38

    It's okay- [laughs] ... if you are only strong in one, but you should be open to both. Our application is built in TypeScript. I need people who are going to be willing to get into the details, to get into the nitty-gritty there.

22. 3:52

    I don't need you to come in as an expert on React, but I do need you to be able to interfos- interface with those people in a really productive way.

23. 4:04

    So I went back to my little GPT, and I said, "I'd really like to understand the when, the why, the who, and the how of all of this process."

24. 4:14

    And so let's go through those things. First up, when?

25. 4:19

    Well, it actually kinda depends. It depends where you are in your journey. If you're early in your journey, you need SWE skills, you need data profiles, and you need product competency.

26. 4:30

    If you're middle stage, you need SWE some more, probably some more infra. You need data profiles, but you also need some design. This last one is the number one thing that I see teams under-investing in.

27. 4:44

    If you're later stage, you definitely need infra. You need all of the above, but a little bit scaled, and now you need to start actually thinking about machine learning engineers.

28. 4:53

    This is when you wanna start... Yeah?

29. 4:55

    Could you define data profiles?

30. 4:57

    Yeah, totally. So data scientists, data analysts, people that have a lot of experience looking at distributions of data and saying, "Wait a minute, that's strange." Looking at user output and saying, "Hmm, this is actually quite different than we were expecting."

31. 5:13

    Looking at sort of like product analytics and saying, "You know what?

32. 5:19

    This retention is pretty poor." One thing that I would sort of ask everybody in this room to do for yourself right now is, how good should retention be on an AI product?

33. 5:30

    If you don't know that about your product, both what the comparables are for other products, you're missing an opportunity. You're missing an opportunity to hire people who have been doing this for a long time.

34. 5:40

    They will up-level your team. So in the later stages, now you need to start talking about MLEs, maybe because you need to fine-tune a model, maybe because you need to fine-tune an embedding model, maybe because you need people that already understand search and retrieval.

35. 5:58

    You should start early. However, it is extremely dangerous to accidentally fall into the trap of the Mythical Man-Month. This is true in all softwares, all product development. It is somehow magnified in AI.

36. 6:16

    I see more people overdoing this in AI than I do in other, other domains. So if the AI demo only takes one week, and the AI product clearly takes only four weeks if you throw twenty engineers at it.

37. 6:32

    I don't have twenty engineers, but even if I did, I certainly wouldn't be putting them on the same AI product. There is a very, very, like, important reminder that nine women cannot give birth in a single month.

38. 6:49

    Because all AI products are early by definition,

39. 6:54

    the Mythical Man-Month is especially true for early products. You really, really, really need to be careful here. This is-- When I speak to my peers, this is the number one mistake that I'm hearing.

40. 7:07

    The hiring schedule should reflect the development schedule. So you should start with an early product that you get in front of users, then you should start by building evals, then you should get user feedback, and then you should iterate.

41. 7:23

    If you want more details about this, you can either consult this paper that we released of what we learned ear- last year building LLMs, or you can come to our talk later today to get more information on this.

42. 7:33

    This development schedule, this is not just my opinion. It's not just six of our opinions. This is tried and tested, speaking to a lot of other people. This is what works, this order of operations.

43. 7:45

    So if this order of operations works, then your hiring had better be well-aligned to it.

44. 7:52

    Data needs to come much earlier than in traditional product engineering efforts. You asked the question about, like, what do data profiles look like? One of the things that is very, very true about AI, partly because of this development schedule, but also because of the type of products that we're trying to build, you really need to be looking

45. 8:10

    at your data. And all of us can look at the data, but some people are literally professionals at just looking at the data. That intuition takes a long time to build and will level up your team.

46. 8:25

    So unfortunately, why is a little bit of a harder question existentially, so we'll try to scope it down a teeny bit to just be why hire for these teams?

47. 8:35

    Why not just use your existing resources? The hiring theses for this initial team is going to look like the following. And the reason I wanted to kind of give you these theses is because ultimately, your leadership is probably asking you, "What's your hiring thesis?

48. 8:53

    What's your hiring thesis? We have this person on this other team. Why can't we just pull them over?"

49. 8:58

    If you can pull them over and satisfy these thesis, then you don't need to make a hire. If you can't, then you do. So for the full stack engineer, the hiring thesis is they're gonna integrate your system with an LLM provider, not very thrilling, but key, and build the minimum infrastructure.

50. 9:17

    The hiring thesis for the data scientist is evaluation, quality, and user data, continuously improving your AI product. I've already talked about this a couple of times in this talk already.

51. 9:28

    It is extremely important. A product person. I'm not super specific here that this needs to be a product manager, a program manager, a product developer, but you need someone whose spike is product.

52. 9:42

    And the reason is because they need to be talking to users. They need to be understanding what the jobs to be done are. If you personally don't know what the jobs to be done are for your application, that's a hole.

53. 9:55

    That's a hole in your team. A designer. A lot of times we think of designers as coming later in the process, but right now, none of us know what the shape should be.

54. 10:08

    Think about early technology and how different it looks for users

55. 10:14

    from what it eventually becomes after you've been doing it for five years. All AI products right now are clownish. You wanna see a great example of clownish?

56. 10:24

    That's a great example of cl- of clownish. And how many applications are we asking people to pay to use that don't look much better than this React? By the way, Claude wrote this React.

57. 10:37

    The reality is your AI application probably looks like shit.

58. 10:43

    I don't say that in a mean way. I just say that in a way of there's a lot of opportunities. I actually think ChatGPT looks like shit.

59. 10:51

    So what I challenge you to do is bring in a professional.

60. 10:56

    And finally, when you need an MLE, it's because you need to push your capabilities beyond what is the commodity intelligence.

61. 11:05

    That delta is what the MLEs are gonna bring.

62. 11:10

    Okay. So who? In my experience, the attributes that are strongly co-varying with a lot of impact are data intuition. We've already spoken about it a little bit, but there's a big difference between, "I made a semantic embedding of all of my documents," and, "I made a semantic embedding of all my documents, and when I looked at the

63. 11:34

    mutual distances, they fall into a very ridge-like jagged structure."

64. 11:42

    The former, okay, you did it. The latter, your retrieval's gonna suck.

65. 11:50

    Product mindedness. We are still trying to figure out what the actual utility of most of this is. I am incredibly skeptical that we are already at the boundary of the value for these things.

66. 12:04

    If we believe that there's a lot more juice to squeeze, then we must also accept that we don't know what the right products are right now. I would hope that for most of us, what we're building right now, what we're laser-focused on, what we're telling our investors is the breakout thing.

67. 12:21

    We look back in two years like [laughs] okay, so it was very naive, but I hope that that's the case for all of us. I don't wanna be building the same thing in two years, and I hope you don't either.

68. 12:32

    Urgency. [laughs] This is always the case for engineering teams, that urgency has a really high value. But when everything changes under your feet every three months, it's even more true.

69. 12:45

    A little ADHD can be useful, too, speaking from personal experience.

70. 12:52

    How? If you are giving LeetCode interviews for your AI engineering hiring, you are doing yourself a major disservice. [laughs]

71. 13:01

    I cannot think-- I've done a lot of LeetCode interviews. I'm personally very good at them. It's just like a stupid thing about me. I promise you this is not me, like, coping.

72. 13:12

    I cannot imagine a LeetCode experience I've ever had that gives signal on what is actually useful for building this shit. So stop it.

73. 13:23

    Make data intuition part of your hiring loop, and so too for product intuition.

74. 13:30

    My hiring loop includes a take-home. That take-home ultimately is a data-cleaning exercise. I've had candidates really surprised. They're like, "Okay, this seemed really easy. Did I, like, misunderstand the problem?"

75. 13:47

    And I'm like, "No, you did a lovely job. Thank you. You didn't overcomplicate things. You extracted the meaning from the data. You were able to look at the data and make some conclusions."

76. 13:59

    That sure sounds a whole hell of a lot like what I need them to do on the job.

77. 14:04

    My coding challenge, most of the people in this room would think is, like, too easy. But I promise you, I get a whole hell of a lot of signal out of it.

78. 14:15

    Invest in your coding challenge. Invest in data intuition and product intuition. One thing that Hex does that I think is really amazing is we have a product design interview.

79. 14:26

    Not my idea, but damn do I love it. Brian, would you be good enough to flip back to that one previous one? [laughs] The one that says, uh, stupid LeetCode?

80. 14:34

    Hell yeah. [laughs] Look for people who are paying attention, but not necessarily just riding the wave. I understand it's very exciting. I understand that a lot of people are really enthused right now, and they really wanna get involved.

81. 14:48

    That's great. What I'm really looking for, though, is people that are going a little bit deeper. They're playing with other AI products, and they're forming opinions about what is good and what is bad.

82. 15:01

    I recently had the privilege of hiring an AI engineer who had written a blog post about, like, AI design patterns.

83. 15:08

    There was a certain extent to which just from that blog post alone, I could have predicted that she was going to get hired. That's not to say that, like, I'm hiring based on blog posts, but the amount of awareness codified in that single blog post about design patterns, design thinking, what AI should feel like, that's a lot

84. 15:29

    of attentiveness. I need that on my team. She also happens to be technically very competent. [laughs]

85. 15:40

    So those are my main guidance for hiring these teams, not just the AI engineer profile itself, but more generally how to build these teams.

86. 15:54

    But I was curious if I could get my chatbot to give us any sort of, like, alpha. And so we'll go ahead and ask this question live and see what it says.

87. 16:04

    Oh, it has an opinion. I think it wants to speak to you directly. So AI Leader GPT has some messages for you.

88. 16:13

    Unfortunately, it's not pleased with me taking all of its good ideas and delivering them as if they're my own. [smacks lips] [sighs]

89. 16:21

    So this is AI GPT's key alpha. This is your bonus information for the day. It wants you to work with experts.

90. 16:31

    How many of you have worked with experts before in ML and AI teams?

91. 16:38

    How many of you for whom that you did worked with that in, like, data labeling? Maybe human-in-the-loop style.

92. 16:47

    So this is the key thing that can take what you are building and make it go much more smoothly. Work directly with the people that understand what you want the AI to do.

93. 17:02

    The AI capability should be modeled after interactions with experts.

94. 17:09

    If you're building a customer support bot and you don't have customer support people using that every day, you're insane. I'm building a data science co-pilot. I am a data scientist.

95. 17:22

    I talk to our data scientists every single week without exception. We ask them to use every single thing.

96. 17:30

    So this is so important. They are the secret to success here. I don't care how smart you are as an engineering leader. I don't care how smart you are as a machine learning engineer.

97. 17:42

    The only product that you could possibly be building that doesn't require you to work with other experts is if you're building an AI bot for generating fucking ML and AI to products.

98. 17:55

    That is the only one, because then you are still the expert.

99. 17:59

    This is by far and away the most important thing that you should be thinking about beyond hiring.

100. 18:07

     Thanks. Questions already

101. 18:17

     Fantastic talk. Thank you so much. [laughs] I learned a ton.

102. 18:19

     Thank you.

103. 18:20

     Um, could you maybe a- at like a high level give an example for, um, data intuition and the product design side sor- sort of prompts that you're doing? 'Cause I, I, I f- I find...

104. 18:31

     I think the coding one's a little bit more deterministic and easy, but those I'm not really sure where to start. But th- it sounds like an excellent way to conduct the interview.

105. 18:39

     Yeah. So this is specifically for getting data intuition signaled during the interview process? Yeah. So for me, it is actually part of the interview process. Like, I'm giving them a large set of data, and I'm asking them to, like, form some opinions about some of the data contained therein.

106. 18:55

     So roughly a clustering problem. But the secret is there's, like, no actual good clusters. There's no actual, like, objective way to cluster that data. So what I'm hoping that they're going to do is be able to pull out some sort of, like, latent meaning in that data.

107. 19:11

     So I do this as a take-home, because, one, I think people... I mean, we all know that people program substantially worse during an interview process. And also, like, how often is your manager staring over your shoulder when you're doing data anal- uh, data analysis?

108. 19:26

     Not that often. So, like, I don't see a lot of value in that. I give them seven days to complete a take-home challenge. They get to look at the data.

109. 19:32

     They get to write up a little report. And then I do a live interview with them where I give them feedback on their proposal. So this gives me a couple things.

110. 19:40

     One, I get to see how well did they do, and I get to really, like, talk to them about it. Two, if I misunderstood something about their approach, them talking through that notebook with me is a really good opportunity for me to say, "Oh, actually I misunderstood what you did."

111. 19:56

     Three, I'm gonna give them a lot of feedback. This has two important effects. One, they're going to learn what it's like to get feedback from me. Am I an asshole?

112. 20:07

     I guess we'll find out, and they'll know by the end of the interview if giving, getting feedback from me sucks. That's really important for them to make a decision about working for me.

113. 20:16

     And then on the flip side, I get to learn what they're like to interact with when I'm giving feedback. If they did a really great job, a lot of my feedback will, like, "This is really cool.

114. 20:24

     Where could we go next?" One of the things that your responsibility is as a leader is to always have feedback, period. And so it's your responsibility during the interview process to show them what that experience is gonna feel like.

115. 20:39

     That's a big part of the matching problem for hiring. And then finally, this interview is an opportunity for us to talk about sort of like how do they think through what is the minimum deliverable on some given task?

116. 20:53

     I give them four hours over seven days. If they turn something in that's clearly 15 hours' worth of work, that's a red flag. If they turn something in that's, like, really well scoped for four hours, that's a green flag.

117. 21:08

     And frankly, if they turn in something that's, like, overly simplified, I have the opportunity to say, "Hey, I think this is maybe, like, a little bit, like, under what I was expecting.

118. 21:18

     What was your logic?" And sometimes, and frankly I've hired one of these people, the feedback was, "I really didn't see much value in going any deeper until we reviewed this."

119. 21:29

     Talk about a green flag. That's a fuck yeah. Like [laughs] get in here. Like, so that is why I think the style of interviewing is so important, and getting that, like, data intuition out of it is the core goal.

120. 21:44

     But this format allows me to sort of like tag on a lot of extra signal.

121. 21:49

     And the product design part, sorry, that was sort of-

122. 21:52

     That one's a little bit harder for me to get into, but basically we ask you to, like, design a physical product, and they meet with our design team, and it's incredible.

123. 22:00

     I did it myself, and it was my first time ever doing a product design interview. And I was like, "I wanna work for this company." Like, this is so heads up and so clever, this is a great company.

124. 22:11

     Hi there. Uh, didn't hear a single plug for security. Um-

125. 22:16

     Oh, yeah

126. 22:17

     ... w- where does that live on this story arc?

127. 22:20

     Totally valid. Um, I tend to think that security responsibility doesn't lie within the team building the AI capabilities. I think that they should be security savvy, but I think ultimately, like, most organizations should have security professionals that are able to help you make great decisions.

128. 22:39

     I do believe a lot, and my security friends are going to, like, roast me for this, but, like, I do believe a lot that, like, [laughs]

129. 22:46

     strong engineers, strong software engineers should be constantly thinking about sort of like the adversarial nature of humans interacting with software, and be thinking about, like, where they're bringing up risks.

130. 22:58

     But I tend to think a lot about security as lying outside the team. You raised your eyebrow in reaction to my comment, so I'd like to ask. Tell me why you think I'm wrong.

131. 23:08

     So I, I work in a really heavily federally controlled environment at Los Alamos National Laboratory, and-

132. 23:14

     I've heard of it. [laughs]

133. 23:17

     Yeah. Several of our-

134. 23:17

     It's blowing up lately. [laughs]

135. 23:19

     So where we've had the most success is bringing security people to be part of our internal product teams at the beginning because we face a federal, highly regulated, we're gonna get a billion questions about this, um, and we don't have to retrain everybody outside in a separate security team.

136. 23:39

     So that's why I raised my eyebrow. We've, we've seen that pattern work to, to grab a smart and engaged security person early, um, to be part of that effort rather than trying to bolt on for a defense ops product.

137. 23:51

     I love that. I think that's really, like, insightful and really, like, meaningful. In sectors that are a little different than mine, I think that makes 100% of sense. And, like, even in my domain where we have a lot of, like, sensitive customer data, we sign BAs with every, like, provider that we work with.

138. 24:11

     Like, I do... I mean, bluntly, like, I take on a lot of that responsibility personally as the team lead. Um- But I, I can absolutely see value in what you're talking about.

139. 24:22

     I think that is completely right.

140. 24:25

     I would suggest that's part of the product team though, right? Like, you guys have security as a, as a product that gets baked in.

141. 24:31

     Yeah.

142. 24:32

     So it's... Yeah.

143. 24:35

     Yeah. Hi. Um, you're talking about creating new teams and hiring team, and now I see another very important, um, aspect is that upskilling or reskilling existing teams. What would be your advice as well?

144. 24:49

     Because, uh, in my, in my... I mean, it's, you're not always, like, creating new teams. You're always doing with, like, human depth, you know? Like existing teams and software engineer and, you know, most of them can be reluctant or can be, you know, this kind of routine software engineer being there for years and years.

145. 25:08

     What would be your two cents about, like, rescreening, upskilling, and making all this mayonnaise work with a new team, you know?

146. 25:17

     Yeah, I think this is really important. So, um, and I'm gonna repeat back to the question, make sure I totally understand. So your point is I focus a lot on zero to one for teams, but when you want to take an existing product team and you want to, like, add AI capabilities and make sure that they're set

147. 25:32

     up for success. Is that correct? Perfect. Cool. So, um, a really, really good point and really important. Um, I think...

148. 25:41

     I believe that the AI capabilities at any given company should have at least one team who's responsible for building the infrastructure to make that easy. I don't-- I know Netflix is, like, very divergent in thinking from what I'm about to say, so I'll caveat with that.

149. 25:57

     But what I do wanna add is, like,

150. 26:00

     in most companies, every individual product team should not be going up and setting up their relationship with OpenAI separately. They should not be going and figuring out how to, like, build a prompt, like, uh, infrastructure in your software.

151. 26:15

     They should not be understanding what the eval- evaluation system's gonna look like. I really think all of that should be coalesced for every given company. There should be one team responsible for that.

152. 26:26

     And then what I believe works very well is to have other teams, like you're talking about, product teams, then say, "We're going to treat you like a different infra team.

153. 26:36

     We're going to ask certain things of you, and then we're going to have," very similar to the profile that we talked about, "one person who's really responsible on that product team for interfacing with the platform team."

154. 26:48

     I worked at Stitch Fix for a long time, where the data scientist had access to a data platform team, and the data platform team's charter was do whatever they possibly can so the data scientists can move as fast as possible.

155. 27:01

     I really believe in that model, and I've seen it personally as a consumer be incredibly powerful. We were trying to, like, build new models, deploy them, and this is not so different.

156. 27:14

     And so I really believe that, like, having a, a centralized, like, platform team for AI at your company has so much leverage. So that's a little bit of my, like, opinion here.

157. 27:25

     The one caveat is, again, I know Netflix disagrees with this persona, and every individual product team at Netflix has the, like, open, like, opportunity if they just wanna go do it from scratch.

158. 27:36

     Go ahead, have fun. But there's a little bit of, like, reason why that makes sense for Netflix and not for most of us.

159. 27:42

     We're at time, but this is gonna be our last question. Sorry. Let's go.

160. 27:46

     Uh, hey, Bryan. Thanks for the great talk.

161. 27:48

     Thanks.

162. 27:48

     Um, I think I-- my-- what I was hoping to get your opinion on is when you're hiring or maybe not reskilling your team. You mentioned a lot of great attributes that you should look at, like data literacy, your urgency, and your general enthusiasm for, I guess, generative AI products in general.

163. 28:03

     So I was wondering, when you're doing hiring, what do you think are some of the attributes that are non-negotiable, and what do you think are some attributes that are trainable?

164. 28:10

     In the sense that if you hire a software engineer who has a lot of, um, enthusiasm and a lot of skills that could benefit but doesn't really have the data literacy, do you think that's okay?

165. 28:18

     Or if you hire a data scientist who might not have the urgency but the background. So different-

166. 28:21

     I think all three of the attributes I mentioned, there needs to be some kernel there that I can develop. Zero on any of them scares me. But there's one latent feature that I didn't mention that I've never personally successfully trained and is a really big, powerful feature in my model, and that's curiosity.

167. 28:42

     Oh, actually, that's... I think that's a fantastic way to end. [laughs] Give it up for Bryan. [upbeat music]