Structuring a modern AI team

Talk transcript

1. 0:00

   [upbeat music] All right.

2. 0:15

   Thanks, everybody, for joining today. My name is Denys Linkov. I lead the machine learning team at Wisedocs, and I'll be talking about hiring a modern AI team.

3. 0:23

   So who's heard this message before? We are now an AI-first company. We've seen companies like Shopify, Duolingo, Zapier all make these announcements saying that they're AI-first companies.

4. 0:35

   And they're saying that there are new expectations that before you hire a person, you need to make the, the claim that you can't hire an AI agent or use AI.

5. 0:44

   We're now seeing big tech companies and many companies in general sharing how much code is being written by AI systems and how this is going to lead to the extinction of the software engineer.

6. 0:55

   So now, if you're in the position to hire people, you'll ask the question, now what?

7. 1:00

   So I'm gonna talk about three main themes today. The first one is the anatomy of an AI team, the second is the evolution of a generalist, and the third is the question of hiring.

8. 1:10

   So let's start off with the anatomy of a team.

9. 1:13

   So there is a spectrum of different companies, and this is where we should start. We have technology companies where technology is the core value proposition that is being offered.

10. 1:21

    We know big tech companies, we know many startups. There's also verticalized solutions or services companies such as Palantir or the company I work at, Wisedocs. And there's also tech-enabled companies where the core product is not technology, but it is something that benefits dramatically from, from good tech.

11. 1:36

    You can think about banks, retailers, small-medium businesses.

12. 1:40

    So who here is in each group? Who here is at a tech company?

13. 1:44

    Who here is at a verticalized or services company? Who here is in tech-enabled? Okay. Pretty good mix.

14. 1:52

    Now, each of these different companies have different challenges. Typically in a technology company where we've seen blunders is the lack of domain knowledge when launching a product, there's usually some kind of business misalignment with technology.

15. 2:03

    In the middle, either everything goes right or everything goes awfully poorly. And on the tech-enabled side, there's usually some kinda tech challenge, right? Because tech is not your, your core value proposition.

16. 2:14

    So you make different decisions based on this, right? You typically buy data or buy expertise if you're a tech company, right? You go to a vendor and say, "Give me labeled data."

17. 2:23

    In the middle, you either have everything or you have nothing. Uh, and from a tech-enabled company, you usually buy technology either through a service provider or, uh, a true end-to-end solution.

18. 2:36

    Now, I bring this up because every company, every organization, and every person has a different perspective on the role of technology solving our world's problems. This is my stance.

19. 2:45

    I think we have ninety percent of the technology to solve the problems of humanity. Now, this might be a controversial perspective, but I'll show you why.

20. 2:54

    The fax market still exists. Billions of dollars are spent on faxes, and the market is growing. [audience chuckling]

21. 3:06

    In 2017, only three percent of payments in the US were contactless. That number is n- now higher, but we're still paying in archaic ways. Checks are still a massive part of the market.

22. 3:18

    And it took forty years after the introduction of personal computing of the Internet for medical systems and electronic medical records to become digital. This number is much higher now, but it takes time for technology to be adopted.

23. 3:31

    And many of you might have seen this in your industry and your job as well, that technology is not the thing that's stopping you from achieving success.

24. 3:38

    So this is the core question. Is technology the limitation of our success?

25. 3:45

    And it's not about technology, it's how we use technology, and the way you build your team should reflect this by understanding the problems that you have.

26. 3:53

    So going to this question of do you need to hire an AI researcher? A lot of times when ChatGPT was coming out, every team is like, "I need an AI engineer.

27. 4:01

    I need an AI researcher," and it's not always smart to do so, right? Up until you hit a certain scale or a certain need, a specialty, it does not make sense to hire an AI researcher to work on models.

28. 4:11

    Pre-training models, even fine-tuning models of a certain capacity is not necessarily the first thing that you need to achieve the value that you need to get. There's a lot of transformation work that, that goes in before that.

29. 4:23

    Now, in certain domains, the best tech is essential, right? So if you're working at OpenAI or a model provider, Anthropic, Google, or some of the startups, you want the best team who's working on that because that is a product, as we covered.

30. 4:36

    So here I'll, I'll propose a wager for you. Are people here familiar with Pascal's wager philosophy? I'll give you the successor of that, Ampere's wager, if you're fil- familiar with graphics card architectures.

31. 4:48

    Here's your trade. You trade your team for five researchers from the top labs, and maybe you need to throw in some cash and first-round picks for that as well.

32. 4:57

    But do you make this trade? Do you trade your team that has domain knowledge, has worked in the area for five AI researchers? And I want you to think about that.

33. 5:07

    So we go back to the question of what does an AI team need to do?

34. 5:13

    There's a lot of stuff, right? We start off with defining use cases. We wanna go through and integrate with products, right? We're not doing greenfield everywhere. We wanna measure ROI, find the right data.

35. 5:23

    We wanna test and refine workflows, build the interfaces we need for success, sell this product, and make our customers care.

36. 5:32

    And it's not one person who does this job. You can't just say, "AI researchers, go make me ten million dollars from this product," unless a very specific niche.

37. 5:43

    So this means your success is not one job, unless you're a founder, but we'll skip that.

38. 5:49

    So the goal here is that you need to have a comprehensive AI team, and you need to figure out how are you gonna structure that.

39. 5:56

    And the thing that we need to remember is that companies aren't just one team. It's not just my AI team owns this small segment, this deployment or whatever. Otherwise, you ship your org chart and you get some weird product behaviors.

40. 6:11

    So identify to yourself what is your bottleneck? What is stopping you from achieving success?

41. 6:16

    Is it shipping features? Is it acquiring users? Is it retaining users? Are you monetizing correctly? Are there scalability issues? Are there reliability and observability issues, right? All of us have probably run into these things as we were deploying AI products, so we need to make sure we can prioritize all these things and hire accordingly.

42. 6:37

    And these are all questions that you need to answer when building an AI team.

43. 6:42

    So key takeaway here is what kind of team do you need? And only you know that answer.

44. 6:47

    Let's talk about generalists and why I think they're important.

45. 6:51

    So in 2021, I was building a first machine learning team, and I adopted an approach where we hired generalists. We supported them by automation across the board.

46. 7:02

    So at the time, I was hired to a conversational AI company working on a platform. Sorry, let me rephrase that. AI agent building platform. Just wanted to make sure you guys understood what that meant.

47. 7:13

    And I was hired with the mandate of we want ML. That was my job description.

48. 7:19

    Change that. We want AI. So after working with the business teams and leadership team, the-- this was the, the final set of goals we set. We wanna serve hundreds of thousands of concurrent models.

49. 7:29

    It needs to be multi-domain. It has to be low cost, and we wanna support real-time training and serving. Those are some tough goals.

50. 7:36

    So this is what we did. Uh, we wrote a, a custom MLOps platform for deployments to, to match our requirements. We mainly fine-tuned encoder models. We built RAG as a service, and as a team, we own six microservices end to end.

51. 7:51

    So the three areas I focused on building the team was model training, model serving, and business acumen. Now, you might say, "I want top grades in all these things," but that's a lot of money, right?

52. 8:03

    And as a, as a team leader, as somebody who manages the budget, you don't have infinite money. So we have to pick along this axis where do we want each of these skills to lie.

53. 8:13

    For model training, we, we don't want somebody at the very bottom, but we don't need somebody who can train GPT-3. And w- basically, we went across and said, "Okay, what are the key requirements?"

54. 8:23

    For model training, we said somebody in the, the upper half who knows general architectures of models, uh, can do encoder fine-tuning, does some data engineering using Hugging Face is okay.

55. 8:33

    That was the bar we set. On the model serving perspective, on the first round, I was the first engineer at the company. I spent a lot of time on building the ML platform, but that was something I was comfortable with coming from a cloud engineering background.

56. 8:45

    Now, after that, there was enough abstraction built in that we didn't need somebody who knew the intricacies of how Kubernetes works and how we did serving or training, but the capability to use these abstractions and understand the trade-offs that were being made.

57. 9:00

    And what I did focus on is the ability of our engineers to get on calls with customers, right? We didn't need a business development rep who would just call-- cold call people for fun, uh, but we need engineers who didn't say, "My job is coding in a basement."

58. 9:13

    Right? So w- we went through and understood these trade-offs that, that needed to happen.

59. 9:18

    Now, in 2024, I was building another team, uh, the new organization that I joined, and similar approach, but open source had advanced. When I was building the original ML platform, we didn't have things like shadow deployments or A/B testing in a lot of the platforms that existed, and we had a specific use case.

60. 9:36

    Now, since then, what's important to recognize is that all these skills that you're prioritizing don't necessarily need to be one person. They can be multiple people. You just have to find a way to make the team work.

61. 9:47

    So once again, we, we set similar, uh, structures. And in this case, because open source ha- had advanced, uh, in a number of different ways and commercial models had advanced, some of the things shifted around.

62. 10:00

    On the, on the training side, using commercial APIs and, and prompt tuning and model fine-tuning commercial models became important, but we also expanded our scope. We're now using decoder and encoder models, which each have their nuances.

63. 10:13

    Uh, on the serving side, uh, because we were using a open source offering, we didn't need to write our own platform, which was nice. And on the domain side, again, because of the nature of our business of doing medical record processing, there's a whole nuance of what that domain knowledge was.

64. 10:29

    So that bar increased in a different way.

65. 10:33

    So now that we know what kinda skills we need for our team, we can identify this threshold and balance the budget, right? We can't just ask for infinite money unless you're a specific subset of companies.

66. 10:44

    You might have this question, what if I already have a team? I have forty people, a hundred people. What do I do? How do I reskill, upskill? How do I manage this team?

67. 10:53

    So we need to figure out what the goal of the team is, as we were referring to, and I typically like to think about it through inner and outer loops.

68. 11:00

    So inner loops are the daily activities that the team needs to accomplish together every day, uh, to be successful, and the outer loop is the broader set of activities that will set you apart.

69. 11:10

    And you might not need constant interaction with that, but they're really important.

70. 11:14

    So in my current team, this is how we typically structure it. Uh, so we'd have model training, prompting, product requirements, model serving, some domain experts, and the capability to build business cases as the core nucleus of our team.

71. 11:27

    And again, as you're building your team and your function within your domain, these will be different. But this is a framework to understand what are my priorities. And we need to have the expertise in our outer loop as well to, to further differentiate our company and our team.

72. 11:43

    If you have a weak technical loop on the inside, you're gonna struggle with the technical execution. If you have a weak domain loop, you're not gonna find product market fit.

73. 11:51

    So you need to make sure that you really understand those feedback loops and the collaboration loops that exist within your company.

74. 11:59

    Now, depending where you are, uh, at the stage of your AI strategy, uh, all of us fall on a different spectrum. You win with different types of people. You win with generalists at the beginning when you're trying to find that fit, trying to make that basic progress until you get to the point where you exhaust the knowledge,

75. 12:14

    and you need to move into a more specialist model. So once again, on the general side, most companies, as they're going through transformation, fall in that category. Once you get to a really good stage for your model training, serving, and so forth, you need specialists to push the extra five percent of performance there.

76. 12:33

    So generally, my perspective is generalists are good because they're adaptable, and in most cases, you're, you're good enough with a general, uh... a generalist who can do many different things beyond just writing code.

77. 12:46

    Let's talk about upskilling, reskilling, and hiring. So I think there are three main things as we continue go- to go through this AI wave that you need to do.

78. 12:55

    People need to learn to build, you need to become a domain expert, and you need to be human facing. So we've talked about vibe coding and prototyping. We should go from static product requirements to functional prototypes that take those details and, and elicit them, right?

79. 13:09

    We never want to have those conversations again, those dreaded conversations with PMs and engineers being like, "That wasn't in the requirements," or, "That was an ed- an edge case," right?

80. 13:18

    We wanna shorten that feedback loop. We wanna make sure that people are writing evaluations, that domain experts aren't just providing input and feedback, that they're the ones writing the use cases, defining them, and having the literacy to, to work with LLMs directly.

81. 13:32

    We need to make sure that engineers are on customer calls, so we shorten those feedback loops. If your engineers say, "Sorry, I can't talk to a customer," um, that's a learning opportunity.

82. 13:43

    Finally, you need somebody to sell your product.

83. 13:46

    Now, the way my team works is that we have weekly cadences to learn. Every week, we have a new topic, either with myself or other members of the team, that is brought to the table for 30 minutes, and we learn the underlying key priorities of our team and our company, and we make sure that every week we're

84. 14:00

    upskilling ourselves. If this sounds intense, the consequences of not doing this are much higher.

85. 14:06

    Let's close out on hiring. When do you need to hire? I believe that people need to be hired for two main reasons. One is to hold context, and the other is to act on context.

86. 14:17

    So it's important that if you have too few people on your team, things are getting dropped and you can't execute on your priorities. You might ask the question, can't AI agents with a massive context window do this?

87. 14:28

    Maybe to some extent, but you need expertise to be able to verify that this context and this execution is correct, and to have expertise, you need to have context.

88. 14:38

    And finally, humans should be accountable for the systems that we build, uh, as we have in the old IBM quote, right? We can't hold a, a machine accountable.

89. 14:47

    So who do you, who do you need to hire?

90. 14:50

    So we're hiring on a budget, and going back to everything that we've talked about today, you need to know your team composition and the needs that you have to set up this budget, right?

91. 14:59

    If you're trying to hire the top researcher, it's gonna be very expensive. If you're gonna hire a, a generalist AI engineer, will be quite a bit cheaper.

92. 15:07

    Now, it's also important when you're hiring is that you're not just following trends. Who here has heard the trend that junior engineers shouldn't be hired or just using AI agents?

93. 15:17

    Okay, some people are asleep. [laughs] Now, the counterpoint here is why is YC running a s- a school, an AI school for students and young people on AI? Two thousand people coming to, to San Francisco in two weeks.

94. 15:33

    Why are they doing that? Certainly, entry level positions, if they were useless, they wouldn't be bringing in all, all these young people. So make sure that you verify the trends that you're seeing, and, uh, think from first principles.

95. 15:44

    What do I need? What is the team composition? Is it new grads? Is it people with 30 years of experience? Is... What, what are the retraining opportunities, right? Uh, there's lots of ways to, to build a great team.

96. 15:57

    Now, just repeating this because I've seen so many companies do this, ask relevant questions to the job. Stop putting people through leet codes that have nothing to do with the job, uh, and now that LLMs can solve it, it's not a great way to evaluate either.

97. 16:12

    So we go back to Ampere's wager. Um, you have the question of, am I going to have five researchers from the top labs, or am I gonna build my team in a domain specific way?

98. 16:23

    So for example, in my company, I'd rather have the team on the left with the domain expertise, the ability to sell work, have empathy with customers, rather than just having five researchers.

99. 16:33

    That's the way that our domain and company are structured.

100. 16:37

     Now, you can also answer Blackwell's wager, which is, do you want GPUs or a team? Uh, so that's a, a story for another day.

101. 16:46

     So overall, we have three main lessons from today. The first one is it's important to start off from the beginning and say, what tea- what team do you need to win?

102. 16:55

     Once you know that, you'll start noticing that cross-functional teams will continue to be effective, but they'll be built in different ways. The overlap will be greater, but all of us will have the opportunity to work with AI systems and contribute to our product.

103. 17:07

     And finally, we need to continue learning. Th- this is a must, right? The world moves too quickly. We, we have Pelican evaluations now for the past six months rather than the past year, right?

104. 17:18

     Hopefully, that's an illustration of how fast the world works as well. So keep up to date, keep moving, make it part of your culture to keep learning. So thanks everybody for joining.

105. 17:28

     Uh, this-- these are my handles if you wanna connect afterwards, and I'll be here, uh, later on if you have any more questions. Thank you. [upbeat music]