Shipping Products When You Don’t Know What they Can Do

Talk transcript

1. 0:00

   [upbeat music] Uh, yeah, I mean, the actual title has curse words in it.

2. 0:17

   I will probably be cursing a lot. I didn't know if I would get into the track if I actually published the curse words. [laughs] Uh, I'm one of the founders of Teammates.

3. 0:24

   I'm gonna wear my product manager hat today. I'm assuming this room is, like, mostly product folks, probably product-minded engineers as well. Um, but I'm gonna just, like, wear the product hat.

4. 0:34

   Um, a little bit about Teammates very quickly. We make, uh, a platform for designing and managing an entire digital workforce. So in AI engineer parlance, right, we're building agents.

5. 0:44

   Um, but I would think of it, like, two ticks up from that because what we really believe, it is the experience, the interaction patterns of humans and computers working together.

6. 0:53

   So I wanna talk to you about my favorite teammate. Uh, this is Stacy, Stacy Hand. She, she actually got promoted since this slide. She's an L3 engineer right now on our team.

7. 1:03

   Um, she's awesome. She looks like a hamster. All of our customers get to design whatever teammates and avatars they want. They give them personalities. It's all really fun. And, uh, Stacy lives inside all of our collaboration tools, right?

8. 1:14

   So she has a Google Workspace account, right, for Gmail. She has a Slack account. We truly leaned into giving all of our teammates identity.

9. 1:24

   And she sends emails, or I forward her emails, and she hangs out in Slack, like, in the public channels. And she's Gen Alpha, which, like, is... I don't know what.

10. 1:34

    I feel really old. I don't know what she's talking about. She's constantly like, "Six Seven," and I'm like, "What are you talking about?" And I can tell from this room that none of you are, have [REDACTED:age].

11. 1:43

    Yes. [laughs] No? Okay. [laughs] Uh, there you go. So yeah, you're rolling your eyes as well. But anyway, this is Stacy, and this is sort of how my sales pitch goes, right?

12. 1:50

    It's, it's, you know, a little m- more formal than this, but, like, this is generally the pitch. And, um,

13. 1:57

    I got asked a question at some point recently, which was... Oh, yeah, more of the pitch, right? She, like, uh, shares Google Docs, Google Sheets, and she said, "Hey..."

14. 2:04

    Or a customer said, "Hey, can I tag my teammate in a Google Doc comment?" And this, like, gave me pause 'cause I was like, "Well, I had never actually thought about that before."

15. 2:14

    And so in the back of my mind, I'm like, "Well, of course you can. Your question is, like, what's gonna happen?" So I'm like, okay. So I'm like, you know, doing math in my head.

16. 2:20

    I'm like, "Okay. Well, we don't have webhooks. She probably won't... Or, like, a webhook from the comment. Okay. But she's gonna get the email notification in the email that comes from Google.

17. 2:28

    Does it have the comment and the conte- uh, or maybe a link?" Well, I'm like, "I have no idea." Right? Like, I actually don't know what's gonna happen. And this was, like, the impetus for this talk is, like, how do I ship a product?

18. 2:41

    How do I develop a product? How do I talk to customers? How do I instill trust when I don't know what my own product can do?

19. 2:48

    And, like, it's really weird, and sometimes I'm like, "Well, is this just 'cause I'm an idiot?" And like, well, since it's my talk here, I'm gonna say no. And sometimes I'm like, "Well, is this because what we're building is so far out there?"

20. 3:00

    Right? These are just, like, truly autonomous agents that can use any t- And it's like, I don't think that's it either. I think what's happening is the product management discipline is going to undergo a transformation, a shift in evolution, whatever you call it, that is super profound, and we may or may not totally realize it yet.

21. 3:16

    Because I think in the engineering world, we're like, "Oh, well, we have, uh, you know, tools in our IDEs, and we have code gen." And, like, we sort of are starting to squint in understanding maybe how the discipline is changing.

22. 3:27

    I don't think we really understand how product development is changing and evolving, and, like, what are the new tools and practices, and how do we forget everything we've learned in the past?

23. 3:38

    Um, why is this true, right? If it's, if it... The answer's not Ben's an idiot, and the answer is, uh, not this is, we're way out there. It's two reasons.

24. 3:45

    Number one, if all our products are built on top of LLMs, and plus or minus they are, like, we don't know and we can never know what the LLMs know, right?

25. 3:54

    So it's, like, inherently in what we're building is, like, we don't know what the foundation is. Like, you don't have to know what your database, like, how it works, but, like, you generally know that it's, like, the surface area, the interface that's exposed.

26. 4:03

    We don't understand this for the, the models. And the other thing is the expectations from customers are just boundless, right? We're just like, "Hey, here's a text box." I mean, that's probably not a good interface, but, like, essentially we're like, "Here's a free text box."

27. 4:16

    And if it's anything other than, like, a Help Me Write button, you're essentially inviting customers and users to just do whatever they want, right? So we have this, like, boundless surface area built on top of a product that we don't understand.

28. 4:28

    And so the question now is, like, how do we adapt? So let's me- let me actually pick on this Google Doc comment thing for a second, right? So if I was wearing my, like, traditional PM hat, and I'm like, "Okay.

29. 4:39

    Well, I need to make a feature that's going to, uh,

30. 4:44

    read and respond to Google Doc comments." And so in my head, I'm like, "Okay. Well, uh, does Stacy have access to the Google Doc? Uh, if she gets tagged in the comment, should she reply directly in the comment?

31. 4:56

    Should she reply at all? What happens if somebody else comments in the thread? What if someone comments in the thread that's not addressed to her? What if it's someone else?

32. 5:04

    What if it's, what if it's her Doc and someone else commented to someone else, but she gets the noti-" Like, there's just so much to, like, think about and reason about.

33. 5:11

    And so I'm like, "Okay. Well, I'm not building a Google Doc commenting product, so I'm not gonna spec all of those things out."

34. 5:19

    And, like, what's worse is, like, you also probably wanna tag her in Linear tickets, right? And what, what's, what's the book? Like, uh, If You Give a Mouse a Cookie, right?

35. 5:26

    It's like, if you give a mouse a cookie, well, you probably wanna, like, tag her in Figma as well, and you probably wanna tag her in LinkedIn posts. And, like, and so we're not a team that's building a generic commenting reply agent system, right?

36. 5:40

    So then the question is, like, what are we supposed to do, right? As, like, a product manager who realizes, "Okay. I have this, like, boundless surface area. How does the practice need to change," right?

37. 5:49

    And this here, this is the core of, like, what I wanna, what I wanna talk about today.

38. 5:54

    So I'll do, like, three, uh, highfalutin ivory tower ideas, and then I'll talk through some, like, practical ways to, to make this real.

39. 6:02

    So the first one is this mindset shift to, like, think in affordances and not, like, specific requirements. So it's not if... You know, as a user, if Stacy replies in the comment thread and she has re- like, that's not how we would think about it anymore.

40. 6:17

    It's the affordance. Oh, she has affordances to comment, or she has affordances to communicate, or, or to email, or to collaborate, and we're going to trust the LLMs, we're gonna trust the agentic workflow, the work planning, like, all of the things inside of our, um, you know, our beautiful 12-factor agent.

41. 6:32

    We're gonna assume that that will understand. But it's the affordances that we need to think about, not the individual features. Which is really weird, and it's not typically how product people have ever thought before.

42. 6:42

    And I would say, actually, this goes even further, which is behavior is emergent. And this was the other thing that I ju- did not expect at all, like, starting in this space, was, uh, we don't...

43. 6:54

    Not only do we not know if things work, sometimes they do, and they work in ways we didn't expect. And so I feel like our job as product people is to discover functionality, is what are the right building blocks, right?

44. 7:05

    What are the right Lego bricks that we either give our engineering team, our product, our customers, let them compose, and can we discover emergent behavior? And that is one of the reasons that, like, this is the most exciting time I've ever built, 'cause we're actually building things and then discovering what they can do themselves, and that sort

45. 7:21

    of became the new job, in a sense, is discovering what's possible. 'Cause if you asked me, like, I could not sit down in front of a Google Doc and be like, "Oh, let me, like, type out what this thing should..."

46. 7:31

    I, I, I can't. I don't know how to do it.

47. 7:34

    And well, fr- even if I could, how do I then communicate it, right? So how do you... we communicate to a development team, to a backlog? How do you communicate exactly what should be happening?

48. 7:44

    It's like Figma doesn't, like, have the affordances for this, right? My, my PRD doesn't, like, have the affordance for, like, "Well, you should probably talk a little bit less gen alpha because you're making Ben feel old," or like, "Hey, you should be really polite."

49. 7:56

    How do we communicate and express these, these concepts? Right? So I think these are, like, the three, you know, high-level, uh, ways that, um, our practice needs to change, but, like, let's make it a little more concrete.

50. 8:09

    Okay, so evals. I'm talking about evals. Okay. It's really hard to make a slide with graphics of evals. I feel bad for the eval com- like, how do you ex- illustrate an eval?

51. 8:20

    So I'm gonna make you just look at pictures of various teammates from, uh, you know, across all of our customers. Um, okay. Who, who hates raising their hand at conferences when the speaker asks them? [laughing]

52. 8:31

    Okay, awesome. So here's my question, which is, okay, for the engineers here, who, like, legit, like, don't lie, like, writes and runs their evals?

53. 8:41

    Good number. And of the product people, who has visibility into the evals?

54. 8:47

    There's a hand. It's not bad. And, and do you look at them just because you have the visibility? All right. One. One and a half, two? Okay, great. So I would posit that evals...

55. 8:57

    Actually, I'll back up, right? So we all talk about evals. We're all gonna be embarrassed to say that we don't know really what they are. Evals are a testing framework for probabilistic AI for agents, right?

56. 9:07

    Like, if we think about the, uh, deterministic code, right? I withdraw $100 from the ATM. My bank account should have $100 less, right? Great. And I can test that, and I can write code to test that.

57. 9:20

    When the test is, like, was she snarky in Slack? It's like, well, how do you test that? How do you write that test, right? So we come up with this, this whole new discipline of evals, which is, well,

58. 9:32

    she should be a little bit snarky and a little bit funny, but not mean. And then we hand it off to another LLM to say, "Okay, well, hey, was that reply, like, did it meet that criteria?

59. 9:41

    And how often did it?" Um, it doesn't have to be 100%, right? So she should be, like, pretty snarky, but, like, not mean 80% of the time, or whatever the, uh, uh, business logic that you want, right?

60. 9:54

    So these are evals, and this is the world of evals. But here's what I would posit, which is it is the only way that we know what our software can do,

61. 10:03

    right? And which is why I love the idea of product people looking at the evals, right? Looking at, uh... 'Cause they become the new specification for the product, right?

62. 10:12

    And so as we're watching, you know, if you're downstairs in the expo gallery, you're seeing, like, new software. It's like, "Hey, bring the team in." And this re- a little bit reminds me of, like, the old, you know, for the, the old-timers here, like, behavior-driven development.

63. 10:22

    There was this period of time when it was like, "Oh, the business people are gonna write the tests, and that will get converted to code, and then the code will run."

64. 10:28

    And, like, the truth is, like, no one ever wanted to do that. Like, no business per- I don't even know who a business person is, but, like, they wanted-- We're, we're gonna do that.

65. 10:35

    But I actually think this is different, and I think this is pretty, um, a meaningful way to actually understand what the product can do, and a little bit begin to specify what it can do.

66. 10:46

    Okay. So I have vibe coding for a second, which we, which we all do, we all talk about. But I wanna talk about vibe coding in a, in a way that's really constructive.

67. 10:55

    And, mm, how do I sort of say this? It's very, very hard. And I, I kind of was like, like, "Oh, you can't do it in Figma. You can't do it in a PRD."

68. 11:02

    Like, what do I really mean? Well, it's very hard to, like, sit down in front of a blank piece of paper and, um, write what the teammate, the agent experience should be.

69. 11:13

    It's just really hard. It's hard to, like, imagine it, and it's not until you feel it. I mean, so much of what we're doing in this, like, human computer interface is visceral.

70. 11:22

    It's feel. It is like, "Oh, well, like, do they ask too many questions? Like, how many questions is too many? Oh, wouldn't it be great if they clarified exactly what you meant?"

71. 11:31

    Well, turns out that's really annoying. But when I wrote, like, the first spec, I'm like, "Then the teammate should ask a lot of clarifying questions." And we gave it to users, and they were like, "This sucks."

72. 11:39

    And I was like, "How would I have ever known that?" And the answer is because it's so easy to prototype and vibe code something and get the feels. And so this is the next thing that I'm, like, pretty excited about as a new product management tool.

73. 11:52

    It is being able to feel and experience, um, what it's like to interact with a computer, but, uh, uh, without just, like, uh, writing it or hoping that you have a clickable prototype that will work.

74. 12:06

    I will also mention that we have to be careful with vibe coding because I do not mean sit in the meeting and say to the engineering team, "How come this is taking two weeks?

75. 12:13

    I finished the feature during the meeting." Like, that doesn't- [laughs]

76. 12:18

    That doesn't win you any points, right? So it is no, no, no. This is never going to production, but what this does is it gives you the feel, the, the experience, right?

77. 12:27

    And so this is like the only way I know to like actually test and feel it out.

78. 12:32

    But do, do you, um, do you remember like the, the Claude, um, certainty issue? Certainly, I mean certainly. There was this period, right, where every time you ask Claude something, he'd be like, "Certainly."

79. 12:40

    And like, that probably like seemed really good when you were testing it for the very first time, and then like the fourth time when you're like, "Hey, can you do my taxes?"

80. 12:46

    He's like, "Certainly." "Can you write my like, uh, acceptance speech?" "Certainly." Like, oh, this is actually really annoying. But you don't realize that until you experience it. So like, that's why I like the vibe coding.

81. 12:57

    Okay. So great, we did all this development, and then the question is like, "Hey, we pushed to prod. Does it work?" I'm like, "I told you I don't know."

82. 13:05

    The question is like, how do you test? How do you like know that it's gonna do, uh, the things that you said it was gonna do? And I sort of alluded to this.

83. 13:12

    I'll go through this quickly. It's just really discover. Discover the functionality. And

84. 13:17

    there's an old joke. I'll tell the joke. QA engineer walks into a bar, orders a beer, orders two beers, orders zero beers, orders negative one beers, orders a lizard, orders a beer with a emoji, right?

85. 13:29

    He's like, "Great. This like bar is good to open." And the first customer walks in, asks where the bathroom is, and the bar blows up. [laughs]

86. 13:36

    Right? Like, great, great old joke. It's kind of how I feel these days. Like, I just sit and I'm like, "Oh, you know what would be cool? If they were to like start posting comments on LinkedIn about...

87. 13:47

    And then what if, what if they were like, every time I added like a track to my Spotify account, they can..." Like, these just like crazy ideas, but this is where like the emergent behavior comes from, right?

88. 13:56

    And so it's this mindset of like, let's just try. Let's just experiment. And it's, it's this like kind of growth mindset shift from like I'm going to write the features and the requirements to no, we're gonna figure it out.

89. 14:12

    This was a little bit unexpected for me, and this is, um,

90. 14:17

    how do you sort of report to engineering and then have things fixed by engineering, and what counts as a bug in this world? And that is really, really strange.

91. 14:27

    And I think as sort of, I don't know if it's like just a product role or maybe in a support role, like how do you know what is appropriate to escalate, to put onto the backlog, to flag as a bug, right?

92. 14:36

    It's like I'll, I'll keep picking on, on Stacy. You know, she, she gives me a really hard time, so it's fine. Uh, it's like, "Hey, she used too many emojis.

93. 14:44

    Like, put it in, in, in Linear." It's like, well, it's not really a bug. Like, show me in the spec where you told me not to use too many emojis, right?

94. 14:51

    It's, uh, it's almost like, um, like in our tickets, it's like, oh, you know, closed, done. Closed, duplicate. We need like closed, LLMs be like crazy, yo. Like, I don't know how to fix this, like just because it's probabilistically generated, so how do we know if it's right or wrong?

95. 15:07

    How do you know if it's a feature or if it's a bug? Right? And I think there's this element of, um, credibility that we need to build up. It's like, hey, we actually under-- we understand that for some use cases, like 80% is good enough, right?

96. 15:20

    This eval, we talk about evals, if it's passing 90% of the time, like that's a go. If it falls below 90%, right, that's red and we're not gonna ship it.

97. 15:29

    So I'll actually come back to evals for a second because if the eval becomes the spec and we can say, "Hey, we said at, you know, 100%, even though this is probability, you should never give a refund if a customer like can't prove that they bought the thing or whatever," like it is, it's like great.

98. 15:43

    That is our metric, and we can say, "Yeah, this is a bug." But if it's just a, a feel, it becomes really difficult. Again, this was totally unexpected that like, uh, debugging and assigning bugs would become like, uh, controversial.

99. 15:57

    Okay. Customers. So this part is, uh, I found this really weird, right? So if I think about like not wearing my like founder hat, but wearing my like typical product manager hat, right?

100. 16:09

     Like I go into a customer meeting, usually go with a salesperson, and like I'm gonna play a role, right? And so what's the role? Well, I'm either gonna play like visionary.

101. 16:17

     I'm gonna like, "Hey, here's our vision for the product. Here's our roadmap for the future. Like, let me help you understand, customer, like how we're, you're gonna come along on this journey with us."

102. 16:27

     Or, uh, sometimes I'll play the role of honest broker, right? It's like, "Listen, sales is like giving you a whole bunch of like... they're just like selling you a bunch of, of vaporware.

103. 16:35

     Let me tell you what's real. Let me tell you like, um, exactly what you can expect." And that's a role that you play, right? And I usually preface this with like the sales team beforehand.

104. 16:42

     It's like, "Yeah, I'm gonna be the honest broker, and like we'll give the customer confidence."

105. 16:47

     Today, I'm like, "Okay, I told you our vision for the future, our roadmap," and the customer's like, "You're full of shit. Like, none of this actually works." I'm like, "Right, I can't really paint the vision because no one b- actually believes it.

106. 16:58

     It sounds like witchcraft." And then I'm like, "Oh, well, then I'll be the honest broker, and I'll tell you how things work." But I just told you I have no idea how it works, right?

107. 17:04

     So this became very strange because I can't play either of the roles that I'm supposed to be playing. The f- future sounds like witchcraft. The present is literally I don't know.

108. 17:13

     So how do we do this? Um, I'll tell you how I've been doing it now. I don't know if this is like a 2025 answer or if this is like a durable answer.

109. 17:22

     Like if we believe that all of our products are for like, for all time, going to be probabilistic, then like we probably have to figure out how this world works.

110. 17:28

     What I've been doing now is really saying, "Look, we're inventing the future together," right? "We're pulling the future forward. The reason you are talking to like a crazy startup like this and you are thinking truly about like the future of how, uh, you know, AI and agents are gonna transform your business is because you are a future

111. 17:43

     thinker, and we are gonna do it together." And it's a little bit like, "Hey, let's compliment the customer. Let's like..." But it's not just like a false, you know, uh, uh, blowing smoke.

112. 17:51

     It's like, no, truly we need to figure this out together. And, you know, for 2025, I think that's actually the thing that is working the best, uh, best for me.

113. 17:59

     It's like, no, no, no, we have to do it together. And honestly, if you are expecting something different, like it, it's not time. It's not time for you to like embrace this world because this is, th- this is the, the way this world is going to work.

114. 18:13

     And so, I don't know, I'll conclude with like I've never had more fun building. I've never felt m- like both more inept and like more excited about what, what I'm doing or just the experience of throwing something out in the world and then just like having my jaw dropped, like I can't believe this happened.

115. 18:30

     And not only that, when we upgrade the models that are like underneath them, they just suddenly get smarter, and that's really weird too, right? It's just like all of a sudden they s- like start checking their work.

116. 18:39

     They're like, "Oh yeah, I just did a, a, a query to make sure that the row is properly inserted." And I was like, "Hmm, who told you to do that?"

117. 18:46

     And like, they're like, "I don't know. It just seemed like a good idea." I'm like, "That is a good idea. God, I wish I thought of that." [laughs]

118. 18:52

     Uh, but anyway, but I think this is the new world that we're working in. Um, the discipline, the product discipline, I think is going to change for everyone, and it's gonna change faster than we expect, and we all need to like adapt to just like operating in a world and forget so much of what we used to

119. 19:08

     know, right? A lot of the core, core ideas, listen to customers, solve real problems, like all of that obviously still applies. But the tools, the techniques that we've like relied on forever, I think are all getting upended.

120. 19:18

     And so anyway, glad you're all at the AI Engineer Conference. It's awesome to have product people here working together because, you know, we all have to, uh, you know, build awesome products together.

121. 19:26

     So thank you very much. [audience applauds] [upbeat music]