AI Engineer World's Fair 2025
Shipping Products When You Don’t Know What they Can Do
Read the talk
Shipping Products When You Don’t Know What They Can Do
An agent with email, Slack, and document access can surprise its own builders. Product work shifts toward discovering capabilities, testing behavior, and agreeing on what counts as good enough.
From a talk by Ben Stein
Can you tag your digital coworker in a document?
Stacy Hand has a Google Workspace account, a Gmail inbox, and a Slack identity. She sends email, receives forwarded messages, participates in public channels, and shares Google Docs and Sheets. She is also a hamster avatar: an agent at Teammates, the digital-workforce platform cofounded by Ben Stein. Customers design their teammates’ avatars and personalities; Stacy’s Gen Alpha persona includes slang that leaves Stein feeling old. Since the introductory slide was made, he says, she has been promoted to L3 engineer. The product is built around the experience of people and computers working together inside familiar collaboration tools.
Then a customer asks whether they can tag their teammate in a Google Doc comment. Tagging the account is straightforward. The difficult question is what happens next. Stein starts tracing the possible path: the team has not implemented a comment webhook, but Google should send Stacy an email notification. Would that email contain the comment, enough context, or a link she could follow? He realizes he cannot predict his own product’s response. Google’s documented notification behavior supports the email route—the notification includes comment and sender information—but that does not establish what Stacy will do with it. The missing webhook here belongs to Teammates’ integration, not a claim that Google offers no webhook notifications.
The uncertainty reaches beyond one integration. How do you develop, ship, explain, and earn trust in a product whose capabilities you cannot fully enumerate? Stein approaches that question wearing his product-manager hat. Engineering already has visible signs of change in IDE assistants and code generation. Product development’s corresponding changes—in tools, practices, and ways of expressing intent—are less settled.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One comment becomes an unbounded specification
Two kinds of uncertainty compound. First, builders cannot fully know what the underlying LLM knows or can do. A database can hide its implementation while still exposing an interface whose operations you understand. An LLM does not offer the same bounded account of its capabilities. Second, customers bring open-ended expectations. A narrow action such as a Help Me Write button limits the invitation; a free-text box invites users to ask for almost anything. The result is an expansive product surface resting on a foundation whose behavioral limits are unclear.
Try specifying the Google Doc interaction in the traditional way, and the branches multiply:
- Access: Does Stacy have permission to read the document?
- Response: Should she reply at all, and should the reply appear directly in the comment thread?
- Participation: What happens when someone else joins the thread or leaves a comment addressed to another person?
- Ownership: What if Stacy owns the document and receives a notification about a conversation between other people?
Each is a reasonable product question. Together, they begin to describe a commenting product that Teammates is not trying to build.
The scope does not stop at Google Docs. A customer who can tag Stacy there will reasonably want to tag her in Linear tickets, Figma, and LinkedIn posts. Stein compares the expansion to If You Give a Mouse a Cookie: satisfying one request creates the expectation of the next. Teammates is not building a generic comment-reply system either. Enumerating every conditional behavior across every collaboration tool would turn the product practice into an ever-expanding integration specification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design affordances, then discover behavior
The first shift is to think in affordances: the actions and interactions available to the agent. Instead of specifying every condition under which Stacy should reply to a particular comment, give her ways to comment, communicate, email, and collaborate. The LLM, agentic workflow, and work-planning system then determine how to use those capabilities in context. Stein invokes the 12-Factor Agents vocabulary while describing the engineering machinery underneath that product approach. The product decision moves toward choosing the capabilities available for composition.
The second shift follows from the first: behavior can emerge from the composition of capabilities. Sometimes an interaction works even though nobody explicitly designed it, and sometimes it works in an unexpected way. Stein describes the building blocks as Lego bricks supplied to engineers, the product, or customers. Choosing those bricks and discovering what their combinations enable becomes part of the job. Functionality is something the team learns about after building, as well as something it specifies beforehand.
The third shift concerns communication. Even when a product manager recognizes the desired behavior, how does it become a development task or backlog item? A Figma mockup can show a layout; a product requirements document can describe a flow. Neither readily captures an instruction such as making Stacy sound less Gen Alpha or making her more polite. These are qualities of an interaction that unfolds over time. The team needs a way to express them that engineering can act on and product can inspect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evals part of the product specification
That need leads to evals. Stein’s informal audience poll finds engineers who write and run them, but far fewer product people who can see them or actively inspect them. His concern is the separation: the evidence that describes the product’s behavior is not necessarily reaching the people responsible for defining it.
A deterministic test can check that withdrawing $100 reduces a bank balance by $100. A behavioral test has a different shape. Was Stacy’s Slack reply snarky? Was it funny without being mean? Stein describes handing the reply and those criteria to another LLM, which judges whether the response meets the desired standard. Repeating that process gives the team a way to ask how often the behavior is acceptable. His illustrative tone criterion is acceptable snark without meanness 80% of the time; it is a proposed business threshold, not a measured result.
Evals become a shared specification of acceptable behavior. Stein calls them the only way to know what this kind of software can do, which makes product participation central rather than optional. Looking at the criteria and outcomes lets product people understand current capabilities and begin to specify desired ones.
There is an echo of behavior-driven development: business participants write tests, those tests become executable, and the tests describe the intended system. Stein remembers the gap between that ambition and people’s willingness to write the tests. He sees a stronger reason to participate here. With an agent, inspecting the eval is a direct way to find out what the product actually does, not merely a different format for a specification everyone already understands.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Experience the interaction before committing to it
Some criteria are hard to choose until you have experienced the behavior. Stein’s first specification encouraged teammates to ask many clarifying questions. On paper, that sounded helpful: clarification should improve understanding. In use, customers found the repeated questions annoying. The specification had captured a plausible idea, but missed the cost of living through it.
This is the constructive role Stein gives vibe coding in product work. A working prototype lets a product manager feel the timing, frequency, and character of an agent’s responses. It provides something to react to when a blank document is too abstract and a clickable prototype cannot adequately express the interaction. The prototype becomes a tool for exploring and communicating the experience.
The boundary matters. Building a prototype during a meeting does not invalidate an engineering estimate of two weeks. Stein explicitly treats these prototypes as work that will not go to production. Their purpose is to make the desired experience concrete, not to establish that the production implementation is finished.
Repeated use also exposes problems that a first impression hides. Stein recalls a period when Claude seemed to begin every answer with “Certainly.” A single response could feel reassuring. Across requests to do taxes and write an acceptance speech, the same opening became irritating. The quality being evaluated belongs to the sequence of interactions, not just an isolated answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Shipping does not end discovery
Pushing the product to production does not resolve the question of what it can do. Stein returns to discovery through an old QA joke. An engineer walks into a bar and orders one beer, two beers, zero beers, negative one beers, a lizard, and a beer with an emoji. The tests pass. Then the first customer asks where the bathroom is, and the bar explodes. The engineer explored unusual inputs inside an assumed interaction; the customer introduced an ordinary request outside it.
Agent exploration requires that second kind of imagination. What would happen if a teammate started commenting on LinkedIn? Could adding a track to a Spotify account trigger some useful action? These are ideas Stein proposes trying, not capabilities he demonstrates. They illustrate a practice of probing combinations that were absent from the initial requirements. The team supplies capabilities, tries interactions, and discovers where useful behavior appears.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agree on what makes a bug
Discovery creates another practical problem: what should product or support escalate to engineering? Suppose Stacy uses too many emojis, and someone files a Linear ticket. Without an agreed criterion, engineering can reasonably ask where the specification defined too many. Stein jokes that ticket systems need a resolution category for LLMs behaving crazily, alongside closed and duplicate. Behind the joke is a real disagreement about whether an unwanted response is a bug, a preference, or ordinary variation.
Explicit acceptance criteria give the team a basis for that decision. Stein offers different thresholds for different kinds of expectations:
| Example | Acceptance rule | Consequence |
|---|---|---|
| Tolerant use case | 80% can be sufficient for some use cases. | Agree on the tolerance. |
| Release gate | An eval passing at least 90% permits shipping; below 90% blocks it. | Hold a failing release. |
| Refund policy | Require 100% compliance with refusing refunds when the customer cannot prove the purchase. | Treat a violation as a bug. |
These are illustrative product criteria, not reported Teammates performance. The refund example expresses a requirement; a perfect observed eval score would not prove universal compliance. What matters for bug assignment is that the team has stated the expected behavior before debating a particular failure.
The release decision itself can be a small, deterministic function once the behavioral judgments exist. For example, this TypeScript function applies Stein’s illustrative release threshold to a set of graded outcomes:
typescript
function releaseDecision(
outcomes: readonly boolean[]
): "ship" | "hold" {
if (outcomes.length === 0) return "hold";
const passed = outcomes.filter(Boolean).length;
const passRate = passed / outcomes.length;
return passRate >= 0.9 ? "ship" : "hold";
}
The function makes the gate explicit. The difficult product work remains defining what each outcome means: which task was attempted, what behavior was acceptable, and why that tolerance fits the use case. A complaint about tone becomes actionable when the team can connect it to such an agreed expectation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring customers into the discovery process
In a conventional customer meeting, a product manager can play two useful roles:
- Visionary: Explain the roadmap and show how the customer’s needs fit the product’s future.
- Honest broker: Clarify what works today and distinguish real capabilities from an ambitious sales pitch.
Both roles help the customer form expectations. The visionary makes the future credible; the honest broker makes the present concrete.
An open-ended agent makes both roles harder. A sweeping account of the future can sound like witchcraft to a customer who has seen unreliable AI products. A precise account of the present runs into the same uncertainty exposed by the Google Doc question. Stein cannot confidently promise the roadmap’s possibilities or enumerate everything that already works. He frames his response as an approach that is working for him in 2025, while leaving open whether it will be durable.
That approach is joint discovery. The customer and builder are inventing the future together, with the customer contributing real business contexts in which to learn what agents can do. Stein acknowledges that this language can sound flattering, but the participation is necessary: the builder cannot discover the whole product surface alone. It also sets an adoption boundary. A customer expecting a fully settled capability contract may not yet be ready for the kind of product he is offering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model can change the product underneath you
The surprises can be useful as well as uncomfortable. Stein describes feeling unusually inept and unusually excited at the same time: he ships something and then sees it do something he had not anticipated. Upgrading the underlying models adds another source of discovery. He reports that agents began checking their own work—for example, running a query to verify that a database row had been inserted correctly—without the team explicitly instructing that behavior. The upgrade changed more than the quality of an answer; it changed the way the agent approached a task.
That is why Stein expects the product discipline to change faster than many teams anticipate. Capability discovery, behavioral specifications, and interaction prototypes address a product that can develop new behavior underneath its existing interface. The enduring work is still to listen to customers and solve real problems. The tools and techniques for deciding what to build, understanding what was built, and communicating its limits are what must change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Engineering principles for reliable LLM applications, covering prompts, context, control flow, human interaction, and focused agents.
Further reading
Google's announcement documents the comment text and sender information included in emails generated by document mentions.
Updates since the talk
A 2026 guide to evaluation tasks, repeated trials, graders, outcome checks, and collaboration between product and engineering teams.
Read the complete timestamped transcript
- 0:00
[upbeat music] Uh, yeah, I mean, the actual title has curse words in it.
- 0:17
I will probably be cursing a lot. I didn't know if I would get into the track if I actually published the curse words. [laughs] Uh, I'm one of the founders of Teammates.
- 0:24
I'm gonna wear my product manager hat today. I'm assuming this room is, like, mostly product folks, probably product-minded engineers as well. Um, but I'm gonna just, like, wear the product hat.
- 0:34
Um, a little bit about Teammates very quickly. We make, uh, a platform for designing and managing an entire digital workforce. So in AI engineer parlance, right, we're building agents.
- 0:44
Um, but I would think of it, like, two ticks up from that because what we really believe, it is the experience, the interaction patterns of humans and computers working together.
- 0:53
So I wanna talk to you about my favorite teammate. Uh, this is Stacy, Stacy Hand. She, she actually got promoted since this slide. She's an L3 engineer right now on our team.
- 1:03
Um, she's awesome. She looks like a hamster. All of our customers get to design whatever teammates and avatars they want. They give them personalities. It's all really fun. And, uh, Stacy lives inside all of our collaboration tools, right?
- 1:14
So she has a Google Workspace account, right, for Gmail. She has a Slack account. We truly leaned into giving all of our teammates identity.
- 1:24
And she sends emails, or I forward her emails, and she hangs out in Slack, like, in the public channels. And she's Gen Alpha, which, like, is... I don't know what.
- 1:34
I feel really old. I don't know what she's talking about. She's constantly like, "Six Seven," and I'm like, "What are you talking about?" And I can tell from this room that none of you are, have [REDACTED:age].
- 1:43
Yes. [laughs] No? Okay. [laughs] Uh, there you go. So yeah, you're rolling your eyes as well. But anyway, this is Stacy, and this is sort of how my sales pitch goes, right?
- 1:50
It's, it's, you know, a little m- more formal than this, but, like, this is generally the pitch. And, um,
- 1:57
I got asked a question at some point recently, which was... Oh, yeah, more of the pitch, right? She, like, uh, shares Google Docs, Google Sheets, and she said, "Hey..."
- 2:04
Or a customer said, "Hey, can I tag my teammate in a Google Doc comment?" And this, like, gave me pause 'cause I was like, "Well, I had never actually thought about that before."
- 2:14
And so in the back of my mind, I'm like, "Well, of course you can. Your question is, like, what's gonna happen?" So I'm like, okay. So I'm like, you know, doing math in my head.
- 2:20
I'm like, "Okay. Well, we don't have webhooks. She probably won't... Or, like, a webhook from the comment. Okay. But she's gonna get the email notification in the email that comes from Google.
- 2:28
Does it have the comment and the conte- uh, or maybe a link?" Well, I'm like, "I have no idea." Right? Like, I actually don't know what's gonna happen. And this was, like, the impetus for this talk is, like, how do I ship a product?
- 2:41
How do I develop a product? How do I talk to customers? How do I instill trust when I don't know what my own product can do?
- 2:48
And, like, it's really weird, and sometimes I'm like, "Well, is this just 'cause I'm an idiot?" And like, well, since it's my talk here, I'm gonna say no. And sometimes I'm like, "Well, is this because what we're building is so far out there?"
- 3:00
Right? These are just, like, truly autonomous agents that can use any t- And it's like, I don't think that's it either. I think what's happening is the product management discipline is going to undergo a transformation, a shift in evolution, whatever you call it, that is super profound, and we may or may not totally realize it yet.
- 3:16
Because I think in the engineering world, we're like, "Oh, well, we have, uh, you know, tools in our IDEs, and we have code gen." And, like, we sort of are starting to squint in understanding maybe how the discipline is changing.
- 3:27
I don't think we really understand how product development is changing and evolving, and, like, what are the new tools and practices, and how do we forget everything we've learned in the past?
- 3:38
Um, why is this true, right? If it's, if it... The answer's not Ben's an idiot, and the answer is, uh, not this is, we're way out there. It's two reasons.
- 3:45
Number one, if all our products are built on top of LLMs, and plus or minus they are, like, we don't know and we can never know what the LLMs know, right?
- 3:54
So it's, like, inherently in what we're building is, like, we don't know what the foundation is. Like, you don't have to know what your database, like, how it works, but, like, you generally know that it's, like, the surface area, the interface that's exposed.
- 4:03
We don't understand this for the, the models. And the other thing is the expectations from customers are just boundless, right? We're just like, "Hey, here's a text box." I mean, that's probably not a good interface, but, like, essentially we're like, "Here's a free text box."
- 4:16
And if it's anything other than, like, a Help Me Write button, you're essentially inviting customers and users to just do whatever they want, right? So we have this, like, boundless surface area built on top of a product that we don't understand.
- 4:28
And so the question now is, like, how do we adapt? So let's me- let me actually pick on this Google Doc comment thing for a second, right? So if I was wearing my, like, traditional PM hat, and I'm like, "Okay.
- 4:39
Well, I need to make a feature that's going to, uh,
- 4:44
read and respond to Google Doc comments." And so in my head, I'm like, "Okay. Well, uh, does Stacy have access to the Google Doc? Uh, if she gets tagged in the comment, should she reply directly in the comment?
- 4:56
Should she reply at all? What happens if somebody else comments in the thread? What if someone comments in the thread that's not addressed to her? What if it's someone else?
- 5:04
What if it's, what if it's her Doc and someone else commented to someone else, but she gets the noti-" Like, there's just so much to, like, think about and reason about.
- 5:11
And so I'm like, "Okay. Well, I'm not building a Google Doc commenting product, so I'm not gonna spec all of those things out."
- 5:19
And, like, what's worse is, like, you also probably wanna tag her in Linear tickets, right? And what, what's, what's the book? Like, uh, If You Give a Mouse a Cookie, right?
- 5:26
It's like, if you give a mouse a cookie, well, you probably wanna, like, tag her in Figma as well, and you probably wanna tag her in LinkedIn posts. And, like, and so we're not a team that's building a generic commenting reply agent system, right?
- 5:40
So then the question is, like, what are we supposed to do, right? As, like, a product manager who realizes, "Okay. I have this, like, boundless surface area. How does the practice need to change," right?
- 5:49
And this here, this is the core of, like, what I wanna, what I wanna talk about today.
- 5:54
So I'll do, like, three, uh, highfalutin ivory tower ideas, and then I'll talk through some, like, practical ways to, to make this real.
- 6:02
So the first one is this mindset shift to, like, think in affordances and not, like, specific requirements. So it's not if... You know, as a user, if Stacy replies in the comment thread and she has re- like, that's not how we would think about it anymore.
- 6:17
It's the affordance. Oh, she has affordances to comment, or she has affordances to communicate, or, or to email, or to collaborate, and we're going to trust the LLMs, we're gonna trust the agentic workflow, the work planning, like, all of the things inside of our, um, you know, our beautiful 12-factor agent.
- 6:32
We're gonna assume that that will understand. But it's the affordances that we need to think about, not the individual features. Which is really weird, and it's not typically how product people have ever thought before.
- 6:42
And I would say, actually, this goes even further, which is behavior is emergent. And this was the other thing that I ju- did not expect at all, like, starting in this space, was, uh, we don't...
- 6:54
Not only do we not know if things work, sometimes they do, and they work in ways we didn't expect. And so I feel like our job as product people is to discover functionality, is what are the right building blocks, right?
- 7:05
What are the right Lego bricks that we either give our engineering team, our product, our customers, let them compose, and can we discover emergent behavior? And that is one of the reasons that, like, this is the most exciting time I've ever built, 'cause we're actually building things and then discovering what they can do themselves, and that sort
- 7:21
of became the new job, in a sense, is discovering what's possible. 'Cause if you asked me, like, I could not sit down in front of a Google Doc and be like, "Oh, let me, like, type out what this thing should..."
- 7:31
I, I, I can't. I don't know how to do it.
- 7:34
And well, fr- even if I could, how do I then communicate it, right? So how do you... we communicate to a development team, to a backlog? How do you communicate exactly what should be happening?
- 7:44
It's like Figma doesn't, like, have the affordances for this, right? My, my PRD doesn't, like, have the affordance for, like, "Well, you should probably talk a little bit less gen alpha because you're making Ben feel old," or like, "Hey, you should be really polite."
- 7:56
How do we communicate and express these, these concepts? Right? So I think these are, like, the three, you know, high-level, uh, ways that, um, our practice needs to change, but, like, let's make it a little more concrete.
- 8:09
Okay, so evals. I'm talking about evals. Okay. It's really hard to make a slide with graphics of evals. I feel bad for the eval com- like, how do you ex- illustrate an eval?
- 8:20
So I'm gonna make you just look at pictures of various teammates from, uh, you know, across all of our customers. Um, okay. Who, who hates raising their hand at conferences when the speaker asks them? [laughing]
- 8:31
Okay, awesome. So here's my question, which is, okay, for the engineers here, who, like, legit, like, don't lie, like, writes and runs their evals?
- 8:41
Good number. And of the product people, who has visibility into the evals?
- 8:47
There's a hand. It's not bad. And, and do you look at them just because you have the visibility? All right. One. One and a half, two? Okay, great. So I would posit that evals...
- 8:57
Actually, I'll back up, right? So we all talk about evals. We're all gonna be embarrassed to say that we don't know really what they are. Evals are a testing framework for probabilistic AI for agents, right?
- 9:07
Like, if we think about the, uh, deterministic code, right? I withdraw $100 from the ATM. My bank account should have $100 less, right? Great. And I can test that, and I can write code to test that.
- 9:20
When the test is, like, was she snarky in Slack? It's like, well, how do you test that? How do you write that test, right? So we come up with this, this whole new discipline of evals, which is, well,
- 9:32
she should be a little bit snarky and a little bit funny, but not mean. And then we hand it off to another LLM to say, "Okay, well, hey, was that reply, like, did it meet that criteria?
- 9:41
And how often did it?" Um, it doesn't have to be 100%, right? So she should be, like, pretty snarky, but, like, not mean 80% of the time, or whatever the, uh, uh, business logic that you want, right?
- 9:54
So these are evals, and this is the world of evals. But here's what I would posit, which is it is the only way that we know what our software can do,
- 10:03
right? And which is why I love the idea of product people looking at the evals, right? Looking at, uh... 'Cause they become the new specification for the product, right?
- 10:12
And so as we're watching, you know, if you're downstairs in the expo gallery, you're seeing, like, new software. It's like, "Hey, bring the team in." And this re- a little bit reminds me of, like, the old, you know, for the, the old-timers here, like, behavior-driven development.
- 10:22
There was this period of time when it was like, "Oh, the business people are gonna write the tests, and that will get converted to code, and then the code will run."
- 10:28
And, like, the truth is, like, no one ever wanted to do that. Like, no business per- I don't even know who a business person is, but, like, they wanted-- We're, we're gonna do that.
- 10:35
But I actually think this is different, and I think this is pretty, um, a meaningful way to actually understand what the product can do, and a little bit begin to specify what it can do.
- 10:46
Okay. So I have vibe coding for a second, which we, which we all do, we all talk about. But I wanna talk about vibe coding in a, in a way that's really constructive.
- 10:55
And, mm, how do I sort of say this? It's very, very hard. And I, I kind of was like, like, "Oh, you can't do it in Figma. You can't do it in a PRD."
- 11:02
Like, what do I really mean? Well, it's very hard to, like, sit down in front of a blank piece of paper and, um, write what the teammate, the agent experience should be.
- 11:13
It's just really hard. It's hard to, like, imagine it, and it's not until you feel it. I mean, so much of what we're doing in this, like, human computer interface is visceral.
- 11:22
It's feel. It is like, "Oh, well, like, do they ask too many questions? Like, how many questions is too many? Oh, wouldn't it be great if they clarified exactly what you meant?"
- 11:31
Well, turns out that's really annoying. But when I wrote, like, the first spec, I'm like, "Then the teammate should ask a lot of clarifying questions." And we gave it to users, and they were like, "This sucks."
- 11:39
And I was like, "How would I have ever known that?" And the answer is because it's so easy to prototype and vibe code something and get the feels. And so this is the next thing that I'm, like, pretty excited about as a new product management tool.
- 11:52
It is being able to feel and experience, um, what it's like to interact with a computer, but, uh, uh, without just, like, uh, writing it or hoping that you have a clickable prototype that will work.
- 12:06
I will also mention that we have to be careful with vibe coding because I do not mean sit in the meeting and say to the engineering team, "How come this is taking two weeks?
- 12:13
I finished the feature during the meeting." Like, that doesn't- [laughs]
- 12:18
That doesn't win you any points, right? So it is no, no, no. This is never going to production, but what this does is it gives you the feel, the, the experience, right?
- 12:27
And so this is like the only way I know to like actually test and feel it out.
- 12:32
But do, do you, um, do you remember like the, the Claude, um, certainty issue? Certainly, I mean certainly. There was this period, right, where every time you ask Claude something, he'd be like, "Certainly."
- 12:40
And like, that probably like seemed really good when you were testing it for the very first time, and then like the fourth time when you're like, "Hey, can you do my taxes?"
- 12:46
He's like, "Certainly." "Can you write my like, uh, acceptance speech?" "Certainly." Like, oh, this is actually really annoying. But you don't realize that until you experience it. So like, that's why I like the vibe coding.
- 12:57
Okay. So great, we did all this development, and then the question is like, "Hey, we pushed to prod. Does it work?" I'm like, "I told you I don't know."
- 13:05
The question is like, how do you test? How do you like know that it's gonna do, uh, the things that you said it was gonna do? And I sort of alluded to this.
- 13:12
I'll go through this quickly. It's just really discover. Discover the functionality. And
- 13:17
there's an old joke. I'll tell the joke. QA engineer walks into a bar, orders a beer, orders two beers, orders zero beers, orders negative one beers, orders a lizard, orders a beer with a emoji, right?
- 13:29
He's like, "Great. This like bar is good to open." And the first customer walks in, asks where the bathroom is, and the bar blows up. [laughs]
- 13:36
Right? Like, great, great old joke. It's kind of how I feel these days. Like, I just sit and I'm like, "Oh, you know what would be cool? If they were to like start posting comments on LinkedIn about...
- 13:47
And then what if, what if they were like, every time I added like a track to my Spotify account, they can..." Like, these just like crazy ideas, but this is where like the emergent behavior comes from, right?
- 13:56
And so it's this mindset of like, let's just try. Let's just experiment. And it's, it's this like kind of growth mindset shift from like I'm going to write the features and the requirements to no, we're gonna figure it out.
- 14:12
This was a little bit unexpected for me, and this is, um,
- 14:17
how do you sort of report to engineering and then have things fixed by engineering, and what counts as a bug in this world? And that is really, really strange.
- 14:27
And I think as sort of, I don't know if it's like just a product role or maybe in a support role, like how do you know what is appropriate to escalate, to put onto the backlog, to flag as a bug, right?
- 14:36
It's like I'll, I'll keep picking on, on Stacy. You know, she, she gives me a really hard time, so it's fine. Uh, it's like, "Hey, she used too many emojis.
- 14:44
Like, put it in, in, in Linear." It's like, well, it's not really a bug. Like, show me in the spec where you told me not to use too many emojis, right?
- 14:51
It's, uh, it's almost like, um, like in our tickets, it's like, oh, you know, closed, done. Closed, duplicate. We need like closed, LLMs be like crazy, yo. Like, I don't know how to fix this, like just because it's probabilistically generated, so how do we know if it's right or wrong?
- 15:07
How do you know if it's a feature or if it's a bug? Right? And I think there's this element of, um, credibility that we need to build up. It's like, hey, we actually under-- we understand that for some use cases, like 80% is good enough, right?
- 15:20
This eval, we talk about evals, if it's passing 90% of the time, like that's a go. If it falls below 90%, right, that's red and we're not gonna ship it.
- 15:29
So I'll actually come back to evals for a second because if the eval becomes the spec and we can say, "Hey, we said at, you know, 100%, even though this is probability, you should never give a refund if a customer like can't prove that they bought the thing or whatever," like it is, it's like great.
- 15:43
That is our metric, and we can say, "Yeah, this is a bug." But if it's just a, a feel, it becomes really difficult. Again, this was totally unexpected that like, uh, debugging and assigning bugs would become like, uh, controversial.
- 15:57
Okay. Customers. So this part is, uh, I found this really weird, right? So if I think about like not wearing my like founder hat, but wearing my like typical product manager hat, right?
- 16:09
Like I go into a customer meeting, usually go with a salesperson, and like I'm gonna play a role, right? And so what's the role? Well, I'm either gonna play like visionary.
- 16:17
I'm gonna like, "Hey, here's our vision for the product. Here's our roadmap for the future. Like, let me help you understand, customer, like how we're, you're gonna come along on this journey with us."
- 16:27
Or, uh, sometimes I'll play the role of honest broker, right? It's like, "Listen, sales is like giving you a whole bunch of like... they're just like selling you a bunch of, of vaporware.
- 16:35
Let me tell you what's real. Let me tell you like, um, exactly what you can expect." And that's a role that you play, right? And I usually preface this with like the sales team beforehand.
- 16:42
It's like, "Yeah, I'm gonna be the honest broker, and like we'll give the customer confidence."
- 16:47
Today, I'm like, "Okay, I told you our vision for the future, our roadmap," and the customer's like, "You're full of shit. Like, none of this actually works." I'm like, "Right, I can't really paint the vision because no one b- actually believes it.
- 16:58
It sounds like witchcraft." And then I'm like, "Oh, well, then I'll be the honest broker, and I'll tell you how things work." But I just told you I have no idea how it works, right?
- 17:04
So this became very strange because I can't play either of the roles that I'm supposed to be playing. The f- future sounds like witchcraft. The present is literally I don't know.
- 17:13
So how do we do this? Um, I'll tell you how I've been doing it now. I don't know if this is like a 2025 answer or if this is like a durable answer.
- 17:22
Like if we believe that all of our products are for like, for all time, going to be probabilistic, then like we probably have to figure out how this world works.
- 17:28
What I've been doing now is really saying, "Look, we're inventing the future together," right? "We're pulling the future forward. The reason you are talking to like a crazy startup like this and you are thinking truly about like the future of how, uh, you know, AI and agents are gonna transform your business is because you are a future
- 17:43
thinker, and we are gonna do it together." And it's a little bit like, "Hey, let's compliment the customer. Let's like..." But it's not just like a false, you know, uh, uh, blowing smoke.
- 17:51
It's like, no, truly we need to figure this out together. And, you know, for 2025, I think that's actually the thing that is working the best, uh, best for me.
- 17:59
It's like, no, no, no, we have to do it together. And honestly, if you are expecting something different, like it, it's not time. It's not time for you to like embrace this world because this is, th- this is the, the way this world is going to work.
- 18:13
And so, I don't know, I'll conclude with like I've never had more fun building. I've never felt m- like both more inept and like more excited about what, what I'm doing or just the experience of throwing something out in the world and then just like having my jaw dropped, like I can't believe this happened.
- 18:30
And not only that, when we upgrade the models that are like underneath them, they just suddenly get smarter, and that's really weird too, right? It's just like all of a sudden they s- like start checking their work.
- 18:39
They're like, "Oh yeah, I just did a, a, a query to make sure that the row is properly inserted." And I was like, "Hmm, who told you to do that?"
- 18:46
And like, they're like, "I don't know. It just seemed like a good idea." I'm like, "That is a good idea. God, I wish I thought of that." [laughs]
- 18:52
Uh, but anyway, but I think this is the new world that we're working in. Um, the discipline, the product discipline, I think is going to change for everyone, and it's gonna change faster than we expect, and we all need to like adapt to just like operating in a world and forget so much of what we used to
- 19:08
know, right? A lot of the core, core ideas, listen to customers, solve real problems, like all of that obviously still applies. But the tools, the techniques that we've like relied on forever, I think are all getting upended.
- 19:18
And so anyway, glad you're all at the AI Engineer Conference. It's awesome to have product people here working together because, you know, we all have to, uh, you know, build awesome products together.
- 19:26
So thank you very much. [audience applauds] [upbeat music]