AI Engineer World's Fair 2025
Everything is ugly, so go build something that isn't
Read the talk
Everything is ugly, so go build something that isn't
AI makes more things possible, but a useful product still needs a clear job, a trustworthy first experience, and the restraint to leave capabilities out.
From a talk by Raiza Martin
What does product do when everyone can build?
What does product work mean when product managers, engineers, and designers can increasingly do one another’s jobs? Raiza Martin opens with a show of hands across those disciplines, then describes product as a multilayer cake whose first layer is how builders understand themselves. A PM who could write SQL once had a recognizable technical distinction. With ChatGPT able to generate the query, the more consequential skill becomes knowing which question to ask. That does not make every person an expert; it makes expertise easier to access or simulate, and makes a job title less sufficient as an account of someone’s contribution.
The next layer is the team. Documents and slides now arrive with invisible contributors: ChatGPT, Claude, or Gemini may have helped produce them. Even after reading something herself, Martin gives it to ChatGPT, compares its takeaways with hers, and asks what she missed. Her illustrative five-person team therefore has five additional ChatGPT collaborators. The organizational consequences are still uncertain, but the practical question has already changed: what will people do with this additional capacity?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The gap between intent and interaction
The third layer is the product itself: an idea drawn out of a person and translated into technology. Martin sees existing products as occupying an awkward transitional period. Interfaces designed before AI reflect the interactions their builders could imagine and implement at the time. Her deliberately ordinary examples are pressing a key to make a J appear and operating a microwave. Once richer interactions become possible, even these familiar mechanisms start to look constrained. Her description of products as the ugliest they will ever be expresses an expectation that the available interaction vocabulary is about to expand.
The user completes the layers. ChatGPT, Cursor, and Claude teach people that they can state an intent and receive something useful. Returning to products that require much more translation between intent and action then feels frustrating. Martin calls that gap consumer unrest: expectations rise across the entire product landscape, not only inside AI applications. It creates an opening to rebuild familiar tools around newly possible interactions.
That opening also has a personal cost. Changing roles and teams bring uncertainty about future work for PMs, engineers, and designers. Martin’s response is to define product people by their responsibility rather than their title: anyone supplying the force behind a product can find a small opportunity and expand it, like a popcorn kernel. An organization can share that responsibility across its members. The useful question becomes how to turn that opportunity into a product worth using; the answer here is a set of building principles, not an implementation recipe.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Clarity supplies the force to build
Making a product occupy a meaningful place in someone’s life takes sustained force. Martin initially wanted to describe the experience as violent; her team persuaded her to use forceful instead. The distinction she cares about is between making something as a hobby and pushing it into existence as a product people depend on. That second task requires conviction strong enough to survive resistance.
Personal clarity precedes organizational alignment. Before a team can agree on what it is building, someone needs a clear account of what matters and why. Martin treats technology choices, hiring, and organizational effort as expressions of energy supplied by people. Knowing the what and the why provides the energy to keep working with teammates, stakeholders, and users. She divides that clarity into vision, purpose, and taste.
The early NotebookLM, introduced as Project Tailwind at Google I/O in 2023, gives that principle a concrete origin. Martin recalls substantial skepticism about the idea, enough to make her question it herself. But she had returned to college full time after dropping out, while also working full time at Google building NotebookLM. Her need was immediate: put documents and slides in one place, then converse with them to accomplish useful work. This was the intended workflow, not the original format-support list: the initial NotebookLM rollout supported Google Docs, with Slides support arriving later.
She wanted to build that experience from the ground up, rather than attach it to another product. When users, stakeholders, or teammates did not understand its value, her own circumstances supplied a reason to continue. Personal clarity did not remove disagreement; it made the work of carrying an unfamiliar idea through disagreement possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with the job, not the pixels
Clarity becomes actionable when it names an outcome. Taste, in this account, concerns more than aesthetics: what is the single outcome the product must deliver reliably for every user? That purpose provides a criterion for deciding whether a feature adds value or becomes baggage. Without it, an impressive demo or a compelling social video can masquerade as a product. Martin calls this AI demo disease.
Purpose makes it possible to reject novelty when novelty dilutes the core job. Martin’s point is that the AI label is not the benefit users came for. They have an intent; the product should fulfill it in a way that feels natural and inevitable. That standard is harder to meet than making a model capability visible in an interface.
For NotebookLM, her target was to put 50 items into the tool and do useful work across them. Summarization was an entry point: if the product could make a body of material understandable, it could open the way to other operations across that material. Both the interface and the generation of a useful summary were difficult. Once implemented, however, the automatic summary served two purposes at once. It gave people useful information immediately, and it taught them what this unfamiliar collection-of-sources interaction was for. The 50 files describe the target workflow here, not a summary-quality result.
The feature looked basic from the outside. Its significance was that it acted as a tutorial through useful output, rather than requiring users to understand the product before receiving value. Martin connects that decision to sustained attention to the outcome: the apparently small feature emerged from repeatedly asking what the user needed to accomplish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A product promise needs reliable basics
A product makes a promise about what it can do, and a user spends time and attention testing that promise. Martin describes trust as oxygen, with many products starting on something like negative credit: people may not want to try them at all, and those who do have little patience for anything unrelated to their task. One way to earn trust is to expose the model’s limits instead of concealing them. Users need a coherent experience at the point where the model stops being useful.
A product also exposes its builders’ workflow and assumptions. When it fails, the response should account for the person experiencing that failure. Designing and instrumenting only the best case leaves the worst case without a considered human experience. Get deterministic behavior right before adding delightful probabilistic behavior. An AI application still has to function as a good application; the model does not excuse unreliable basics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first summary is a trust test
In early NotebookLM, users uploaded sources and asked for summaries. Their requests often included a strict scope: summarize this document and only this document, or summarize only this concept. Smaller context windows made these requests particularly difficult. The older interface shown here has source cards, an empty Notes area, and a summary request being entered; it shows the starting workflow, not a generated result or an error.
Martin reports that roughly 90% of users’ first queries in early NotebookLM were summarization queries. She does not specify the cohort, observation period, or measurement method. The important behavioral interpretation is that summarization was a test: a user had heard about the tool, visited the website, uploaded material, and then asked it to perform an apparently straightforward task on that material.
If the summary failed, all of that effort ended in a broken promise. Martin describes users leaving and not returning, which made summarization a prolonged priority for her. The engineering importance of a feature therefore cannot be inferred from how exciting it looks in a demo. A basic first action can carry the burden of proving that the entire product deserves another interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Earn the right to surprise
The user experiences this progression quickly. Martin describes the initial opportunity to earn trust as a matter of seconds, perhaps a minute, emphasizing how little patience a new product receives. Once it has earned that trust, it has room to delight. NotebookLM’s podcast-like Audio Overviews illustrate the change: users upload documents and receive a conversation whose phrasing and exchanges can be unexpected, funny, or goofy. The surprise introduces playfulness into working with familiar material.
Delight occupies a boundary between technical capability and user expectation. A trusted product can move one step beyond what feels familiar. Move too far and unfamiliarity becomes discomfort; Martin uses people’s unease around robots as an analogy. That makes the design task more specific than maximizing novelty: introduce something surprising that the user is ready to accept.
Delight should come through agency, not trickery. In NotebookLM, the user knows which document they supplied and chooses to transform it. The result can be unpredictable while the activity remains self-directed. That is different from pressing an unexplained random button: the user has contributed something meaningful and can recognize their role in the outcome. Martin describes the balance as feeling “equal parts me and the machine.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the product in the user's circumstances
Too many features make it harder to create that experience. Martin draws a sharp distinction between shipping model capabilities and shipping new outcomes. A team needs answers to four connected questions:
- Outcome: What is the product optimizing for?
- Path: How is the user supposed to reach it?
- Obstacles: What prevents the user from getting there?
- Discovery: How does the product show the user what is possible?
These questions tie capability selection to an actual experience rather than a feature inventory.
Answering them requires sustained use. Martin distinguishes testing a product from living with it: the latter exposes whether its capabilities matter inside a real routine. If the builder is not the intended user, the work includes inhabiting that user’s circumstances. This is how a team develops judgment about the boundary between what the technology can do and what people are prepared to adopt.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide what the product should do
Delight demonstrates what a product can do. Judgment determines what it should do. Martin admits repeatedly making the kitchen-sink mistake herself: collecting model capabilities, adding some visual personality, and mistaking the collection for a point of view. Research previews can usefully reveal new capabilities. But a user looking at another broad capability bundle may reasonably prefer to use ChatGPT. Adding everything to see what sticks does not establish a distinct reason for the product to exist.
Her decision criterion is whether the product respects the user’s time, data, and agency. Restraint becomes an innovation multiplier because it concentrates effort on an outcome that can become excellent. When someone asks how such a simple product can compete, focus is the answer: do one thing sufficiently well that users begin to experience the difference. Working deeply on that one thing also improves the team’s intuition about where useful surprise becomes possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Huxe and the kitchen-sink trap
Consider trying a new product that does 30 things without making clear when you would use it. The number of capabilities does not resolve the uncertainty; it can create it. Martin traces that lack of focus back through the layers: unclear personal intent leads to unclear product direction, which makes it difficult to choose useful model capabilities. She encountered this in her own work on Huxe, whose early prototype combined chat, podcasts, news, image generation, and video generation. She initially found it beautiful and exciting—her dream product.
Daily use changed her assessment. She found herself using only one thing. Martin reports that several hundred people tried Huxe and likewise used only one thing in the product. Her account does not identify that feature or specify the observation period. The lesson is the gap between the breadth that excited the builder and the narrow role the product occupied in use.
Her comparison is Spotify: it may offer many capabilities, but her reason to open it is to play music. Huxe even had deliberately goofy chat to distinguish its tone from ChatGPT, yet that personality did not resolve the lack of focus. The three-screen montage shows content cards, messaging, and audio playback, including a player titled Fighting Pests With Sterile Flies. Looking back at the broad interface, Martin recognized her own version of AI demo disease: something that looked impressive without giving people a sufficiently clear reason to love it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the product's promise intact
The principles form a dependency chain. Clarity supplies the energy for difficult work. Purpose directs that energy toward a specific outcome. Delivering on that purpose earns trust and user belief; belief creates permission to introduce delight, which reveals further potential. The kitchen-sink check keeps the chain honest by asking whether each capability still serves those commitments. Building something that is not ugly, in this sense, means making the product’s intent, behavior, and place in the user’s life fit together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Raiza Martin and Steven Johnson introduce NotebookLM, formerly Project Tailwind, and explain its original source-grounded summaries and questions.
The launch explanation of podcast-like discussions generated from a user's sources, including the feature's original limitations.
Further reading
A historical account of Slides support, citations and automatic source transformations, coauthored by Martin.
Practical notebook organization and source-exploration advice, documenting the 50-source capacity in October 2024.
Updates since the talk
- Huxe's service wind-downArticle
Huxe's official announcement scheduling the end of its service for May 28, 2026.
Read the complete timestamped transcript
- 0:00
[upbeat music] Really quick show of hands, how many folks actually work in product?
- 0:18
Wow, okay. Engineering? UX? Okay, I feel like there's definitely some overlap there, right? But that's, that's exactly what we're seeing happen right now. Sorry, I have to keep walking over here 'cause I'm so short.
- 0:30
Like, if I stand here, I can't see you. Um, but what, what's crazy is I feel like we're all doing all the jobs now, right? Like, that's the crazy thing about AI.
- 0:41
That's the crazy thing about right now. And, uh, I wonder how much longer it's gonna be relevant for to ask, you know, "What do you do? What do you do at your company?"
- 0:51
Because chances are you're probably doing everything, and you can, right, with, with some ease. And so I think a lot of questions, people ask me this all the time, students in particular ask me this all the time, uh, so what does product do, like, in this new world?
- 1:04
And so the way that I think about product is kind of like a multi-layer cake, and I think that it starts with how we think about ourselves. And I know this is, like, kind of a weird topic.
- 1:14
It's, like, not super technical, but I think it's probably one of the most important, which is, uh, I remember when it was considered technical to be, uh, the type of PM that could write your own SQL queries, right?
- 1:28
But, y- you know, if I said that to you now, like, it's kind of silly because everybody knows how to write their own SQL queries because ChatGPT does, right?
- 1:35
Like, all you have to do is to know what question you're asking, and then you can do it, right? But it's not just, like, these little tasks that you can do.
- 1:43
It's also, like, the entire roles themselves that are changing because it's easier than ever to access expertise or simulate it, right? So in a world where the jobs are blending together, I feel like it's even more important to understand, you know, where are you coming from?
- 2:02
What is the value that you bring, you know, coming from yourself? And then there's this other thing that's really interesting that's happening, which is teams are becoming drastically reconfigured.
- 2:12
And even at, like, a super basic level, like, if you think about it, every team now has a bunch of, like, invisible participants, right? Like, every doc, every slide, every little bit of thing that is passed around or created, there's, like, a ChatGPT or a Claude or a Gemini behind that thing, right?
- 2:31
Like, most of the time, even when I read something, like, with my own raw eyeballs, I'll still give it to ChatGPT and be like, "What did I miss?" Right?
- 2:39
Like, "Here's my takeaways. What's yours?" And that's crazy because teams are now fully augmented, right? We've got, like, five people on a team, and we've got five ChatGPTs, and that's crazy.
- 2:50
We don't really know what that means, right? But we've got, we've got superpowers now. How are we gonna use it?
- 2:55
And then there's this layer which I think is, like, super interesting, which is this is where the title slide comes from, where, um, I think products come from, from people, like, deep within people, right?
- 3:06
You pull an idea out of yourself, and you translate it into technology, and that's what a product is. But when you look at every product that we are using, you can tell we are in the clunky, awkward years, right?
- 3:21
You can tell everything is about to change. And literally everything we are using is the ugliest that it will ever be. Because everything we use was created in, like, the pre-AI era, where, you know, we had to imagine, "Well, how do I make this thing work?
- 3:36
Like, if I press this button, like, it makes a J on my screen." Like, now it feels, like, kind of strange. Like, when you think about the richness of, like, the interactions that are possible, it now feels kind of strange to use, like, an everyday thing like a microwave, or, or m- maybe it's just me.
- 3:51
Maybe I'm the only one who wants, like, an AI microwave, right? Or whatever, whatever that thing does. Um, but I think what I'm trying to say is, like, we're starting to assemble the shapes of what we think AI is really capable of and what it can deliver, and I think that the best products out there haven't been
- 4:07
discovered yet. How do we discover it? Well, I think the answer lies in the final layer and sort of typically, right, what a product manager would say. I think it's in the user layer.
- 4:18
And, uh, what I mean by this is I don't know if you all have noticed it, but there is this kind of, like, consumer unrest, right? Where as we use products like ChatGPT, Cursor, Claude, right, you start to see, whoa, it's super easy to use this shit.
- 4:38
It's just, like, I have to say what I want, and something magical comes back, and now I have to go use the rest of this dumb th- like, dumb products out there that don't do that.
- 4:47
And so you have a little bit of this chasm now where you have these super powerful, really intuitive, really smart products, and you have the rest of the world, which is just, like, janky.
- 4:57
And I think that what we're going to see is that there's going to be a phase where everything gets rebuilt, right? So
- 5:08
how should we think about rebuilding? Well, first of all, I think we should not undersell the fact that there is a lot of chaos. Like, all the time, uh, I think I try to emphasize this to people, even though things are, like, really cool and pretty magical, it's also pretty fucking hard because each of these layers is
- 5:29
being effectively rewritten, and that is not without cost, right? Like, even the first question I ask, which is like, "Hey, what do you do for a living?" Like, it's kind of weird.
- 5:37
Like, it's really uncomfortable to be like, "I don't know." Like, are they still gonna be hiring product managers next year? Not sure. Engineers? Don't know. Designers? Maybe, right? But for the most part, right, that, that unrest lives inside of us at each of these layers, and that's crazy.
- 5:52
And so I started out this spiel asking, you know, what is the role of product? And ultimately,
- 5:58
I think that complementary to chaos is always opportunity. Right? And our job as product people, I d- I don't say product managers, I think just like whoever you are, right?
- 6:09
Like, if you embody sort of like the force behind a product, this applies to you. Your job is to find the nugget of opportunity out there and explode it like a popcorn kernel, right?
- 6:22
Like, that is your singular job. And I think it's actually kind of exciting if you work at an organization where every person embraces this mission.
- 6:31
Okay, so my talk is largely about this opportunity and how I think you should go about exploding it. Um, I think there are a lot of talks that you could attend that tell you sort of practically how to, you know, technically build products.
- 6:42
But like I, I really wanna talk through sort of the principles that I use to drive product building, right? So let's jump in. What does it take to build a great AI product?
- 6:54
So I think first I wanna acknowledge that for folks who have shipped things, uh, especially, you know, if you're a part of a team or a company, I think that building a product is a forceful experience.
- 7:08
Like, I tried to think about the right word to use here. I actually first was like, "I think it's a violent experience." [laughs] And my team was like, "I don't know if you can say that," right?
- 7:17
Like, "I'm pretty sure people don't know if like a tech job is violent." And I was like, "Okay, okay, okay. It's forceful," right? And what I mean when I say that is I think that you almost have to force a product into existence, and I don't mean sort of like the hobby stuff, right?
- 7:35
Like, I mean to truly build something that can meaningfully exist as a product that has a place in people's lives, like that takes a lot of force. And I think that to, to do that, to do that particularly well, you need a lot of personal clarity, right?
- 7:52
And it's like, it's like this thing that is inside of like an individual person, right? Sometimes we talk about clarity, we talk about like, oh, the team has to be clear on this, the org has to be clear on this.
- 8:03
It... No, right? Like, I don't think that's where it starts. I actually think it has to start with sort of a singular individual that like carries this clarity with them, right?
- 8:13
Because once you know the what of, of what you're building and the why of it, that's real energy. And I think when people talk about technology, we're always talking about technology, tech, the stack, et cetera, right?
- 8:26
Hiring. I think everything we are talking about is just a transformed energy that comes from people. And so ultimately, this personal clarity is what's going to give you the energy to push your team, to push your stakeholders, and to push your users because it's like, it's, it's really hard, and your primary role is just to cultivate that
- 8:47
relentlessly. And I think it's three things, right? It's the clarity of your vision, the clarity of your purpose, and the clarity of taste, which I'll talk more about in just a little bit.
- 8:57
I'll tell you a short story, which is, has anybody ever seen this or used this thing?
- 9:03
No? Yes? Okay. Well, this was the first version of NotebookLM. It was called Tailwind. We announced it at Google I/O in 2023, and I will never forget the road to get to this thing, right?
- 9:15
Like, I think the amount of people that told me it was stupid was actually like, like fascinatingly high. Uh, and it, it was really great that I was like, "Wow, I think it might actually be stupid," but it's like that force, right, that personal clarity that gave me the energy to keep driving forward with it.
- 9:35
And in reality, the reason why I had so much personal clarity, I don't think a lot of people know this, but I had dropped out of college, and I went back to school full time when I was working at Google.
- 9:45
And so I was full time in college, I was full time building NotebookLM, and I was like, "I don't know how else to explain this to you, but if I could just have a tool where I put a bunch of shit in it, right?
- 9:55
Like, just a bunch of docs, a bunch of slides, and I just chat with it and it does something for me, that seems really valuable. Like, I've never been able to do that before," right?
- 10:06
And I'm not just bolting it on. Like, I wanna build this thing from the ground up. And so every time somebody would tell me it was stupid, whether it was a user, a stakeholder, teammates even would be like, "I don't understand," right?
- 10:17
I would say, "No, I do though," right? Like, "I get it. I know why this thing is important." And so personal clarity will get you far, and it will get your team far.
- 10:28
Um, and so I highly recommend starting from this place of just like cultivating this energy.
- 10:35
Okay, so now you have clarity. Great. How do you turn that into a real thing? Well, I think that the first thing that I always tell people to do is you have to start with the job and not the pixels, right?
- 10:45
Because when we talk about taste, I think sometimes people feel that that is about an aesthetic thing, but I actually think it's about an outcome, right? I think it's about the question of what is the single outcome that your product has to deliver every time for every user flawlessly?
- 11:06
Like, that is purpose, right? Purpose is the North Star that tells you whether a feature is, uh, gold or if it's just baggage. And it's the antidote to, uh, a really common problem that I see all the time, which is AI demo disease, which is, "Hey, this thing is a cool demo.
- 11:24
I made it. It demos really well. I made a really cool Twitter video," or whatever. But these are not real products, right? If they're grounded in hype, if they're grounded in sort of like trying to ride the waves of chaos, you're not gonna get anywhere.
- 11:38
So purpose is what helps you say no to novelty when it's diluting your core job, right? Users literally do not give a shit if something is AI. I think people are actually kinda tired of the word.
- 11:49
I think that p- what people care about ultimately is when they have an intent and you deliver it to them in a way that feels inevitable, right? And I think we go back to that energy, right?
- 12:01
It takes energy to get there. It's very hard, and you need to be purpose obsessed in order to get there. And here's another example, um, that I want to give, which is I was singularly obsessed with this use case of, like, I want to put 50 things in the tool, and I want to be able to do
- 12:17
stuff with it. And summarization was a really big one because I figured if you could do this, you could certainly do a lot more things across data, right? And, uh, it was really hard to do this.
- 12:28
It was really hard for UX reasons. It was really hard for sort of the actual way that we were able to generate this in a smart way. But the trade-off is that once we built this auto summary into NotebookLM, it made it so much easier for people to understand a really foreign concept at the time, right?
- 12:47
Which is, like, the concept that I would put 50 files in one place, and I would interact with it in this different way that we hadn't really done before.
- 12:55
It, it was like a little token feature that was both a tutorial and was useful at a glance. But it's like you could not have arrived at this type of idea, which looks really basic, just, like, from the outside of it, but you could not have arrived at it if you were not sort of obsessively trying to
- 13:12
drive at, like, the value that you're trying to deliver to somebody.
- 13:18
And so I think that when we think about the value and what we're trying to give to users, I think that in reality, the value of a product is a promise, right?
- 13:29
Like, ultimately, any product is a promise to a user, and you're making claims about what it can do, right? So then the user tries it. But trust is oxygen, and using a product is like a transaction between the user and the company, and without it, you're nothing, right?
- 13:47
You have, like, a pretty limited credit. Like, most products get a credit of, like, negative one because not everybody's gonna wanna try your shit, and then, like, the person that does, like, they don't have the patience for whatever things you put in there.
- 13:58
They only have patience for one thing. And so I think one of the best ways that you can actually build trust is to expose the edges, right? Show people where the model is dumb and make it seamless between the user and the product.
- 14:13
Like, don't paper over it because even though we have these really smart, incredible models, the way that we are building them is still, like, a very human type of thing.
- 14:24
So when you, when you give a product to a person, you are exposing your own process, your own thought process, your own workflow to them, and you're making a promise about how it works.
- 14:34
And so when it, when it fails, when it doesn't work, think about how you go about it in the most human way possible, right? Like, I, I see this all the time where people are trying to instrument for, like, the best use case but not the worst case.
- 14:46
And so you kinda have this, like, weird half-baked product and space where it's purely machine and very little human involved. [coughs]
- 14:55
I think on this note, I want to say something that sounds really dumb and basic, but I think that you have to nail the deterministic things before the delightful probabilistic bits.
- 15:07
Because at the end of the day, a good app is still just a really good app. Like, it's just an app, right? And it's like, doesn't matter what you jam in there, it's still an app.
- 15:20
Uh, this is another one of, like, sort of the older, uh, user interfaces we had in Notebook, but users routinely would do this thing where they would upload sources, and they would enter a query like, "Summarize this doc."
- 15:34
"Summarize this doc and this doc only." "Summarize only this concept." And this was very hard to do in the early days of NotebookLM, particularly with the smaller context windows.
- 15:45
And one of the things that we saw was the query type for summarization was like, uh, in terms of, like, the, um, the percent of, like, first queries, it was like 90%.
- 15:57
90% of, like, users, their first query was a summarization query. And it was this testing behavior, right? People were trying it out. They were like, "Oh, I heard about this thing.
- 16:07
Okay, saw the website. Cool. I'm gonna upload a thing." Like, think about all the steps the user went through to get there. User gets there, they upload something, they enter summarize, and it, it borks, and it didn't work.
- 16:21
So the user leaves. They leave forever. But, like, think about, like, the amount of time that, like, that person put into it because you made a promise to them that it was gonna do exactly that.
- 16:31
And so I think that one of the things I want to say about trust is it's not, it's not cheap, right? If you get a user to try your product, make it as good as fucking possible the first time because they're not gonna come back.
- 16:43
In fact, it was, like, so detrimental, the summarization use case, that it was, like, all I thought about, uh, for a very long, for a very long time.
- 16:53
Okay. So let's, let's say you manage to earn the trust of your users. You've got a bunch of them. They love it. And really, uh, to be honest with you, in, in real life, right, like this, this flow that we're describing, it's actually, like, a matter of seconds for a person.
- 17:08
This is, like, one minute, right, for you to deliver on this whole thing. Um, but you get to earn the next thing after you have trust, which is the potential to delight.
- 17:17
And I think with Notebook, this was, like, really cool, where I feel like, um, the delightfulness was that it was unexpected, and it was kind of funny. Where people would upload documents, and then they would make a podcast, [chuckles] and they were like, "I don't know what it's going to say."
- 17:32
Like, it could be kinda goofy. It could be pretty funny. And I think there's, like, an aspect in there that is, like, just very playful, right? Like, delightfulness is almost-- is very similar to playfulness, and it sort of lives, I think, in this interesting space between technical capabilities and user expectations.
- 17:52
Because it's like you kinda have to meet users where they are, but you have to push them just a little bit. You know? Where once you've built trust, you can push just one step past what's familiar and not spook anybody.
- 18:05
Because I, I think we, we've seen this too, right? Like, when things are, like, too weird, it's, like, spooky, and people are like, "I don't know. I, I'm not gonna look at it anymore."
- 18:11
I feel like people felt this way about robots for a long time. And so I think that in reality, kinda just going back to the trust piece, we get one chance to really make the machine feel like magic.
- 18:22
And so my, my tip here is actually just, like, a very, like, kind of a tactical one, which is I think you need to surface delightfulness through agency and not trickery, right?
- 18:33
Where it's like the user has to feel like they are a part of it, that they are steering, and it feels self-directed. And that was one of the big things about NotebookLM, which was like, it wasn't, like, a random button, right?
- 18:44
It was like I knew the specific document I had uploaded. I knew that it was going to make something magical for me. But it felt like equal parts me and the machine, right?
- 18:53
It wasn't just the machine. Like, it felt like there was, like, a healthy tension between there where the machine had a real opportunity to delight me.
- 19:02
Um, I'm not gonna show an example here in particular, but I wanna make a point about delightfulness, which is it is actually really hard to delight people if you're doing too much shit, right?
- 19:11
And I really think that you're either shipping model capabilities or actual new outcomes. There is no real in between here. And this is where sort of like the stack will, will come to kind of haunt you, where do you know the outcome that you are optimizing for?
- 19:26
Do you know how the user is supposed to get there? Do you know what is preventing users from getting there? You know, do you know how to show them what is possible with the system?
- 19:35
And I think these are sort of like kind of deep, gnarly questions, and it's just, like, something that you will only get to by using the product a lot yourself.
- 19:44
I think that that is the way to build a delightful product, right? Is to not test your product, but to sort of live and breathe it and, and live the life of your users.
- 19:53
Like, who do you think is gonna use this thing? Like, if you are not that person, then you have to become that person because that's the only way that I think you can start to feel at the borders of what people are ready for and what the technology is capable of, right?
- 20:06
I think, I think that's ultimately, like, the, the little trick to getting within that space and building something interesting.
- 20:15
Okay. Finally, I think that delight shows us what the product can do and what's magical about it, but I think what a lot of people are not actually as judicious about is what it should do.
- 20:30
Uh, and this is, this is kind of like a weird piece of advice, especially because, like, I've made this mistake many, many, many times. Like, I only say this from sort of the tried and tested experience of being a kitchen sink person, right?
- 20:41
And we've all used products like this, where, uh, when you look around you, there's, there's plenty of examples where it kind of feels like you have not shipped your POV on the world.
- 20:51
You actually just shipped a kitchen sink of model capabilities with your slight flair on it, right? Like, maybe you changed the color. And that's kinda cool because, you know, I, I think, like, we're all still living in the era of, like, research previews.
- 21:02
Like, it's cool to know what the models can do. Like, it's ever-increasing in capability. But I think, like, to true end users, like, it's probably not that interesting, right?
- 21:10
Because that is just ChatGPT. Like, I'll just use ChatGPT. I don't think you're gonna do a better job than them. Um, and it's probably not right to jam everything in there and try to see what's sticking with who, right?
- 21:24
So the barrier to shipping, especially nowadays, it is not... It's not about the capability, right? It's about judgment. And the thing that I try to ask myself is, you know, does this product respect user time, data, and agency?
- 21:39
Because I really believe that restraint in particular, right? This is a new innovation multiplier. Like, if you yourself have that personal clarity, you have that purpose, and you are focused, you're driven, you're resilient, like 100%, people are gonna tell you stuff like, "Whoa, this is like the simplest thing ever.
- 21:57
How do you aim to compete?" Well, exactly. By being focused, right? By doing one thing incredibly reasonably well, and you will get to a point where the thing is so excellent that you are beginning to really delight people.
- 22:09
Like, you will have more of an intuition about the borders that I was talking about in the previous slide.
- 22:16
I think one last note I'll say on delightfulness is that, uh, I mean, on the kitchen sink, is that users are not that different from you or me, right?
- 22:25
Like, we are all users of something. And so think about the last time you tried something new. Like, think about how much patience you had for it, and think about how annoying it was when you tried it and you're like, "I just don't know what I would do with this thing.
- 22:36
Like, it does, like, 30 things, but I don't know when I would use it," right? And that's just, like, what happens when you sort of let the chaos overwhelm you and you are not clear about what the whole point of, like, your thing is, right?
- 22:48
And that, that comes, that comes from a place that I think is, like, the entire stack. Like, if you yourself don't know what you're trying to do with yourself, then you don't know what to do with your product, then you don't know which model capabilities are actually useful, you know, in the context of your outcome, and you
- 23:04
end up with a kitchen sink, which is not great. It's not a great experience. This actually happened to me recently. This is Huxe. Um, uh, I don't know... I, I mean, I do know actually, but we ended up in a place where we had an app that did all these things.
- 23:20
And I was like, "It's so beautiful." Okay? It does this thing. You can chat with it. You can make podcasts. You can read the news. It generates images. It generates videos.
- 23:30
I was so excited. It was, like, my dream product, until it wasn't. Because then I started using it every day, and I'm like, "Whoa, I don't really use any of this.
- 23:39
I just use one thing." And we gave it to a bunch of users. We gave it to several hundred people, and they also just used one thing in the product, and I was like, "Whoa."
- 23:48
You know what? I think in real life, people only have the bandwidth to sort of be like, "This is my one thing," right? Like, I'm sure Spotify does a lot of things, but I still use it for one thing, which is to play music.
- 23:58
I don't know what else it really does, right? I'm sure, like, it's, like, a huge team. It's like a bajillion dollar industry, but that's what it does. And so when we were working on Huxe, I'm like, "Whoa, how crazy.
- 24:07
Oh, I forgot. It does chat too," which was wild, except I made it, like, really goofy instead of ChatGPT. It's just crazy. But I think, like, to look at this product now and to see sort of like the lack of focus in the product direction, it's like, hey, this has a little bit of like that demo disease,
- 24:22
right? Where it looks cool, but in real life, like, nobody's gonna love that thing.
- 24:27
And so I just wanna reiterate kind of the message here, which is clarity is what is gonna give you the energy for the job, and the job is hard, so you're gonna need lots of that.
- 24:36
Purpose is what keeps us focused on it and makes sure that we know what outcome we are marching towards, and it is the trust in that purpose that's gonna earn us the belief, right, of our users.
- 24:49
And the belief is what's going to get us to a place where we have an opportunity to delight, right? And that delight is gonna prove potential. And the kitchen sink is just like a checks and balances kind of a thing, and it keeps us honest and focused about whether or not we're meeting the first four.
- 25:04
And that's how I bu- I believe that we build something that isn't ugly. Thank you. [upbeat music]