AI Engineer World's Fair 2024
The era of unbounded products: Designing for Multimodal I/O
Read the talk
Designing unbounded products people can understand
As AI expands what users can ask, show, and change, interface structure becomes essential: it reveals capabilities, keeps work stable, and makes personalization understandable.
From a talk by Ben Hylak
How do users know what a product can do?
How does someone discover the capabilities of a product that extends beyond a mouse and monitor? Ben Hylak calls these unbounded products. His experience with them began with robotics around eighth grade, continued with rockets at SpaceX, and included four years on the Apple Vision Pro design team developing the first visionOS. He introduces Dawn as a company helping teams including GitHub and Can of Soup build more predictable AI products. The recurring design problem is making a large space of possibilities understandable.
AI expands that space further. Users can type, talk, supply images, or show video. They can also plead, bargain, and confide—social behaviors that carry expectations beyond the format of the input. When the interface does not explain its boundaries, users assume it can do things it cannot. A failed attempt can then convince them that it cannot even do the things it supports. An unclear capability boundary can hide real capability.
Consider how people learn to use ChatGPT. Someone hears that a friend used it for travel planning and tries the same task. Technical users often keep experimenting after a failure because they already believe the underlying models are capable. Other users try once, find that it does not work for them, and never return. A product cannot depend on every user supplying that initial confidence. Understanding how to earn it requires looking first at earlier interfaces, then at contemporary AI products, and finally at more personalized forms of interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From visible buttons to open-ended situations
Traditional screen software makes a relatively clear agreement with its users. They swipe, click, or tap; the developer specifies what happens. Visible buttons expose available actions. Those same actions are straightforward to instrument: attach an Amplitude or Mixpanel event to a button press, and the product team can see what someone did. The interface supplies both a map of capabilities and a vocabulary for observing usage.
Even a small expansion in input can change the design space dramatically. Multi-touch adds a second pointer, but that also introduces relative distance and rotation. Hylak credits those extra dimensions with helping make small smartphone screens practical. The expansion becomes more consequential when software operates in the physical world: his example of software roaming San Francisco streets is accompanied by a photograph of a burning car. The environment can now create situations far outside a developer's intended interaction sequence.
For Vision Pro, the questions begin with ordinary movement. Someone opens apps in the living room, walks into the bedroom, and lies down. What should happen to the apps? A laptop operating system can largely leave that question to the position of the laptop; a spatial interface must decide how software relates to the user and the room. Plane travel, a nearby friend, an inability to move one's neck, or being confined to bed introduce further requirements. These are the spatial equivalent of unexpected AI inputs and the evaluation cases they generate.
A blank slate does not resolve those possibilities. It leaves users and designers facing them all at once. Structure creates clarity by giving an otherwise open-ended product recognizable places, actions, and expectations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the unfamiliar a recognizable shape
The first structural decision in visionOS is what to show immediately. Its home screen foregrounds apps, people, and environments: the categories the team wants users to recognize as central. This is familiar from the iPhone, but Hylak contrasts it with earlier VR menus whose contents made it harder to answer a basic question: what is this device good for? The first view should make the product's priorities legible quickly.
The next decision is hierarchy. In the visionOS hierarchy Hylak describes, the home menu is the point of departure and return. Apps occupy bounded windows that users can move and resize. An individual window can then go full screen. Each level gives the user a different scope of interaction without requiring them to invent their own organization of an unlimited space.
Familiarity makes that hierarchy easier to learn. Dawn's early prototype used an exploratory star cluster: enjoyable to navigate, but eventually replaced with tables, graphs, and examples. The same reasoning explains why the visionOS TV app resembles the tvOS TV app. Reusing a recognizable design gives people an orientation aid when the surrounding experience is unfamiliar; it is not evidence of laziness. Control Center serves a similar purpose. The three decisions reinforce one another: foreground what matters, establish a hierarchy, and use familiar forms to make that hierarchy understandable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A journal and a search engine should not feel the same
These principles do not imply that every AI product needs the same layout. The appropriate structure depends on what the application is for. In the interfaces Hylak shows, Dot is a companion that he also uses as a journal. Pinching out separates the conversation into days. Tapping a person—or, in his example, his two co-founders—reveals structured information and a timeline of mentions. The conversation remains the source material, but days and people become ways to navigate it.
Perplexity uses a different arrangement to communicate a different purpose. It promotes the query to a page title, highlights the sources, and places the answer underneath. The answer occupies a full page, making the interaction feel more like a search result than one turn in an ongoing conversation. In the displayed example, the query concerns coffee shops to work from in Paris; the final slide outlines the query, source cards, and answer as distinct regions. Structure tells the user how to read the result before they read its contents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the work from drifting away in chat
A control can be useful in isolation and still be confusing in its surrounding interface. Hylak illustrates this with a Vercel chatbot demo, while emphasizing his admiration for Vercel's design work more broadly. Asking to buy Doge produces a purchase interface with a quantity slider. That is a sensible improvement over repeatedly typing quantities into chat.
The difficulty appears as the conversation continues. Follow-up messages push the control away, and eventually there are two purchase interfaces in the transcript. Returning to the first one and pressing purchase means interacting with a different conversational state. Hylak compares the drifting interface to the floating house in Up: the user has a useful object, but its location and relationship to the current task are no longer stable.
Separate the work object from the conversation that changes it. A stable workspace can update as messages arrive without displacing the user's point of interaction. Claude Artifacts applies this pattern to an app being developed: the app occupies a separate area alongside chat, and the user can switch between versions without scrolling through messages. The conversation records the process; the artifact provides a persistent place to inspect the result.
A small TypeScript state model illustrates that separation. The artifact keeps its identity while a revision is appended and selected; viewing an older revision does not require modifying or searching the conversation.
typescript
type Revision = {
id: string;
source: string;
};
type Artifact = {
id: string;
revisions: Revision[];
selectedRevisionId: string;
};
function addRevision(
artifact: Artifact,
revision: Revision
): Artifact {
if (artifact.revisions.some(item => item.id === revision.id)) {
throw new Error("Revision already exists");
}
return {
...artifact,
revisions: [...artifact.revisions, revision],
selectedRevisionId: revision.id,
};
}
function selectRevision(
artifact: Artifact,
revisionId: string
): Artifact {
if (!artifact.revisions.some(item => item.id === revisionId)) {
throw new Error("Unknown revision");
}
return { ...artifact, selectedRevisionId: revisionId };
}
const first: Artifact = {
id: "greeting-app",
revisions: [{ id: "v1", source: "<h1>Hello</h1>" }],
selectedRevisionId: "v1",
};
const updated = addRevision(first, {
id: "v2",
source: "<h1>Hello, friend</h1>",
});
const viewingOriginal = selectRevision(updated, "v1");
Here, viewingOriginal still contains both revisions. The selected view changes, but the work object and its accumulated history remain available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make changes recoverable and context understandable
Versioning addresses a related obstacle: the fear that another attempt will destroy something useful. Hylak points to message editing and version switching as features present in the original ChatGPT experience. Underneath that interaction is a branching conversation tree. Editing an earlier message creates an alternative path rather than simply replacing everything that followed. In v0, he sees the same benefit expressed through a familiar interface that resembles working in Google Slides: users can keep iterating on a UI while retaining access to earlier versions.
Familiar boundaries matter for memory as well. Hylak finds it uncomfortable that information about a medical problem could remain available while he works on JavaScript in a different chat. A project supplies a more understandable context boundary: material relevant to one body of work belongs together. Claude Projects makes that boundary concrete through supplied project knowledge and instructions; it should not be understood as automatic recall of every conversation within a project. The design advantage is that users can reason about why a piece of context is present.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make agent steps visible as columns
Agents introduce unfamiliar ideas: multiple tasks, intermediate results, and data passed from one step to another. A spreadsheet turns those ideas into a recognizable arrangement. Hylak says that the real-world agent uses he has personally encountered have been in spreadsheets, and uses Clay to explain why the format works. Each column represents a user-defined step. Moving across a row builds up the context required by later steps.
| Spreadsheet element | Workflow meaning |
|---|---|
| Column | A user-defined agent step |
| Row | One instance of the workflow |
| Successive columns | Accumulated context and intermediate results |
| Final column in the example | A personalized email |
This layout also gives users a natural way to expand execution gradually:
- Run a small initial set of rows and inspect the results.
- Adjust the steps until the workflow behaves as intended.
- Apply the workflow to a much larger set.
Hylak illustrates the progression with ten initial rows followed by fifty thousand or a hundred thousand. Those are example batch sizes for the procedure, not measured throughput. The useful mechanism is the ability to inspect intermediate work before applying it broadly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Offer a useful starting point
A blank prompt field asks users to discover both what the product can do and how to ask for it. Examples and presets reduce that burden in several distinct ways:
- Suggested tasks: ChatGPT offers starting points such as comforting a friend or planning a relaxing day. These make possible uses visible.
- Working examples: v0 adds an Explore page to its suggestions, letting users see what other people are making successfully.
- Named transformations: Notion offers a tone menu instead of requiring users to compose an instruction about being a concise GPT. The product team can validate these predefined operations.
Each pattern moves some of the work of discovering a useful prompt from the user into the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Direct controls need more than broad labels
The next step is to make generation feel less like indirect manipulation through carefully chosen words. Hylak recalls Linus's earlier metaphor of a llama trying to drive a car from the back seat with a pool noodle. It captures the distance between an intended change and the prompt used to induce it. Hylak expects less prompt engineering as interfaces expose more direct controls. Apple's image-generation interface, which combines concepts, and demos that manipulate emotional expressions suggest that direction.
The Figma Slides launch provides another example. Hylak describes text controls spanning professional, casual, expanded, and concise. His reference to their release the previous day belongs to the June 2024 launch context. The appeal is immediate: users adjust the property they care about instead of writing instructions for how to change it.
But a label such as casual conceals important distinctions. A Fortune 500 company and a direct-to-consumer cosmetics brand advertising on TikTok may both want casual copy, yet expect very different language. The same person writes differently to a best friend and a coworker. A small menu can simplify interaction by collapsing distinctions that actually matter. The design challenge is to make control easier without making its vocabulary reductive.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From coarse presets to feature steering
Hylak's proposed response is to increase the number of available presets enormously. His millionfold or billionfold language expresses the ambition: enough specificity to cover distinctions that a few generic labels cannot. He sees sparse autoencoders as a promising route toward identifying more granular features that an interface could expose.
Golden Gate Claude demonstrates one form of feature steering. Anthropic increased an internal Golden Gate Bridge feature in Claude 3 Sonnet, causing the model to bring the bridge into its responses persistently. The intervention changed internal activations rather than supplying a prompt or performing traditional fine-tuning. It showed that an identified feature could influence behavior, but Anthropic also warned of unexpected effects. The temporary public experiment is no longer available; it does not establish reliable control over arbitrary user preferences.
The following image demonstration makes the interface possibility more tangible. Hylak shows controls for the play of light and shadows, serene forest streams, and Venetian canals, describing the changes as controllable and predictable. The selected frame, labeled @gytdau, shows a canal scene beside reference thumbnails and the feature controls. Here the user-facing vocabulary concerns properties of the image, rather than an elaborate instruction for generating it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve the right controls, then personalize them
A much larger preset space creates another discovery problem: how does anyone find the relevant options? Hylak proposes ranked presets that are personalized, searchable, and invocable through natural language. They need not all appear as visible controls. A request such as more friendly could retrieve dimensions involving kindness, interpersonal closeness, and confrontation. The user might adjust those dimensions directly, or the application might use them behind the interface. Natural language becomes a way to locate structured controls rather than the only mechanism for influencing output.
Once developers can define those features, Hylak proposes tuning them separately for each user. This is developer-defined personalization: the product supplies the dimensions, while their settings vary with the person. He contrasts that possibility with the fragility of text prompts, where removing a word can change the output. The proposed advantage is a more explicit set of personalization controls; the talk presents this as a future direction, not a demonstrated personalization architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure whether the result fits the user
As an application behaves differently for different people, its measurement problem changes too. Hylak describes a shift from evals toward analytics. Some questions still have objectively correct answers: identifying the first president is his example. But the right tone for a particular user's summary is not determined in the same way. In these domains, a shared answer key cannot fully describe success.
The measurement target becomes whether the product understands what users ask for and meets their needs. That preserves a place for factual correctness while recognizing that personalized output also has to fit an individual. An increasingly unbounded product therefore needs more than the ability to generate many possible results: it needs a way to understand which results are useful to the person using it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic explains its experiment amplifying a Golden Gate Bridge feature in Claude 3 Sonnet. The temporary public demo is no longer available.
The original announcement explains project knowledge, custom instructions, and a separate Artifacts workspace alongside chat.
Figma’s launch article introduces Slides and its AI tools for changing text length, tone, and language.
Read the complete timestamped transcript
- 0:00
[upbeat music] I'm so excited to be here with you guys today at I think what is, uh, probably the coolest, uh, AI conference,
- 0:21
uh, in the world at such an exciting, exciting time in history, I think, especially for AI products. Um,
- 0:29
if you don't already know me from either demos on Twitter or sometimes probably ill-advised spicy takes on Twitter, uh, my name is Ben Hylack, and, uh, I'm the founder of Dawn.
- 0:44
So at Dawn, we help some of the best companies in the world, everyone from GitHub to Can of Soup, build better,
- 0:57
more predictable AI products. My entire life, I've been
- 1:05
like really obsessed with building and designing unbounded products. So unbounded products are products that transcend the, uh, the, the mouse, the monitor in some way, right?
- 1:21
So for me, that started with robotics, uh, when I was-- I think the first one was when I was in eighth grade. Uh, eventually rockets, uh, at SpaceX. So these are very, [chuckles] these are very unbounded products, right?
- 1:36
Um, and then most recently, uh, I was on the design team for the Apple Vision Pro for four years. So we designed the first version of visionOS.
- 1:52
I think that AI makes products less bounded than they've ever been, right? You can type, you can talk, you can show images or show video, just like we just saw.
- 2:10
You can also sort of plead, you can bargain, you can confide, right? These are very interesting sort of input modalities. And this unboundedness
- 2:23
often makes products unpredictable, right? Confusing, hard to understand.
- 2:30
Users assume your product can do things that it can't. They try to do those things, doesn't work,
- 2:39
and they walk away thinking that it can't even do the things that it can.
- 2:45
When you talk to people, and specifically people that are not in this room, how they use ChatGPT, how they learn how to use it,
- 2:54
it's often word of mouth, right? So they hear one of their friends say that they used it for travel planning, and then they go use it for travel planning.
- 3:03
A lot of us, a lot of us in this room, especially like people that are more technical, we often learn through trial and error, right? So we just keep trying, keep trying.
- 3:11
We keep trying because we know that these models are good, right? We know that it's impressive. Um, but a lot of people are not. [chuckles] They don't do the trial and error thing, right?
- 3:18
So they try it once. Eh, it doesn't work. Uh, they don't try again.
- 3:24
And so this talk is about making good AI products.
- 3:30
And to that end, I'm gonna cover just three things.
- 3:35
So those three things are the past, right? So how have products become more unbounded, and what has worked for unbounded products in the past?
- 3:45
The present, which is AI products today. What are sort of good design patterns and bad design patterns? And then the third point is gonna be the future, right? So again, just three things, just the past, the present, and the future.
- 4:01
Easy. So we're gonna start with the past.
- 4:07
So most software that we use lives on a screen, right? And you use it just by typing-- Sorry, you use it primarily by swiping, clicking, and tapping, right? Uh, when you click something, whatever the developer expected to happen is what happens, depending on how good of a developer you are.
- 4:31
It's easy for users to understand what your app can do. They look, they see the buttons, they get it. It's also very easy for you to understand what your users are doing.
- 4:41
You just add a Amplitude or Mixpanel call on a button press. You see what they did.
- 4:47
So if you think about one of the biggest changes to this, uh, previous to the last two years was multi-touch, right?
- 4:57
And this is just like instead of one pointer, you have two. But just by adding that second pointer, you get relative distance, you get rotation, right? And just this one little change like largely made the smartphone possible, right?
- 5:12
Like, m- largely made it easy to use a screen that is that small.
- 5:17
And now it's just getting crazy, right? It's like we have unbounded products everywhere. Products are so unbounded. You have software, you know, just freely roaming the streets of San Francisco, getting attacked by fiery mobs, right?
- 5:28
So this is getting crazy. Um, and so I wanna talk a little bit about just one unbounded product that's--
- 5:38
I got to work on, uh, which is the Vision Pro. And what I wanna talk about is just three lessons that we learned while we were designing it, lessons that I think aren't as intuitive looking from the outside in.
- 5:54
So I think that unbounded products are often defined by this what if question. Like,
- 6:00
when we were starting-
- 6:03
It's like users get themselves into the craziest situations. So something as simple as, oh, well, what if someone's in the living room and then they move to the bedroom and they lay down on their bed, right?
- 6:11
What should happen to your apps? If you're designing macOS that's on a laptop, you don't have to worry about that, but that's something we had to think about, and there's hundreds of more questions like this, right?
- 6:22
What if someone's on a plane? What if someone's next to their friend? What if someone has a, a, a disability of some sort, right? Like, uh, they cannot move.
- 6:30
They can't move their neck. They're bedridden. So all of these what-ifs, and I think this is again what defines unbounded products, right? A-all of us that are building AI products, we're constantly thinking, you know, oh, like, what if someone puts in this?
- 6:42
What if someone puts in that? And there's evals, et cetera, et cetera.
- 6:47
And so without structure, you just have chaos, right? You have a blank slate. You have all these what-ifs. Um, infinite world of possibilities. And so it's really on us as product designers to add structure, and structure is what creates clarity.
- 7:04
So again, I wanna talk about three ways we added structure. The first was highlighting what matters and doing it really fast. So the first thing you see in visionOS is a home screen.
- 7:16
It has apps, it has people, and it has environments. So those are the things that we think matter when you're using visionOS.
- 7:25
So they're the first thing you see. Might not sound that novel, and it's not. In a [laughs] lot of ways, it's the same thing that happens on your iPhone. But when you compare it to VR products that came before,
- 7:38
it's very hard to understand how you're actually... You know, what is this thing good for when you look at this menu?
- 7:46
The second point is hierarchy. H-hierarchy is what gives unbounded products a shape and a purpose, right? It's what helps users understand what it's good for, what they should use it for.
- 8:03
So again, we have the home menu. That's kind of where everything starts and ends for visionOS.
- 8:09
We have windows. They have bounds. You can resize them and move them.
- 8:18
And any individual window can go full screen, right? So that was our hierarchy.
- 8:24
The last point, which is really important, and I think the easiest way to make u-unbounded products feel familiar is-- Sorry, intuitive. I got ahead of myself. It's familiar. Familiarity.
- 8:38
Uh, it was something we hit [laughs] when we were building Dawn. Our first kind of prototype was this star cluster thing that you could explore. It was really fun. Um, nowadays it looks a lot more like this, which are, you know, tables, and we have graphs and examples.
- 8:54
Again, it's just structure, clarity. And I think that it's no accident that, you know, the TV app on visionOS looks a lot [laughs] like the TV app on tvOS, right?
- 9:09
It's not an accident. It wasn't laziness. Uh, when people are sort of in uncharted territory, you wanna give them as many signs as home-- signs of home as possible.
- 9:22
Same thing for Control Center, right? When people see visionOS, they already know how to use it.
- 9:29
So again, these three points: highlighting what matters, bringing that to the forefront,
- 9:37
establishing hierarchy, and then leveraging familiarity. All right.
- 9:45
And so that was the past. Now we're gonna talk about the present, and specifically,
- 9:51
we're gonna talk about AI products. We're gonna talk about ways,
- 9:57
both good and bad, that products have been incorporating structure into their AI features.
- 10:07
It's really important to note the right struct-- Sorry, the right structure is very unique to your app, right? That's the whole point is that it gives your app a shape.
- 10:18
It helps your user understand what it's actually for.
- 10:22
So let's take something like Dot, right? Dot is a companion. Dot is sort of a journal, at least for me. And so the structure they added was that if you pinch out, you can see each day separated, right?
- 10:36
It feels a lot like a journal. And if you tap a person or two people, in this case my co-founders, I can see this, again, structured information about them and a timeline
- 10:48
every time I mention them to Dot. And so again, you're pulling that structure out of the chat.
- 10:55
Perplexity does a really good job of using structure to make their experience feel more like a search engine and less like ChatGPT, less like a chat you're having a conversation with, right?
- 11:05
And they do this by, you know, really pulling your title, you know, your query up top as like a title. You know, highlighting the sources it came from, and then having the answer below that, right?
- 11:18
And then having that take up the full page kind of regardless. So it makes it feel, again, like more like one shot, less something you're having a back and forth with.
- 11:29
Now I wanna talk about, uh, sort of an anti-pattern I've seen, which is, um, this is in the, the Vercel, uh, chatbot demo. I think Vercel does some of the coolest design work in the entire world.
- 11:41
I didn't like this one. Um, so this is like having this idea of almost ephemeral UI, but inside the flow of chat, right? And I get the appeal, right?
- 11:51
So actually, if we go back here... Sorry, this was the video I meant to show. Um, you know, you have a slider, right? So instead of having to, like...
- 11:57
You inquire about, you know, you wanna buy Doge, and it shows you this UI so you can adjust exactly how many instead of having to do it over text.
- 12:04
It's, it, you know, could be good. The problem is that when it's stuck inside this sort of unstructured thing, it starts, like, floating away as I try to ask follow-ups, right?
- 12:15
And then at some point, I even have two of them, right? So I go back up to the first one, I press purchase, and now I'm interacting with something that's completely di- different.
- 12:23
So it reminds me a lot of sort of that, the house in Up, right? It's just kind of up, up away.
- 12:29
So instead of trying to put structure, stuff it into this unstructured thing, I think the answer is you pull it out, right? You pull it off to the side.
- 12:39
And what that means is that as the conversation continues, you can just sort of update that structure
- 12:45
without disrupting where the user is. And that's exactly what Claude did with artifacts, right? And I think why it's so successful is that they pulled out the structure, which is the app you're working on and iterating on, from the actual conversation.
- 13:00
And then so as you make changes, you can even go between the versions here without even having to scroll the conversation, right? So it's beautiful.
- 13:08
And it actually brings us to another thing that I think has been really effective for AI apps, which is this almost concept of version control.
- 13:16
So this was actually one of the, like, shipping, like, o- original ChatGPT features, which is kind of crazy. But if you edit a message, um, you can go between the versions, right?
- 13:26
And it actually maintains this entire tree. It's very complicated, um, but super powerful.
- 13:32
With v0, Vercel did something, again, amazing, where it feels extremely familiar, almost like you're working on Google Slides or something. Um, but you can go back and iterate, keep iterating on UI without having to be afraid that you, you're losing something, right?
- 13:48
So again, versions. Again, I think fa- familiarity is really one of the most important things for unbounded products. Um, I think Claude did an excellent job with this. Again, I'm, I'm hyping them up here.
- 14:01
But, uh, ChatGPT introduced memory across all of your chats, right? Completely unbounded. So when I tell it something about, you know, some sort of medical problem as I'm working on a JavaScript, it's like, you know, it, it knows that, which is very weird to me.
- 14:15
Um, I think this idea of projects and that structure of a project is very familiar. Um, so sharing context across a project makes more sense.
- 14:24
Agents are something that are extremely unfamiliar to most people,
- 14:29
and, uh, this idea of having, you know, all these different tasks, and you're feeding data between steps, whatever. But you know what is familiar
- 14:38
are spreadsheets, right? Spreadsheets are extremely familiar to, uh, not to me, actually, but to a lot of people. And, um,
- 14:46
I think the only real uses of agents I've seen in, in the world, in the real world, are, are spreadsheets. So this is Clay, right? And each column is essentially a step that an agent is taking, the user is defining.
- 14:59
So it's going across, it's building up kind of context across the spreadsheet. Each row, it's often almost you, you, you, you do it like a eval, right? So it's like you, you run the first ten rows, and then you run the next fifty thousand, a hundred thousand, right?
- 15:14
So you, you get it right. And you can see here, eventually you end up with the personalized email as the last column, um, but with all these steps in between.
- 15:24
The next thing that I think is extremely effective in helping people understand what your app is for and skipping all the sort of noise of prompt hacking, prompt engineering are examples and presets.
- 15:38
So ChatGPT, I think, was the first for this, where they had these, you know, message to comfort a friend, plan a relaxing day, and so on.
- 15:48
v0 does an awesome job with this, right? We're not, not just having those suggest-- those suggestions below, but they also have an explore page where you can see what other users are doing, what's actually working, right?
- 15:58
Again, try to, like, shortcut this, like, prompt, uh, you know, blank canvas problem.
- 16:06
Notion as well, right? They have a simple menu where you can change tone for text instead of having to, like, be like, uh, you know, "You are a very concise GPT," whatever, whatever, whatever, right?
- 16:16
So you're just using these tried and proven, uh, things that Notion can validate.
- 16:21
And that last point brings us to the future, right? So where are interfaces going in the future?
- 16:28
Uh, Linus gave an awesome talk last year, uh, where he described prompt engineering as this almost trying to drive a car, a llama trying to drive a car with a pool noodle from the back seat was, I think, his, his metaphor.
- 16:40
Uh, and there's some real truth to this, right?
- 16:43
And so I think, first of all, the future has a lot less prompt engineering. And we're already seeing this, right? We're already seeing this with, um, generative images, you know, the way that Apple designed it, where you're mixing and matching these different concepts.
- 16:59
Uh, you're able-- you know, there's a ton of demos on Twitter of people, you know, essentially you're going between emotions here in a more intuitive way.
- 17:07
And then just yesterday, Figma released this way of adjusting the tone of text, right, where you're going between professional, casual, expanded, concise.
- 17:16
The problem with this is that casual means a lot of different things, right? Casual for a Fortune five hundred company and a, um, you know, direct-to-consumer cosmetics brand, uh, you know, with ads on, on TikTok, right?
- 17:30
These are very different things. Casual when talking to your best friend or a coworker, these are different. So how do we avoid being reductive when trying to offer these sorts of presets?
- 17:43
And the answer is you just, like, mil-- I, I don't know exactly how many zeros I put here, but you just, like, million X or billion X the number of presets, right?
- 17:50
So you have enough presets for everything. And
- 17:54
I think sparse autoencoders show a really promising path towards that. So if you guys have tried Golden Gate Claude, where you can kind of identify the one feature of Golden Gate bridgeness and amplify it, and it makes Claude obsessed with bridge-- uh, Golden Gate Bridges, uh, specifically, or the Golden Gate Bridge.
- 18:12
Uh, my friend Gidis has an amazing demo towards this, but for manipulating images, right? So you can see here he's increasing the amount of play of light and shadows, increasing the amount of serene forest streams or Venetian canals, um, in a, again, a very controllable and predictable way.
- 18:30
Okay, but, uh [laughs], so now we have a million, billion options, whatever. How do we avoid too many options?
- 18:37
I think this gets to point three, which is ranked presets. So these are presets that are personalized, searchable, and even invoked through natural language. They might not even be directly visible to the user.
- 18:49
So the user types in something like more friendly,
- 18:52
and you pull up the corresponding presets, like kindness, how close you are, how confrontational it is. Again, maybe they're directly editing it, maybe they're not.
- 19:03
And this gets to the last point, or second to last point, which is developer-defined personalization. So as soon as you're able to define those sort of features,
- 19:11
you can start tuning them per user. So each user,
- 19:16
in a way that you can't do with just text prompts today, right? Because text prompts are sort of this fragile house of cards where if you remove one word, the whole output changes.
- 19:26
So you're able to tune it per user.
- 19:30
And the last point, and especially true as you start, your, your app is gonna become increasingly different per user,
- 19:39
is shifting from evals to analytics. I don't think there's going to be some objectively correct, for a lot of domains, answer to things. Like, who was the first president?
- 19:49
Yes. But the right sort of tone for a summary for a specific user, I don't think so. And so I think that increasingly it's going to be about how do you understand if you're meeting the needs of your users and what they're asking for.
- 20:06
So, uh, that's it. Thank you so much. Oh yeah, we'll skip this one. And thank you so much for coming. [outro music]