AI Engineer Summit 2023
The Intelligent Interface
Read the talk
The Intelligent Interface
A camera-aware laptop, an expressive headset, and an imagined image canvas explore how computers can choose their responses from the context around a request.
From a talk by Samantha Whitmore and Jason Yuan
Sensing, thinking, and reacting
What should our relationship with a computer look like when its interface can interpret context? For Samantha Whitmore and Jason Yuan, co-founders of New Computer, that question starts with reconsidering the metaphors and abstractions that computing has long treated as fixed. Before choosing a better interface, they ask what intelligence does: take information from different sources, make sense of it, and choose a response.
Yuan describes this as sense → think → react. People perceive their surroundings, process what they perceive, and decide how to respond. In his framing, processing multiple sensory inputs together supports short-term memory and a theory of mind: an interpretation of what is happening, how others relate to us, and what we should do next.
Language is one way to communicate and think, but it is only part of this loop. Onstage, Yuan makes eye contact, gestures with his gloves, looks toward Whitmore, and points at objects. Audience laughter provides another signal. These channels operate together; the words alone do not describe the whole interaction. A computer that responds only to explicit language has access to a narrower picture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Language opens the loop
ChatGPT made an approximation of this loop tangible: language enters, some reasoning process intervenes, and language returns. Whitmore leaves the internal nature of that reasoning unspecified. The interesting design opportunity is what comes next—how interaction could feel more natural when the computer can work with more than the words a person deliberately supplies.
The demonstrations move from a prototype running on an ordinary computer to imagined interactions with future hardware. Their range includes gaze, implicit and explicit gestures, sound, expression, language, touch, attention, and drawing. These are ingredients for interaction, not a requirement to use every channel at once. The demonstrations are exploratory stories, and the founders explicitly distinguish them from the product New Computer was building.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When should a laptop listen?
The laptop prototype changes the question from whether text or audio is the right interface to when each is appropriate. Its camera supplies real-time pose measurements describing Whitmore’s position relative to the screen. An LLM uses those measurements to select how the interaction should proceed.
| Context detected | Modality selected |
|---|---|
| User is close to the screen | Keyboard input |
| User is facing the screen | Text output |
Proximity and orientation answer different questions. Being near the keyboard suggests how the user can provide input; looking at the screen suggests where a response can be presented. The prototype exposes these decisions in a side panel.
Whitmore then walks away. Once the camera no longer detects her, the intended behavior is to start listening so she can converse while moving around the room. She calls out, but the system does not answer promptly. This is the counterpart to its intended behavior at the keyboard: when she is typing nearby, it should avoid attending to her voice or ambient sounds.
The system eventually speaks, after an awkward pause and some laughter. The demonstration therefore shows a delayed transition into spoken interaction, not a seamless handoff. No measured latency or established cause accompanies the delay. Behind that imperfect execution is a concrete design: use proximity and screen orientation to decide whether to listen, speak, or display text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Context becomes world state
The goal is for the computer to adapt to a person’s circumstances instead of requiring a fixed posture and interaction method. The prototype separates sensing from response generation through world state, an interpretation of the user’s current situation. Its architecture proceeds in three stages:
- Process the image stream with pose detection and the audio stream with voice activity detection.
- Use an LLM to update world state from those signals, including where the user is and whether they are speaking.
- Generate a response using that state, selecting both what to say and how to present it.
Whitmore describes this as a simple, rough prototype that could be built with the technology available at the time. The architectural point is that context can change even when the user is not explicitly addressing the assistant.
That gives an interface two complementary sources of information:
- Explicit inputs: what the user types or says.
- Implicit inputs: where the user is, with possible extensions such as vocal tone or detected emotion.
These signals can affect more than the output medium. Yuan offers a hypothetical social constraint: if the assistant remembers Whitmore complaining about him, and he is now visible beside her, it should wait until he leaves before mentioning it. Context would govern whether a response is appropriate at all, not merely whether it should be spoken.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Blending images as a physical action
The next imagined interface is a Midjourney canvas for iPad. It begins with a generated image of Naruto dressed in Balenciaga. This is a proposed canvas experience; the linked blend documentation does not establish the touch and shaking interactions shown here as a shipped interface. To edit the image, Yuan first zooms out, creating room around it using an already familiar gesture.
He then imagines his cat in the outfit. An assistant with personal context retrieves pictures of his cat, making those images available alongside the generated character. He picks up a cat photo and brings it together with the other image to express blending. The selected photos glow: light becomes the visual metaphor for combining sources, drawing on the familiar idea of colors mixing and concentrating into a new form.
The resulting human–cat hybrid is not what he wants. Rather than switching to a separate editing vocabulary, he asks what physical action already means remixing. A blender bottle provides the answer: shake it. In the concept, shaking remixes the image, carrying the material metaphor through both the initial combination and the next attempt.
Generative intelligence is a probabilistic material. Yuan compares designing with it to designing a phone with an understanding of aluminum or titanium. If generated results are fluid and variable, light, fog, or mercury may offer more suitable metaphors than rigid wood, paper, or metal. The proposal is to make the material’s behavior legible through familiar actions and feedback, rather than merely decorate a conventional interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Selecting, correcting, and remembering at the desk
The final scenario moves the interface onto a physical desk. An imagined wearable has a camera, microphone, and the ability to project feedback. Yuan browses a Beyoncé tour book and points to an inspiring image while verbally expressing interest. The gesture identifies the object; speech supplies the intention. Projected feedback acknowledges that the device has understood and will act. The shown sequence lacks sound, but the proposed interaction combines voice and gesture simultaneously.
That interpretation can be wrong. After accidentally selecting a photo, Yuan flicks it away like dust. The correction uses the same physical vocabulary as the selection instead of requiring a separate undo control. Probabilistic interpretation needs a graceful way out: the interface must make a mistaken inference easy to reject.
With two pieces selected, he wants to share them with two friends but cannot remember which friend he had agreed to spend Halloween with. He asks the assistant to recall that person. The imagined response appears as interface cards beside the book, keeping the selected images and the contextual question in the same workspace. The scenario illustrates recall and recipient discovery; it does not establish that a message has been sent.
The input and output channels do not have to match. Voice input need not produce speech; gesture input need not produce a gesture-based response; visual input need not return another visual interface. Multiple simultaneous inputs can establish intent, while the computer chooses whichever output suits the current setting. Here, a spoken question can lead to projected information beside the physical objects already receiving the user’s attention.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing what an interface should do
This design space does not depend entirely on future hardware. Existing laptops already offer possible contextual signals, including typing speed and vocal tone. Those signals can be processed before being passed to an LLM reasoning layer, which decides both the wording of a response and its presentation. The interface decision becomes part of responding, rather than a fixed wrapper around generated text.
More possible outputs do not make every output useful. Yuan challenges the idea of generating a photo on an Apple Watch when a phone would be more suitable. Intentional device and modality choices still matter. Familiar metaphors—from nature and physics to tools, materials, and ordinary buttons—can give users stable expectations even when the system produces variable results.
Social expectations provide another boundary. Yuan’s example is a banking assistant that should not drift into a deep philosophical conversation simply because its model can sustain one. An agent’s role helps determine which interactions make sense, just as the user’s surroundings help determine the right output channel.
All these interfaces borrow from what people already perceive and do: looking toward someone, dismissing a call, mixing materials, brushing away dust. That makes them understandable, but it also leaves an open question. If these familiar behaviors are the skeuomorphism of intelligent interfaces, what abstraction comes above them? The next interface language remains to be designed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original ChatGPT announcement explains its conversational format, training approach, and limitations.
Further reading
Apple’s 2023 introduction describes a spatial interface controlled through eyes, hands, and voice.
A practical guide to detecting body landmarks in browser images and video with MediaPipe.
Updates since the talk
Instructions for combining images with Midjourney’s Discord blend command and web image prompts.
The founders explain their decision to wind down New Computer and Dot.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, everybody.
- 0:15
Thanks for having us here today. Um, we're super excited to be here. I'm Sam, and I'm one of the co-founders of New Computer.
- 0:22
And I'm Jason, the other co-founder. And we're really excited that we are starting today by letting you all see our pores up close- [laughs] ... um, which is amazing. Um, so, you know, when Sam and I started New Computer, we, we did so because we believe that for so long we've taken certain metaphors and abstractions and tools for
- 0:43
granted, and for the first time in what feels like 40 years,
- 0:49
we can finally change all of that, and we can start thinking from first principles what our relationship, not only with computing, but with intelligence, period, should look like in the future.
- 1:00
So what do we mean by intelligence? Because, uh, you know, sometimes I'm on the internet, and I wonder if it even exists. Um, well, one way to think about intelligence is, uh, the ability to sort of take in lots of information, different types, different volumes from different sources, um, visualized as dots here, and sort of find ways
- 1:22
to make sense of it all, find ways to reason, find ways to find meaning. Um, and as human beings, as carbon-based life forms, we do this through a, a process where at first we use our senses to sort of perceive the world around us.
- 1:36
Um, then we, you know, process that information in our heads, and then given what we think, we then choose a reaction. Um, so if we're lucky, we are blessed with at least five senses, six when I've had four margaritas.
- 1:55
Um, but as humans, we sort of are in- inherently capable of just processing all of this at the same time, and that actually is how our short-term memory gets to work.
- 2:07
Um, and taking in all this context and information, we then get to form what's called a theory of mind. Um, what is going on? What is, you know, how is the world relating to me right now?
- 2:16
What should I be doing about it? So we sense, we think, and then we react. Um, and how do we react? Well,
- 2:26
um, there's a lot of things right now. Uh, but if we take it all the way back to the Stone Age, and we think real simple, um,
- 2:34
a lot of how people used to react and communicate is just unintelligible grunts. Um, and then one day we, that sort of evolved into a language as we know it. [laughs]
- 2:44
Um, and to this day, that's still something that we rely on to communicate and react to the world around us. Um, and that's also how a lot of us think.
- 2:54
So we have language. Um, but the language of communication
- 3:00
is so much broader than just language. We're standing here on stage right now. I'm making eye contact with some of you. Nice shirt. Um, and I'm making gestures. I'm wearing these ridiculous gloves.
- 3:10
I'm looking at Sam. I'm looking at things. I'm pointing at things. Um, and I can hear sort of laughter or I can hear people, you know, thinking. I'm taking lots of information at once, and right now I'm sensing, thinking, and reacting.
- 3:26
So this year, um, well, last year technically, we saw a really amazing thing happen, um, kind of with the advent of Chaffy- ChatGPT, I would say, where we saw the beginnings of a computer start to approximate that same loop, where input was coming in in the form of language.
- 3:44
There was some reasoning process when, however that, uh, actually works. Um, and then the output felt also like language coming back to us. And this was very inspiring to me and Jason, and we've been spending a lot of time this past year thinking about what's next and how this gets to feel even more natural, um, for people
- 4:06
to interact with computers specifically. And so today we wanted to take you on a tour of a few demos. Um, one, um, which you can do with the computer right now, um, and then a few which are kind of with futuristic or, uh, next-generation hardware, which may be available soon.
- 4:23
And knowing that you're all engineers, we know that this will kind of get the sparks flowing, um, the ideas flowing for seeing how, like, you might use, um, some of these things that are coming out soon or things that exist today to build things that feel more natural.
- 4:38
So I'll start by getting to a demo, and I will say, um, this is a live audio visual demo, so I am foolish enough to make that choice. So we will see how it goes.
- 4:52
Um, before we show any demos, it's prudent to point out that none of these represent the product we are building. They are simply-
- 5:02
Yes
- 5:02
... pieces, stories of inspiration. Um-
- 5:06
So the point of this first demo is to imagine we have a lot of things where we're saying, like, okay, is text the right input? Is audio the right input?
- 5:15
And we've been thinking about, it's not if those are the right things, but when. So in this case, you'll see some measurements happening on the left here. What's actually happening is that this has, this has access to my camera, and it's taking, uh, real-time pose measurements of where I am with relevant, relative to the screen.
- 5:35
So I just, it knows I'm at the keyboard basically because it's making that assessment, and you can see the reasoning in the side here, where it's saying, "User is close to screen.
- 5:42
We'll use q- keyboard input. User is facing screen. We'll use text output." And so this, we're using an LLM to actually make that choice as it, as it goes to the response.
- 5:51
So let's try something else. And again, demo gods be nice- [laughs] ... because this may not work at all. [laughs] Um, but if I now walk away,
- 6:01
and it doesn't detect me anymore, it should now actually start listening to me. Hello? Can you hear me? Are you gonna respond?
- 6:09
I think that's a no
- 6:10
It might not respond. [laughs] But basically, what we are attempting to build here is, like, if I want to actually talk to the computer in a really natural way, um, if I'm there next to the keyboard, I should not, it should not be paying attention to my, uh, voice or any sounds, ambient sounds.
- 6:30
And if I walk away from the keyboard, I might wanna have a conversation with it, like, walk around the room. It is listening. It seems to not, to have decided not to actually talk back.
- 6:39
But, oh, it's talking. [laughs] [laughs] Demo.
- 6:49
Is there something you need help with, Alexa?
- 6:50
That sounds like an interesting project, Samantha. [laughs]
- 6:52
How is your talk going so far?
- 6:53
Yay. [laughs] [clapping] Yes, you can see it paid attention, and it decided to ignore me for a while. [laughs]
- 7:08
But anyway, this is, this is just, like, a toy demo. You can see here we have, um, this is how it's working kind of behind the scenes. It's, like, trying to decide if I'm close to the keyboard, facing the screen, not facing the screen, and use that all as inputs to decide whether it should talk to me
- 7:25
or, um, just display the text as on the interface. Um, cool. So-
- 7:33
The reason why we're, we, we think this is interesting is because we think, you know, people are naturally sensitive to other people. And, um, we,
- 7:45
we think computers, instead of asking people to adapt to computers, to be like, "Come up to me and type," and whatever, should find ways to try to adapt to circumstances and context of people.
- 7:56
Exactly. So, um, again, here it's like, in this case, it's adapting to where I am by using the pose detection, whether or not I'm actually in the process of talking to it, to decide to update its own world state, use an LLM to actually do that, and then use the LLM to respond using the knowledge of that
- 8:14
world state. And so this is a really simple and, as you can see, kind of hacky demo that is what something you could build today. In theory, you could imagine how this could be, like, a really cool native way to, uh, interact with an LLM on your computer, where you don't have to worry about the input modality
- 8:29
at all. Um, so again, takeaways are consider, like, explicit inputs, what I'm typing, what I'm saying, along with implicit, where I am. Um, there's other things you could do with that, like tone and emotion detection.
- 8:43
Um, you could plug in a whole bunch of different signals that you wanna extract from that.
- 8:46
And you can even imagine if I'm in the frame with Sam, and the agent knows Sam, and she had recently been complaining about me, it should probably not bring that up until I leave the frame. [laughs]
- 8:57
Yep. And as we mentioned that, um, using it as a reasoning engine. And then next one. Cool. And yeah, and then we're adapting. So we wanna get to the futuristic stuff.
- 9:08
Um, Jason has been spending a lot of time imagining this, so he's gonna walk you through a few things that might exist shortly in the near future when new hardware comes out.
- 9:16
So, um, when we think future, we still think the sensing-thinking-react loop will, will take place. To
- 9:25
preface all of this, these are my personal speculative fictions, not representative of anything that I think might actually happen. Um, and this is a very conservative view of the next 1 to 12 months maybe.
- 9:38
So it's not a true future, future, AGI, God worshiping type situation. Um, so let's start with, uh, what I call, like, a social interface. Um, we're all really excited about, you know, certain headsets being released at certain points.
- 9:52
Um, and [laughs] one thing that I think is interesting about some headsets is they have sensors, and they have hand tracking and eye tracking. Um, and just like how I'm being expressive right now, maybe there comes a day where I can be such with a computer that sort of lives with me.
- 10:08
So here are my... Here I am in my apartment minding my own business. Um,
- 10:14
and my ex decides to, uh, FaceTime me. Um- [laughs]
- 10:21
And now I have to decline the call. You know, with hist- historically, with deterministic interfaces, um, I would've had to, like, find the hang up button or go like, "Hey, Alexa, decline call."
- 10:33
Like, thinking commands, thinking computer speak. But, like, as a person, I can be like, "Fuck off." You know? I can be like, "I'm busy." I can be like, [coughs] "I'm sick."
- 10:41
You know? Like, all of this stuff- [laughs] ... the computer should be able to interpret for me and, you know, send, send, uh, what's his name again? Tos- Toxic Trashiest, whatever, on his merry way.
- 10:51
Um, so explicit social gestures can be a great way to determine user, user intent, like the way I just showed now. Um, but we should also consider interpreting implicit gestures.
- 11:00
If I give a really fast gesture with a slow gesture, my mood, my tone, how far away I am. Um, but we should also be conscious of social cultural norms, different gestures mean different things in different societies.
- 11:11
And it might mean, you know, as you scale your application or hardware to different locales, this is something that you should pay attention to.
- 11:19
Now, I wanna move on to talk about what I call new physics, and this part is super fun. Um, this demo is based on, um, a little, uh, I think on iPad, which, you know, has over five daily active users in the world.
- 11:32
It's very popular. [laughs] Um, and here I'm imagining like, okay, Midjourney, if I was the founder of Midjourney, I would be putting all my resources and making some sort of, uh, Midjourney canvas app for iPad.
- 11:44
So in this one, I've asked Midjourney to create a Balenciaga Naruto, which now I'm realizing kind of looks like me. Um, [laughs] [laughs]
- 11:54
So let's think about the iPad. It's like this big slab that you can, like, touch and fiddle with, right? So what do I wanna do? Okay, I wanna, like, edit this photo.
- 12:02
Um, but first I need to make space. How do I do that? Well, very easy. You just, you know, um, you can just zoom out, and now you have extra space.
- 12:09
Very obvious. We do this all the time. Um, I kinda think my cat would look really good in that outfit, so I kind of wanna find a way to do that here.
- 12:19
Let me just ask this AI real quick. Um, "Hey, random AI, send me pictures of my cat." And You know, the AI knows me and has context and gives me pictures of my cat, and then what do I do here?
- 12:32
Well, why can't we just take one of the photos and
- 12:40
sort of just blend them with the other? Um, and the metaphor you're seeing here as you sort of work with these photos, they start glowing when you pick them up.
- 12:49
And what does light h- You guys know the Pink Floyd, uh, Dark Side of the Moon album cover? Like, we're really familiar with the idea that light can sort of, uh, provide different colors and, and sort of concentrate back into one form, and we're leaning into that metaphor here implicitly.
- 13:04
Um, and so it's now created something that looks 50% human, 50% cat, 100% cringe. I don't really like this. [laughs] How do we remix this? What is a gesture? What is a thing we do in real life that's remixing?
- 13:15
Um, for me, it's a margarita, and for Sam, it's her morning Huel. [laughs] We shake a blender bottle. So why, why can't we work with intelligent materials the same way that we work with real materials?
- 13:29
Ooh.
- 13:29
And just blend it up? This is totally doable right now. David, why aren't you building this? If you don't build this, I'm gonna build this. It's fine. [laughs] Um, so, you know, here the metaphor is like, what we're trying to say is, you know, think about familiar universal metaphors like physics, like light, like metabolic, like squishy, like fog,
- 13:46
whatever, because, you know, if you're designing an iPhone, you have to be very cognizant of the qualities of alu- aluminum and titanium to make an iPhone. But generative intelligence is a probabilistic material that's sort of more fluid.
- 13:59
Maybe it's fog. Maybe it's mercury. Um, and, you know, for this reason, maybe metaphors that are really rigid, like wood or paper or metal, aren't the right metaphors to use for some of these experiences.
- 14:14
Um, so finally, I wanna walk you through an experience that's inherently mixed modal, um, slash mixed reality. Um, let's imagine for a second there's a piece of hardware coming out that's a wearable that has a camera on it and has a microphone and it can maybe project things.
- 14:31
I don't know if such a thing will ever exist, but let's imagine for a second it does. Um- [coughs]
- 14:37
... I'm sort of browsing this book, this Beyoncé tour book, and I see these images that I find really inspiring. Um, what I'm trying to do here is what if I could just point at something on my desk and say, like, "This is cool," and have the sort of device, uh, pick up on that and, and, and
- 14:56
indicate that it's heard me, and it's gonna do something by sort of projection mapping this sort of feedback. Um, this is... You know, this demo doesn't really have sound, but the way this would work is ideally a combination of voice and gesture at the same time.
- 15:09
Um, and obviously, this gesture is really easy to make mistakes with, so any time you work with probabilistic materials, you wanna provide a graceful way out. So in this case, I've accidentally tapped this photo.
- 15:21
Why can't I just flick it away like dust and be like, "That, that's wrong"? I don't wanna press an undo button. I don't wanna press Command V. I just wanna flick it away.
- 15:29
Um, really leaning into the physics of it. Um, so now that I've found two pieces, I'm kind of like, okay, I wanna send this to two of my friends who...
- 15:38
Hmm, there was a friend who I said I would do Halloween with, but I can't really remember their name. Um, what do I do here? I should ask AI.
- 15:47
I should be like, "Who was that friend I said I'd spend Halloween with?" And you'll notice here that, like, we're imagining sort of projection-mapped UI pieces that can work with the context of the world you're in right now, such that you don't have to go fish out a phone or use cumbersome voice commands.
- 16:05
Um, it just all sorts of naturally melds in with the world. Um,
- 16:11
and, you know, crucially, I think one point we want to make is voice in doesn't need to mean voice out, gesture in doesn't need to mean gesture out, and y- visual UI in does not need to mean visual UI out.
- 16:21
We can mix these modalities in real time for whatever makes sense in whatever context you're in.
- 16:27
So given that interactions that require multiple simultaneous inputs are now possible, um, it's our job as designers and developers to sort of think on behalf of the user and think, "When's, what's the appropriate output given the current context?"
- 16:40
And be smart about it. Um, yeah.
- 16:43
Yeah, so again, the takeaways, as we mentioned, it's this idea of we have a lot of sensors and, and contextual modalities available to us as ingredients even today. There'll be more tomorrow, as you kinda saw with these upcoming- [coughs] ...
- 16:56
uh, potential hardware releases. Um, but even now with a laptop with things like typing speed, with things like, uh, the tone of voice, there's a lot of ways that you could gather context and extract signals from it.
- 17:08
You could choose to process it in a variety of different ways. And so all of that can h- now be passed to an LLM and used in a reasoning layer, which decides how, um, both to respond in words and also how to present that information.
- 17:22
Um, and so basically, everything can now be an input, and your output could be everywhere and have every format, um, at the same time. One might say everything everywhere all at once. [laughs]
- 17:36
Well, you wanna be intentional with it. You know? You... If someone wants to generate a photo on their Apple Watch, you're like, "What? Why? Why?" Like, n- no, use your freaking phone.
- 17:44
Jesus. Um, anyway, and the last thing we'll say is, um, probabilistic interfaces are hard because y- they, they have lots of different outputs. So a really great way to sort of ground these interfaces is to lean into familiar metaphors, whether they are from nature, from physics, or even from human-made tools and materials, like buttons for now.
- 18:02
Um, and, you know, social norms is also a material that we work with, right? So your banking AI agent probably shouldn't be able to have a deep philosophical chat with you.
- 18:14
That just socially doesn't make sense to-
- 18:16
That would kill you.
- 18:17
Exactly. Um, but on the same note, we've, we've, we've related all of these interfaces to what humans c- perceive and experience now, but what might a truly intelligent interface look like in the future, where if we think we are...
- 18:32
where we are right now is skeuomorphism, what is the abstraction layer above that? And that's kind of for us to figure out. Um, so
- 18:42
with that, um-
- 18:44
Yeah
- 18:44
... I think that's all.
- 18:45
Thank you. [laughs] [laughs] [audience cheering] [upbeat music]