Climbing the Ladder of Abstraction

[upbeat music] I wanna start with a question.

Does anybody remember what accounting looked like in the early 1900s?

Yeah, me neither. But from what I gather, it was super frustrating- I'm having some trouble with the connection. Please try again in a moment ... and tedious, and it involved a lot of, like, writing letters and numbers, uh, annotating in margins, performing calculations by hand.

Um, you can probably look at these pages and sense how frustrating it is by looking at how many things are crossed out and all the ink blots on the page.

So thankfully, this isn't how the job's done these days. So in 1979, VisiCalc totally changed the game, and this was the first spreadsheet for personal computers.

It became an essential tool for accountants, at least until Lotus 1-2-3 was launched, uh, four years later. And the innovation here wasn't performing the calculations automatically. We already had calculators and computers to do that for us.

But instead, the innovation was having the structured interface that stacked those automatic calculations together into formulas, so that when you change the value of a cell or you add a row to your spreadsheet, uh, all of the spreadsheet numbers would be updated live.

So instead of spending all day doing calculations or manually updating the rows and columns, accountants could now spend more time, uh, worrying about the actual numbers.

Okay. Fortunately or unfortunately, this isn't a spreadsheet conference, so let's get back to talking about AI.

So one of the things I'm most interested in is what are the best ways to combine our new AI superpowers with the interfaces that we use today, or more importantly, the interfaces that we wanna use tomorrow.

So often when people talk about building interfaces with AI, they refer to these two distinct goals, whether it's automation or augmentation. In essence, automation takes rote tasks and does them for the user, which is really great for anything that's super tedious or boring, like copy and pasting data into a table or doing calculations by hand.

And in contrast, augmentation gives the user a new ability or it improves their existing abilities, which is awesome for things that are creative or nuanced, things we don't really trust models with yet, uh, like analyzing data.

And I think this contrast often ignores how related these two concepts really are. Um, automation has become a bit of a buzzword, um, or a trigger word where people are worried about their s- jobs being automated.

And I think this is a very valid concern, and I kind of wanna reframe this dichotomy. So, uh, instead, I think augmentation is composed of smaller automations. If our end goal is to automate tasks or jobs, we'll still need to automate parts of them.

So for example, if the end goal is analyzing data, automating the smaller tasks like aggregating the data into a table or generating visualizations from that table are gonna help focus on your end goal, which is answering the question that motivated the data collection in the first place.

So if we go back to our spreadsheet example, we can think of each cell, the calculations that, uh, that create them as having been automated away. And no one really thinks of spreadsheets as taking people's jobs.

Instead, uh, Excel, what I'm showing here, which is kinda like the current king of spreadsheets, uh, is an essential tool for people who interact with things like financial data.

If we automate these parts behind the scenes, uh, that's the first step towards achieving the goal of augmenting, uh, working with data. So in the future, we can easily imagine having this table aggregated automatically or writing the formulas for us, and having all this work done helps augment us in our greater goal of analyzing and understanding the

data. This is one of the reasons why you might hear me say some things like chatbots aren't necessarily the future. Um, I think that these flexible general tools like calculators and chatbots are wonderful, but then adding the structured interface around them makes them so much more powerful for a ton of different use cases.

What we want is something where the technology behind chatbots is embedded into the interfaces where we're still driving, but the model's automating away the smaller tasks that we find so frustrating.

So what might these interfaces look like? Before answering that question, I wanna introduce one more concept: the ladder of abstraction. So the basic idea here is that the exact same object can be represented at many different levels of detail.

So I think maps are a good example of this.

Um, we take this interface for granted, but Google Maps and other digital maps are incredibly compelling interfaces. They're so well designed, um, and they help represent different tasks involving navigation and localization at different scales.

So here we are at the most zoomed in scale, and we can see all of the different structures within the Monterey Bay Aquarium. We can see individual buildings, the names, the icons for them, maybe routes between the buildings, and this is great for navigat-navigating around the aquarium, but maybe not so great for getting to the aquarium.

As we zoom out, all of these buildings get smaller because they're further away, but that's not the only thing that happens. So at these more zoomed out levels, uh, Google Maps actually starts hiding information.

So I can't see the buildings inside of the aquarium anymore or their icons or names, but instead I can see city streets and, uh, different restaurants, and this will support a different te- set of tasks, like finding a restaurant or a destination and getting to that place

Zooming out even further, we lose those city streets and stores, and instead, we look at highways and terrain. And again, we have a different task here. This level supports longer range travel, getting to and from Monterey.

And then if we go all the way out, we're mostly looking at the shape of states or countries. So if we tried to keep all of that information at higher zoom levels, it would be completely incomprehensible.

There's really only so much information we can fit in our brains and so many pixels on a screen, and most of that detail isn't relevant for the task we're trying to do anyway.

So you could wonder, can we use AI to bring these kinds of principles to other types of interfaces?

For example, what would happen if I zoomed out on a book? What would that even look like?

Typically, when we, we read a book, we're looking at every single word, but that's not the only level we think about. When remembering books we've read in the past or summarizing a book for a friend, we're more concerned with overall topics and plots than specific wording.

And now that we have access to language models, which are amazing at summarizing and transforming text, how can we use them to change the way we read and write?

So here's a quick demo I put together of the first five chapters of Peter Pan,

and [laughs] there's no tricks here. I'm just scrolling through the first chapter. So if we take this and we use an LLM to zoom out, we can see each paragraph change to a one-sentence summary.

And we have a, a mini map to the right, and you can kind of see, uh, how much fewer words there are in the page and how much more quickly I could read this.

If we zoom out another level, we can see summaries of, say, 10 paragraphs at once. And again, you can see in the mini map, we have way less text to read.

And then finally, at that highest zoom level, we've reduced each chapter in one sentence, and here we can fit five chapters on one page.

So if I were writing Peter Pan and I wanted to do something like tweak the pacing or modify the plot structure, viewing the text at this highest zoom level, editing it, and then zooming back in to see how that changed the raw text would be a much nicer workflow than keeping all the parts in your head as

you change it word by word. So another way to think about a book at a high level is with a story arc, um, and this describes the mood mapped over an entire story.

Uh, you might be familiar with Kurt Vonnegut's graphical representation of the most common story arcs. For example, we have Man in a Hole, uh, where the main character gets in trouble, gets out of it, and ends up better for the experience, um, which you'll see in stories like The Hobbit or The Wizard of Oz or Alice in

Wonderland. What if we could, uh, take the semantic value of all the sections in a book and plot that on a, on a graph? And then if we wanted to edit the story, we could go ahead and tweak parts of that graph and see how the raw texts change.

I mainly highlight this because I'm super excited to see how we use AI to innovate on writing tools within the next few years.

But first, let's combine the two concepts. Um, so the first concept is augmentation as stacked automations, and the second concept is traversing the ladder of abstraction for different tasks.

How might this look in a more general product? So I'm on the design team of a startup here in SF named Adept, and at Adept, we're focused on training AI to use software, read screens, and take actions the way humans do.

And our end goal is to make knowledge work easier, so any work on a computer.

So after speaking with a lot of people about what they do day to day at their jobs, we found that much of knowledge work involves getting information, transforming or reasoning about it, and then acting on that information.

So given this really common workflow, one of the things we've been thinking about is, what might it mean to zoom out on any piece of information? So we have some sketches where we're exploring what that might feel like or what it might enable us to do.

I thought it'd be really fun to share one of those with y'all today.

All right, so completely hypothetical situation. Let's say I was going to an awesome conference in San Francisco. What I would do first is I would go to Airbnb. I'd find listings near the venue.

I'd click into the detail page of one of the listings, and there's all this generic information that should work for everybody. But I have specific criteria that'll help me decide whether or not it's the right thing to book.

So I'm gonna be l- digging through this page, looking for things like, how close is it to the venue? Is there a coffee maker? Does it have good Wi-Fi?

That kind of thing. This kind of decision would be much easier if I could zoom out just a little, get rid of all the branding and standard information that isn't really important to me right now, and focus on my deciding factors.

So to start, I can see the name of the listing, maybe the rating, a quick summary, and the total price. And this is all pretty generic so far, but I know this conference is at the esteemed Hotel Nikko.

And I'm typically gonna be looking at a map to find places near that venue. But if I could just, uh, extract the, the walking minutes to the hotel and put that right on the page, that'd be really helpful.

And maybe if that's a little bit far, I can figure out what is the closest BART station, uh, to the listing and then add the walk to BART there as well as a backup way to get to the hotel.

Another thing that's really important to me is the Wi-Fi speed. Um, I know I'm going to be working on my talk the night before, true story [laughs], so I'm gonna need really fast internet.

So I can use AI to pull out the relevant reviews and summarize them as positive or negative to really quickly judge whether the Wi-Fi is gonna work or not.

Additionally, usually Airbnb has like fifty vanity photos for any given listing, and I really just want one photo of the bedroom or living room or kitchen. Um, so if I could just pull those out and put them on the page, that would help me a lot.

And then most importantly, at this higher zoom level, preserving the ability to act on this information. So directly from this page, I can go ahead and reserve this listing or send a message to the host without going back to Airbnb.

That would be really helpful and keep me in control.

And I never really know whether staying at an Airbnb or a hotel is gonna be a better deal, so typically, I'll also look at hotel listings. And it's pretty great to be able to see that same elevated view no matter which site I'm looking at.

Additionally, if I'm gonna compare the hotel with the Airbnb listing, have these, having these similar views side by side is gonna give me a really easy comparison between the two of them.

But what if I wanted to look at fifty listings? Comparing fifty of these individual views would still be a lot of work.

Zooming out a level, I can look at a spreadsheet for all fifty listings with my deciding factors all laid out for easy comparison. So I can quickly eyeball the distribution for total price, uh, get a sense of how quick the walks are for each of the listings, how many positive Wi-Fi reviews there are.

Importantly, I can still take action on this level. So if I see a listing that's a clear winner, I can go ahead and book it right here instead of going back to Airbnb or Hotels.com.

But sometimes the decision isn't so clear-cut or it's more multifaceted than, uh, having the cheapest or the closest listing.

So if I zoom out another level, each listing has been abstracted into a circle on a scatter plot, and these are colored by the Wi-Fi reviews. You can see the, the cheapest listings on the left of this plot with the most expensive ones on the right, and the closest ones to the hotel near the bottom.

And I can pretty quickly see that there's this cluster of listings, uh, that are the cheapest and the closest, and they also have good Wi-Fi. But I just realized my flight gets in at nine AM.

But thankfully, I can still initiate actions from this view. So I can circle these, send a message to all the listings when this-- within this cluster, asking about their policy on early check-ins, and whichever one responds first that I can check in at eleven AM, I'm gonna go ahead and book.

All right, so as we saw, there's so many tasks that are best suited by a specific zoom level.

And what we're currently doing is we're manually abstracting that information in our heads. So in this example, digging through fifty different Airbnb or hotel listings, uh, we're keeping all of the previous ones in our heads to try to find the best one, and this takes a lot of mental energy.

I know I titled my talk Climbing the Ladder of Abstraction. [laughs] That was partially to not rip off Bret Victor, who has a talk titled Up and Down the Ladder of Abstraction.

It's a great talk. But I'm not trying to argue that higher levels are better. Instead, what I'm trying to argue is that we can use AI to generate these different levels, glue them together, and make it easy to move between them.

And I think this could completely change the way that we work with information.

So this is one of the many great explorations we're doing at Adept to make all computer work easier. We're gonna have a lot more to share in the near future.

Stay tuned. Um, and then to sum up, there's three things that I would love for you to take away from this talk. The first is augmenting tasks are gonna look a lot like automating smaller, tedious parts.

No one's thinking of spreadsheets as taking people's jobs, and digital spreadsheets is exactly the kind of innovation that I wanna see in the next few years. Secondly, we often think about information at different levels of abstraction, and let's make this easier by using AI to generate and act on these different levels.

And then lastly, this is the kind of thinking we're doing at Adept. Uh, feel free to follow us, um, or, uh, follow along, check in, uh, and we're at Adept.ai.

All right, thanks for listening. [audience clapping] [upbeat music]