Bounded Autonomy: Between Free Will and Determinism

[upbeat music] Uh, thanks so much for coming, everyone.

Uh, happy Friday. Um, [chuckles] my talk today is, uh, it's called Bounded Autonomy: Um, Between Free Will and Determinism, and it's sort of about changing, uh, the way we think about, uh, our interactions with large language models.

This talk could've been, uh, called a number of different things. Uh, I toyed with the idea of Between Automation and Customization, Between Oversight and Agency, and Between Possibilities and Constraints.

But what it really is is, uh, conventional wisdom for unconventional times, and it's based off some of my personal experience in, uh, designing, uh, agents within industry. Uh, it's for people actively experimenting with agents.

Uh, it's both for beginners who are overwhelmed by the pace of change and experts stuck in a rut in need of a new perspective. What it is not going to be is overly technical, definitive, or prescriptive.

These are, like, my ideas, and I don't expect you to follow them if you're not that interested. [chuckles]

Um, great. I'm, uh, Angus. I'm an AI director at OLIVER. Uh, we're a startup. Uh, we've been in the advertising industry for a few years, and then we switched into almost fully gen AI now.

Um, we've got three thousand staff across forty-six countries. Uh, you probably haven't heard of us, [chuckles] but I guarantee you've definitely seen our work. Uh, you probably didn't realize it.

You probably didn't notice. I didn't when I joined the company, um, because you didn't know it was AI when you saw it. Uh, this is some of our work we've done for Johnnie Walker.

Um, we generate around four thousand assets a day for more than two hundred brands, many of which you've probably interacted with today, uh, maybe even this morning or this afternoon.

Uh, unlike other sort of, uh, gen AI content agencies, we actually put quite a lot of media spend behind these assets. Uh, you know, anything ranging from, uh, twenty grand to a few million, uh, and that enables us to measure these assets' performance in the wild.

This gives us quite a good feedback loop. Uh, we've got huge volumes of data, um, and it allows us for much faster iteration and a more deeper understanding of what actually works.

Right. Who knows what an ad agency looks like? [chuckles]

Anyone? Probably not. I didn't think so. Uh, yeah, you probably think it's maybe a bit like Mad Men. Uh, but essentially, there's three parts of an ad agency. There's the accounts department, there's the creative department, and the strategy department.

Accounts typically has made up fifty percent of the agency before. Now, we have twenty percent creative and twenty percent strat. Uh, accounts manages the client relationships and keeps the project on track.

Uh, creative turns the ideas into compelling ads and content. That's historically been the core of the agency. And then strategy is layered on creative. So that's how do we get to those images?

How do we define the insight, the audience, and the direction behind the work? Uh,

creative and strategy have previously been knowledge work, but they're now increasingly agentic. Uh, we don't just do image generation. We do, uh, ideation. We do copywriting. We do content production.

And those are all done with, uh, forms of agents, right? Uh, I'm on the strategy side. I've been for a long time. And we do, uh, audience insight, trends analysis, competitor analysis, and performance optimization, all with different agents, 'cause we're predominantly customer-facing.

So we operate in quite a high, fast-paced, and high-risk environment. Um, and when we scale these images, it can be just as useful... It can be just as negative for a brand if there's poor reception of these images as if you did a good job.

Uh, why do we use agents? We use them primarily for speed and secondarily for scale. Agents allow us to move faster and be more reactive. Um, for creative teams, this allows us to generate content at speed, and this is especially important for iteration and testing.

For strategy teams, um, agents allow us to scale our research, so we can get much closer to the consumer, and we can, uh, eventually, like, convert them, uh, more easily down the line.

So typically, something we might do is campaign personalization or territory personalization. So that would be, we'll have ideal personas, and then we'll, we'll try and deep research each of those personas or each of those territories.

Uh, a good example is, you know, we do advertising in multiple cities around the world. We might want to localize that for New York. We might want to localize that for Miami.

And in general, the goal is to do much more with much less and create more effective advertising. Uh, so my first piece of advice here today is slow down.

Uh, I think AI is moving very fast. There's quite a blink-and-you'll-miss-it mentality. Uh, everything seems to be coming and going very quickly. There's a lot of change. Uh,

I've seen a lot of tools come and go, uh, in the last few years. But if you did blink and you did miss it, was it really that important?

Probably not. Uh, the actual core of, uh, LLMs hasn't changed, I mean, let's say, at least since the nineteen nineties. Um, Emma Strubell would argue even further back. Uh, but no matter how advanced, uh, LLMs seem, today's large language models don't actually understand the data they're presented with.

And we know this 'cause they've still got several clear limitations, right? So first one is data efficiency. So humans learn from very few examples, whereas models need massive datasets, uh, in order to, uh, come up with relatively simple conclusions.

Uh, I mean, especially, it, it's especially evident when it comes to learning. So models don't continuously learn without forgetting in the way that humans do. It's a closed box, as we'll see in a second.

One could argue that recent gains come more from... less from material breakthroughs and more from sort of brute force model improvements. You just heard in the last presentation the amount of compute required to generate those images, four hundred marathons.

That's maybe too much, [chuckles] uh, to be using. Um, so one of the... This is a famous diagram. This is The Bitter Lesson. Um, [chuckles]

um, so actually, what we end up doing is we end up creating band-aids to go around these model constraints, right? Uh, and most of our tools are band-aids, and you could argue even the way that models themselves are being trained is a form of band-aid, right?

And one of the ways to spot these band-aids is that they're temporary. They're a quick fix. They're not a long-term solution. They're quite superficial. They often mask the symptoms rather than fixing the problem, and they're often inadequate, and they don't fully fix the issues.

Um, so, uh, this is how I like to think of LLMs. It's very simplistic, but, uh, it's just a closed box with knowledge inside. Um, I prefer to think of it as like a flexible database, uh, capable of doing semantic math and anything else.

So it's fully closed. I don't expect any form of, like, emergence or the model to like, actually learn anything. Um, this is quite evident when we're trying to like, especially in advertising, when we use trends.

So you probably come up against, uh, it not recognizing certain models. But for us, the biggest problem is trend identification. If it's really new, the model won't recognize it.

Um, most recent advances in agentic capabilities have been largely due to increased context window.

This is especially, uh, true with like long running agents, because the longer context windows allows them to do longer running tasks. Uh, you've got things it needs to do that, history of actions, tool outputs, structure over time, its goal-goals and plans.

It has to be organized, it has to know what it's doing, and it has to be able to store and retrieve information in the short term. So without large context, uh, the system forgets mid task.

It can't work on long, complex, multi-step workflows. And you may have heard of this famous incident of it deleting a lot of emails 'cause the, uh, context ran out.

Uh, this is a model-sized context difference, uh, GPT-2 versus Gemini 3.5 Pro. Does anyone remember 512 context windows? They're very small.

Um, context windows keep getting larger, but they'll never be enough. Um,

the total knowledge in the world keeps doubling about every twelve hours, so we're always gonna want more, uh, however big they get, it's, uh, it's not gonna be enough.

I think that's the sort of clear, clear problem. So how does that work from a sort of developer's perspective? Um, actually, I find context constrains the models as much as guardrails.

So context is sort of like a soft constraint rather than a hard constraint. So you can, you can feed, feed stuff in, and you can move stuff around and shape indirectly how the model is performing.

So like a practical everyday example of this is not giving the model access to the Internet, uh, and instead giving it high quality documentation, you'll get much better results.

Uh, and, and typically when we do this, we find that models are really bad at, uh, spotting promotional content. So when we're doing advertising, we're looking up, uh, competitors.

It'll, uh, soak up all of the, uh, information they wrote themselves rather than like consumer information, which is ultimately like what we're after. Um, and they're very susceptible to SEO.

Uh, I think as soon as you take the knowledge out of the model and you give it another tool, it's quite, like, limited by the way that it uses that tool.

Um, so in the past, this was originally what we used to do. Uh, so when we had really small context windows, we used to do, uh, I used to do TDIF for cluster labeling.

So you would take the, uh, most frequent words within the cluster. You cluster the text corpus, you'd do the most frequent words within the cluster, cluster, and then you'd label that.

Uh, you'd also do like top, top K and that sort of thing. Uh, whereas now, you know, context assembly is much more dynamic, right? Um, and we've gone from like context, uh, la-lack of context to like too much context, right?

So now I think we need to sort of think more about what we can exclude from that. And the challenge is no longer getting context in, but to a certain extent, keeping the noise out, right?

Uh, and con-con-constraints actually create creativity, right? Abundance stops you being scrappy. So if suddenly, if suddenly progress stops tomorrow, how would you make the most of what you have?

Uh, I think it's important to set sort of self, self-imposed constraints. Everyone knows that you shouldn't use, like, the full context window, but maybe how little of the context window can you use and still get the task done is a more interesting question.

I'm not a token billionaire yet. I know there are some people probably in this room. Um, this is again, parallels with early computing, lack of compute, very difficult times.

Uh, that's from the Model Railroad Club, that was from yesterday. Um, but great things come out of constraints and limitations, right? So if you can build Space War! with only four thousand words, that's a pretty big deal.

Uh, and I don't know if you've ever, uh, seen the developers of Grand Tha-- uh, not, of Crash Bandicoot, uh, but they talk a lot about how they were like using the memory function in the PS2 to get like massive improvements, right?

Uh, so if they can do it, I imagine you probably can. Um, other things to try, maybe using an old... This is just experimental. I'm not su-suggesting you do this in production.

Um, using an older, smaller version, version of a model or harness, um, this can help you understand, like connect with the model maybe a bit more. Um, building your own harness, building your own memory and compaction.

Uh, and also I think like pre-processing and archiving stuff is really important. So how do file systems work? How do knowl-- how do you negotiate knowledge graphs, that sort of thing.

And this will overall, it'll ap-improve your ability to prompt and control the model. You'll be much closer to it, um, and you'll have better fundamentals and better best, best practices.

Uh, and in general, you just have an improved understanding of the data that you're working with. Uh, you never know what's gonna come in handy later. Uh, this is, uh, uh, Rosenblatt actually pulling out, uh, wires in the perceptron, uh, and that's later, you know, formed the basis of what became, uh, dropout.

Uh, the next one is keep it simple. Uh, have you ever built anything and then realized that the model could just do it better on its own? Probably. Got a lot of nods there.

Happened to me recently. I was trying to do my CV. Uh, what would win? [laughs] Um, this was my very complex, uh, CV application. Uh, did not work. Uh, well, it did work, but it didn't work as well as four simple letters.

Uh, those four simple letters were, uh, HTML. And, um, yeah, it was... I was pretty, uh, blown away by this. Um, so that was over a 10X improvement. I would say it's probably a hun- 100X improvement.

Um, so I think just because you have the power of the gods, that doesn't mean you should use it. Uh, models are like naturally verbose, and they tend towards complexity.

Uh, like they are gonna suggest the most complicated solution. So don't waste your time. Don't make, uh, don't waste tokens, and don't make extra work for yourself. Uh, your ideas will collide with reality pretty fast.

Um, so no matter... So what matters the most is building a simple version that works. And I think in, when we're talking about using agents, we talk about shortening that feedback loop, but maybe you should shorten your feedback loop with reality when you're building products.

Um, this is, I think, probably the most interesting bit. Um, so AI at its core is just translation. So this is the Attention Is All You Need paper. It was, um, done on English to French initially.

Uh, and then I think more interesting is the idea of being able to translate text into images, images into audio, uh, audio into video, right?

Uh, and I think there's an argument that, uh, knowledge production is just in itself summarization, right? Uh, like, in this talk, I'm trying to summarize my experiences of the last few years, and we're compacting that down into, uh, like, knowledge or, like, good knowledge, right?

Um, so different types of data can be converted into common internal representations and then transformed into something else. Like, you've got an unstructured input, and you turn it into an unstructured output.

Um, you've got a structured input, and you turn that into an, uh, an unstructured output. Uh, this is like, I think, I guess, what MCP is in, uh, model handoffs, uh, for long-running agents.

This is, this is all, like, a very similar idea, right? You've got something structured on one side, something very unstructured on the other. Uh, and I think these two things can sort of coexist.

And, and if you can manipulate something through a representation space, then that means that the structure of that, uh, data is not an inherent, inherent property of that object, right?

So it's actually more of a property of the representation of the observer. It's like, what do I want to see this piece of content in what format, right? And sometimes, I guess, maybe the easiest way of thinking about it is with slides.

What can you turn into a diagram, and what do you want written? What do you want voiced over? Uh, and I think it's super interesting. So this has a lot of relevant implications when you're building, right?

Um, this means, you know, ideally, you should use multiple representation structures. So you might wanna use, uh, markdown, um, for human-readable hierarchy and authoring. You might wanna use graph relationships, uh, and references.

Um, you might wanna use clustering as well if you're dealing with, like, large bodies of text or maybe more unstructured bodies of text. And you might wanna use folders for stuff you need to retrieve really, really fast.

Um, you could also use timelines if that's, uh, relevant to your task. Um, and finally, [laughs]

uh, there were some... a few more, uh, practical bits of advice. Um, but finally, I think it's so important just to have fun and experiment, right? I think so much of what we do is, uh, for work.

But actually, a lot of this stuff you could only learn through, like, thoughtful play and experimentation. Uh, I go to a lot of hackathons now, more than I ever did before, and I, I just don't see...

I'm not allowed to experiment with this stuff in the way that I would, uh, in, in my role, so a lot of that then helps me, uh, even more.

Uh, I've got one minute left. [laughs] Uh, it's really interesting. I will say one more thing. So I was gonna talk about how agents fit into the workplace and whether you should use more structured agents or less structured agents.

Uh, and I will just say, so I think if you look back to Adam Smith and his pin factory, like, capitalism naturally breaks tasks down into small, easily repeatable chunks, and often we have found workflows to be far more effective.

Um, I was gonna say, um, that I would sh- like, one of the other things, other things I was gonna say is don't automate a job unless you can do it yourself.

Uh, this is my old job. This was social, social media intelligence. Uh, and what you can see here is, like, a practical report. This was done by an LLM.

It clustered the data. It did everything. And this is typically the sort of information that we would want from an advertising agency. Because we're a, a AI engineer, I've done it for, uh, OpenClaw just on the last year's sort of data.

And th- this is all the sort of insight that we would get. This is fifty thousand tweets clustered and then organized into, um, essentially strategies. So we can create this almost instantly for our creative teams or our strategic teams.

Uh, then it's got standout tweets, and it's broken it all down for you. Uh, and I think that's my time there, so thank you very much. [audience clapping] [upbeat music]