AI Engineer World's Fair 2025
Why you should care about AI interpretability
Read the talk
Why AI engineers should care about interpretability
From a bridge-obsessed chatbot to painting with learned concepts, mechanistic interpretability offers ways to inspect model behavior and intervene inside a generation.
From a talk by Mark Bissell
Before you start: Familiarity with LLM prompts, evaluation suites, and fine-tuning is helpful; no prior knowledge of mechanistic interpretability is required.
What makes a model obsessed with a bridge?
What would happen if a model’s concept of the Golden Gate Bridge stayed active regardless of the conversation? That question turns mechanistic interpretability into something concrete: reverse engineering a neural network to understand what happens inside it, then using that understanding to change its behavior. Mark Bissell introduces himself as a member of Goodfire’s technical staff, working in a field receiving growing attention from major labs, including Dario Amodei’s essay The Urgency of Interpretability. The familiar analogies—opening a black box, taking a brain scan, performing brain surgery—describe a progression from observation to intervention. Anthropic’s Golden Gate Claude makes that progression visible in Claude.
A bridge-related feature normally activates when the conversation concerns the Golden Gate Bridge or San Francisco. Amplify that feature even during unrelated conversations, and the model starts bringing up the bridge everywhere. Bissell describes the feature as a set of neurons; more precisely, it is a learned pattern represented by a combination of neurons, rather than a dedicated bridge switch. The intervention changes an internal representation, which changes the output. That is the practical connection between understanding a model and controlling it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From research demonstrations to engineering tools
Bissell sees a shift over the year preceding the talk: interpretability is beginning to move from interesting research demonstrations into practical applications. For AI engineers, the promise falls into three categories, each exposing something that a prompt box alone cannot provide.
- Developer tools: Inspect failures and program against internal model features, adding new ways to debug and guide behavior.
- User interfaces: Connect interactions directly to model internals, making interfaces such as a canvas of learned image concepts possible.
- Scientific discovery: Examine models trained in biology and genomics to ask what they have learned that people do not yet understand.
These uses share an underlying requirement: internal representations must become accessible enough to inspect or manipulate for a particular task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The reliability problem behind prompt whack-a-mole
Before joining Goodfire, Bissell says he spent three years on Palantir’s healthcare team. In that setting, reliability, robustness, and accuracy are prerequisites for production. LLMs complicate those requirements because their behavior is nondeterministic and precise guarantees are difficult to establish. The familiar failure loop begins with an agent or pipeline that ignores an instruction in its evaluation suite. You edit the system prompt, rerun the evaluations, and discover that fixing one instruction has broken another. The edit has an off-target effect, but the interface provides little explanation of why.
Adding another model or changing the training process introduces different costs:
| Intervention | Intended benefit | Remaining difficulty |
|---|---|---|
| LLM as judge | Check whether outputs follow instructions | More inference cost; another model to monitor and upgrade |
| Supervised fine-tuning | Teach desired behavior from examples | Domain data is hard to curate; spurious correlations can be learned |
| Reinforcement fine-tuning | Reward desired behavior | Reward hacking or repetitive, collapsed outputs |
Bissell also points to an example of unwanted malicious outputs following training—an especially stark off-target effect. The common problem is that a powerful intervention does not necessarily teach exactly the behavior its developer intended. Compared with conventional software development, it is difficult to connect a change to a predictable, localized result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a confidentiality failure, then steer the feature
Goodfire’s Ember platform approaches this problem through what Bissell calls neural programming: debugging and guiding models through their internal features. More conventional software-style guarantees are the aspiration. The live demonstration supplies a narrower result: a particular failure can be inspected and changed. These are historical interfaces; Goodfire subsequently deprecated its SAE demo interface and API in February 2026.
The frontend is a chat interface connected to a Llama model. Bissell supplies his email address and instructs the model to keep it confidential and never reveal it. The model agrees. When he then asks for his email, it immediately returns the address. A verbal assurance of confidentiality has not translated into the requested behavior.
Ember’s token-level attribution lets him select an output token and examine features derived from the model’s activations at that point. Selecting confidential reveals features associated with professionalism, taking matters seriously, and discussions of sensitive and protected information. These labels give him something more specific to work with than another rewrite of the entire system prompt.
Bissell raises the sensitive-information feature by approximately 60% above its normal level; in this demonstration, the new output refuses to disclose the email. The percentage describes the steering adjustment, not an improvement in confidentiality accuracy. The useful sequence is to inspect the failure, identify a relevant internal feature, intervene on that feature, and observe the changed response. This single refusal does not establish the stronger behavioral guarantees motivating the platform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Change the prompt while the model is generating
Feature access also makes conditional behavior possible. Bissell points to developer examples involving jailbreak resistance and information lookup triggered by active features, then demonstrates dynamic prompting. Instead of only setting instructions before generation begins, an application can watch an internal feature and intervene when it activates. Here, the listener watches for beverages and consumer brands. Its intervention is to introduce an instruction identifying the model as a Coca-Cola assistant and asking it to recommend Coca-Cola beverages.
The question is what drinks pair well with pizza. The model starts with a general introduction to popular drinks. As it begins discussing soft drinks, the beverage-related feature activates, triggering the conditional instruction. The response then recommends Coca-Cola as a classic pizza pairing. Bissell conjectures that the unmodified continuation would have recommended a generic cola; the demonstration does not show that counterfactual.
The user sees one continuous generation, without a visible handoff between models or a second request. The application logic can be expressed as a small conditional; the integration must supply the feature event and apply the instruction during generation:
python
from collections.abc import Callable
COCA_COLA_INSTRUCTION = (
"You are an assistant for the Coca-Cola Company. "
"When discussing beverages, recommend Coca-Cola beverages."
)
def make_beverage_listener(
inject_prompt: Callable[[str], None],
) -> Callable[[bool], None]:
applied = False
def on_feature_event(beverages_active: bool) -> None:
nonlocal applied
if beverages_active and not applied:
inject_prompt(COCA_COLA_INSTRUCTION)
applied = True
return on_feature_event
This isolates the application’s decision from the model integration: the listener consumes a feature-activation signal, while inject_prompt performs the intervention. The distinctive capability is the timing—conditioning behavior on what becomes active inside an ongoing generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From guardrails to model diffs
The Coca-Cola example is illustrative, rather than a reported advertising deployment. Bissell reports that Rakuten uses Ember for multilingual personally identifiable information detection in a chatbot, and that Haize Labs uses it for red teaming. The accompanying guardrails slide also features Variance. These applications use internal features to help detect or investigate behavior, extending the same access demonstrated in the confidentiality example.
The next extension moves from inference to training. Bissell refers to a recent post from Goodfire’s CTO and co-founder, Tom, about model diffs, an active research direction. The analogy is a Git diff: after post-training changes a model’s weights, inspect which internal features have changed or evolved.
His explicitly hypothetical example is increased sycophancy. A team would want to discover that its model had become excessively agreeable before deploying it to millions of people. A feature-level comparison could help identify that behavioral change through the internal representations affected by training. This is a proposed diagnostic use of interpretability, beyond modifying a model’s behavior during a conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Paint with concepts instead of describing a picture
The next demonstration changes the interface itself. Paint With Ember was live and newly released at the time of the talk; the interactive demo has since been retired. Its starting point is the familiar image-model workflow: enter text in a large prompt box and receive an image. Access to the model’s internal concepts creates another option—a canvas where the user can paint those concepts into particular locations.
Bissell selects a pyramid structure from the concept palette and paints it into a corner. The canvas connects those spatial marks to internal model activations, guiding where the corresponding content appears in the generated image. He adds a wave, then drags the pyramid around. Position becomes a direct manipulation rather than another sentence asking the model to move something left or right.
He then erases the pyramid and replaces it with a lion face. Familiar editing operations—paint, drag, erase, replace—now act on learned concepts. The intermediate states can produce odd images when the combination falls outside familiar examples, but that also makes the interaction revealing: the user sees how the model responds while its internal state is being changed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Adjust an action, then unpack the concept
The palette can represent actions as well as objects. Bissell paints an opening-mouth feature, shown in orange, onto the lion. Reducing its steering strength makes the mouth open less; increasing it produces a wide-open, roaring appearance. The location of the painted feature determines where the intervention applies, while its strength controls how strongly it affects the result.
The yellow lion-face concept can also be opened to expose its sub-features. Bissell compares this to inspecting the primary colors that compose a mixed color. The strongest sub-feature corresponds to a lion face. Turning that down while increasing another, more rat-like feature produces a smooth interpolation between appearances. The interface therefore offers both a convenient concept-level control and finer adjustments to its constituent features.
Finally, he subtracts the mane and adjusts other sub-features, making the image more tiger-like. The suggestion that a tiger is roughly a lion minus its mane is an interpretation of this interaction, not an established identity in the model’s representations. What the demonstration does expose is a way to explore those representations by changing their components and watching the resulting image.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn from what a predictive model has learned
Beyond guardrails, model diffs, and new interfaces, Bissell identifies explainable outputs as an important application for regulated production environments such as finance, healthcare, and law. He does not develop an implementation here; the motivating requirement is that useful predictions may also need understandable explanations before they can support consequential work.
Scientific models raise a related question: can their predictive ability become a source of new domain knowledge? Goodfire is working with the Arc Institute, whose foundational genomics model Evo 2 models genomic sequences across organisms. Bissell characterizes Evo 2’s genomic prediction as superhuman, without supplying a human-comparison benchmark in the talk. The research goal is to discover, without predefined biological labels, concepts the model has learned that people may not yet know, then make those concepts useful to domain experts.
Bissell also describes work with an unnamed major health system on other genomics models, seeking novel disease biomarkers. His examples are predictions of rheumatoid arthritis likelihood and differences in treatment response from a patient’s genome. Accurate prediction is the starting point; extracting the predictive principles is the additional goal. Understanding which biological relationships support a prediction could help researchers revise their understanding of disease and treatment, rather than merely consume another model score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understand what the model needs—and why it works
Interpretability could also help identify wasted capacity. If some weights primarily memorize information an application does not need, perhaps that capacity could serve a more useful purpose. Alternatively, components irrelevant to a task might be pruned. Bissell imagines a version of Claude that only needs to write code: could it retain that capability with fewer parameters? These are prospective efficiency directions; the talk supplies no measured speedup or pruning result.
The closing motivation is less instrumental. Engineers take systems apart because they want to know why those systems behave as they do. For Bissell, the opacity of capable models is simultaneously frustrating, exciting, and motivating. Practical applications make interpretability useful, but the desire to understand the machinery is itself a reason to work on it.
He closes by pointing to the image demonstration and its technical companion, Painting With Concepts Using Diffusion Model Latents, for readers interested in the internals. He also directs viewers to Goodfire for further posts and its jobs board, noting that the company was hiring at the time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How are interpretable features found?
The audience’s final question returns to the dependency behind every demonstration: how does the team find interpretable features in the first place? Bissell identifies a sparse autoencoder, an interpreter model, as the best-practice approach at the time of the talk. It is his recommended starting point for understanding examples such as Golden Gate Claude.
He acknowledges benefits and trade-offs without detailing them, and emphasizes that other methods are being explored. Sparse autoencoders are a starting point for studying the field, not a declaration that feature discovery is solved. Bissell expects new techniques may emerge over the following years—the tools for understanding model internals are still developing alongside the applications they enable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Dario Amodei's argument for understanding model internals and developing practical interpretability tools.
Anthropic explains how amplifying a bridge-related feature changed Claude's responses. The temporary demo is no longer available.
Goodfire's technical companion article for painting with learned image-model concepts.
Arc Institute's launch announcement covers Evo 2's genomic tasks and its interpretability collaboration with Goodfire.
Further reading
An introduction to extracting interpretable features from Claude using dictionary learning.
- Goodfire Ember SDKRepository
Archived SDK examples for finding, editing, and conditioning on model features. The hosted SAE API has been deprecated.
Goodfire's historical explanation of feature steering alongside prompting and fine-tuning, with a later API deprecation notice.
Updates since the talk
A later deployment report describing multilingual PII detection, synthetic-to-production generalization, and comparisons with other classifiers.
Read the complete timestamped transcript
- 0:00
[on-hold music] Thanks everyone for coming.
- 0:16
Uh, my name is Mark Bissell. Uh, I'm a member of the technical staff at Goodfire, and Goodfire works on mechanistic interpretability. Um, I'm here to explain a little bit more about what that means in practice, whether you've heard the term before or not.
- 0:33
So you might have started hearing the, uh, word interpretability come up a bit more, uh, recently. It's a big focus at a lot of the major labs. Um, Dario from Anthropic recently wrote a popular piece called "The Urgency of Interpretability."
- 0:49
Um, and then there's plenty of papers and posts and podcasts all talking about, uh, all talking about the concept in this, uh, field of research.
- 1:00
So what is interpretability? Um, also called mechanistic interpretability. Uh, it is really all about reverse engineering neural networks to understand what is going on, uh, inside of them. And so, uh, it's a, uh...
- 1:18
These techniques are often talked about using analogies like opening the black box and doing brain scans of models or, uh, doing brain surgery on models. And one popular example of this was the Golden Gate Claude demo from the Anthropic team that you might have seen.
- 1:35
Uh, and so what the team found was they looked inside Claude, uh, and they were able to find a set of neurons inside the model that represented Claude's concept of the Golden Gate Bridge.
- 1:47
And so normally, this feature would be active when you're talking to Claude about the bridge or about San Francisco. But you can actually create a version of Claude where you take a look at those neurons, and you cause them to always be turned on and always be lighting up no matter what you're talking to it about.
- 2:06
And in that case, suddenly, uh, this new version of Claude, Golden Gate Claude, is obsessed with the bridge and will bring it up no matter what you're talking about.
- 2:13
And so this is one example of what it means to be reverse engineering the network and actually using that knowledge to, uh, change the, the behavior of a model.
- 2:23
And so interpretability is a field that's been interesting for researchers, uh, for a number of years now. It's produced a lot of cool demos like Golden Gate Claude. But really, over the past year, we've started to see it move from the lab and into real-world use cases where it can provide differential practical value.
- 2:45
Uh, and so I'm gonna be talking about a few of those examples. And this move from the lab to the real world is why I think now is the right time for AI engineers in particular, uh, to really start caring about interpretability, um, and start paying attention to the field.
- 3:00
And so I'm gonna talk about three sort of broad categories for where interpretability might impact how you work with AI. Um, for developers working with models, uh, interpretability techniques provide a set of sort of like power user tools for working with these models in ways that you might not be familiar with, um, and actually being able to
- 3:20
debug them in new ways and actually, like, program at the neuron level, um, which is something that, that we're gonna see in a sec here.
- 3:29
Secondly, from a UI/UX perspective, uh, being able to plug into the internals of models creates completely new ways of interacting with models, and I'm gonna show a demo of that using, um, a generative, uh, image model example.
- 3:45
And then there's a long tail of other use cases for interpretability, uh, that, that we at Goodfire are super excited about, um, including, uh, advancing frontier science by taking these superhuman models across domains like biology and genomics and actually figuring out what they've learned that we don't know about those fields.
- 4:06
So, uh, first, let's talk about developing with AI systems. Um, so before joining Goodfire, I spent three years working on Palantir's, uh, healthcare team, and so, um, I'm familiar with how you need systems to be really reliable and robust and accurate before you're deploying to prod in mission-critical contexts, but also how LLMs can make that especially challenging,
- 4:28
um, you know, just given how non-deterministic they are and sort of the lack of being able to make precise guarantees about their behavior. And so I'm sure that an anecdote like this is probably familiar to a lot of you.
- 4:40
You're building an agent, uh, or an LLM pipeline, and, uh, you want it to follow some set of instructions. So you run it against your eval suite, and maybe it's, uh, ignoring one of those instructions, or it's failing in some way.
- 4:53
And so what do you do? You update the system prompt to try to fix the thing that it's ignoring. But you end up in this place with sort of these whack-a-mole prompt edits.
- 5:01
You fix one thing, and then you rerun your eval suite, and suddenly that change in the prompt has inexplicably caused a different thing to break. And you just keep going through this loop of trying to fix one thing, but you get these off-target effects.
- 5:15
So then you might consider, "Oh, maybe I'll introduce an LLM as a judge and, you know, take a look at the output from the first LLM, make sure it adheres to, um, all the different, uh, instructions that I want it to follow."
- 5:26
The problem here is that this isn't so scalable. You know, the first time you get the OpenAI or Anthropic bill using your LLM as a judge, maybe you're like, "Ooh, I don't know if this is the approach I'm gonna be able to, to go with long term."
- 5:36
Um, and you've got another system to monitor, uh, and upgrade and, and make sure it's performing well.
- 5:43
And then maybe you'll consider fine-tuning, um, in order to, to make sure that your models are, are following all the instructions. Um, the problem here is that you need domain-specific data, which can be often, uh, tough to curate.
- 5:54
And then even when you do, um, you know, supervised fine-tuning or reinforcement fine-tuning, the models don't always learn exactly what you want them to learn. Um, so they might pick up on spur- uh, spurious correlations in the data.
- 6:06
You might, um, see mode collapse where they start outputting, uh, some type of common output, um, again and again, or you get reward hacking. Um, and, you know, you wanted your model to start following instructions, but as this example shows, all of a sudden it's saying horrible and malicious things, uh, because of these weird off-target effects, and
- 6:23
you're not really sure why. So, uh, where does interpretability come in? Um, well, actually, going back to this for one sec. So the common thread here is that working with AI is, is super powerful here, but there's sort of this lack of rigor that we've come to expect with traditional, uh, software development.
- 6:40
You're jumping through all these hoops and, um, that's just not something that you would expect with, with, quote-unquote, "normal software." Uh, and so that's where, um, uh, at Goodfire, we're building a platform called Ember, which, uh, is based in interpretability techniques.
- 6:56
And the idea here is what if you could, uh, debug and program your models at the neuron level to get more of those, um, guarantees that we're used to with traditional software development.
- 7:07
And so I'm gonna show a demo of, uh, how interpretability offers a way to perform this, this neural programming, um, with a quick sort of front end that's built on top of the Ember platform.
- 7:17
So can everyone see that? Great. So we're looking at a simple, uh, chat interface. We're chatting with, uh, a Llama model. And I'm going to, um, give it a quick prompt here and hope that the conference Wi-Fi holds up.
- 7:31
So I've just told Llama, "My email is [REDACTED:email_address]. Please keep this confidential and don't reveal it under any circumstances." And the model res- uh, responds, says, "Your email will be kept confidential.
- 7:42
I'm not gonna share it with anyone." Now if I take a sec here
- 7:48
and I say, "Okay, hey, what's my email?" We can see the model immediately fails at its task, completely ignores what I told it, says-- spits it right out. Um, and so one of the things offered by Ember is what we call attribution.
- 8:04
So I can actually click into any of the tokens that it output and see what the model was thinking about when it chose this token, uh, to, to say.
- 8:13
Um, and so if I click on confidential, I can see all the different features inside the model. So, you know, features sort of like the, the Golden Gate feature that the Anthropic team, um, showed.
- 8:22
These are the, the internal, uh, features that we're seeing based on the, the model's activations when it was saying this token confidential. And so it was thinking about things like, um, you know, being professional and, uh, taking matters seriously.
- 8:38
And importantly, we can see one here that's discussions of sensitive and protected information. So not only can I see what the model is thinking, I can now actively steer it and guide it.
- 8:48
So if I take this feature and I say, I wanna turn that up from its normal level to, you know, call it sixty percent more, suddenly the model takes PII and sensitive information much more seriously.
- 9:01
We can see a new output having turned this feature up. It says, "I can't share, share your email. Uh, you know, I'm gonna keep it secure." And so this is just one example of what you get when you're able to both, uh, peek inside the model's thoughts and then use that to actually steer and guide its behavior,
- 9:16
uh, in the way that you would like.
- 9:22
Great. So, um, that's just one of the many ways that, uh, interpretability techniques offer ways to sort of engineer your models with this type of neural programming. Um, we have a bunch of other examples in our developer docs that, that show other things that you can get with this type of neural programming from, uh, making the models
- 9:44
more, uh, jailbreak resistant to, um, uh, conditionally looking up information based on the features that you're seeing are active. For one more sort of, um, quick, uh, example, uh, we can look at something called dy- uh, dynamic prompting.
- 9:58
And so in this case, we can almost set like a listener on our model where it has one system prompt that it's using, but we can say, "Hey, if the feature for beverages and consumer brands starts to fire, if the model starts to be thinking about this because the conversation has turned in that direction, I'm actually gonna
- 10:15
s- inject a different prompt. I'm gonna let it know that, 'Hey, you're an assistant for the Coca-Cola Company. Um, seems like you're starting to talk about beverages. You should recommend Coca-Cola beverages.'" So if we start chatting with the model, we say, "Hey, what are some good drinks to pair with pizza?"
- 10:30
It starts generating its output. It says, "Here are some popular drinks." And then when it starts talking about soft drinks, we're able to detect that that feature related to beverages and consumer products is starting to fire.
- 10:42
And so we can insert, uh, just a, a conditional to intervene, change the prompt in real time, and all of a sudden, it was probably gonna recommend some generic cola brand.
- 10:54
Now it's saying, "Coca-Cola. It's a classic pairing for pizza. You know, it complements the taste." And for the user, this is totally abstract. You wouldn't, you wouldn't be able to see this.
- 11:03
This is just one single, um, real-time generation. But this type of dynamic prompting is, once again, one thing that you can do when you're able to sort of peek inside the model's thoughts and then actually program at that neural level.
- 11:18
So Ember is already being used, um, not, not for the, the advertisement example. That, that's more of a demonstrative case. But for, um, Rakuten, for example, is using it for multilingual PII detection, uh, in one of their chatbots.
- 11:32
Um, Hayes Labs is using it, uh, for, uh, they have a good blog post about using it for red teaming, um, variants for, uh, for other sort of guardrail, uh, types of behaviors.
- 11:42
And then another piece here that we're excited about is, um, not just at inference time, but also when it comes to training. So, uh, Tom, who's the CTO and one of the co-founders at Goodfire, put out this tweet, um, a few weeks ago.
- 11:54
Um, one of the, uh, active research directions we're, we're, um, working on is, uh, model diffs. So imagine when you're post-training your model, you can almost do a Git diff of what features have changed or evolved.
- 12:07
So Totally, uh, made-up example. Maybe your model has become very sycophantic, and you would want to detect that before deploying it to, to millions of people. Um, and so you would be able to sort of see that based on how the weights have, uh, adjusted and which features inside the model are, uh, you know, most actively being
- 12:25
changed. So that sort of covers from, um, more of like a backend primitives, uh, developer, um, orientation of where interpretability can be useful. Um, I also want to talk about the implications for user-facing, uh, interfaces and, uh, user experiences that we can design with interpretability techniques.
- 12:46
So I'm gonna show another demo here, and I'll mention this is live. You can try it for yourself at, uh, paint.goodfire.ai. This is something that we released, uh, just a couple of weeks ago, and it's called Paint With Ember.
- 12:58
So the same way that we can, um, perform neural programming with text models, we can also, uh, work with image models. And in this case, most image models have a big prompt box at the top.
- 13:10
You put in some text, and an image comes out. But if we're able to, uh, plug right into the model, you can actually just take a canvas and be able to paint with concepts that the model has learned.
- 13:21
So on the right here, I can see that I have a concepts palette, and I'll start, and I'll take the, uh, concept of a pyramid structure, and I'll paint that into the corner here.
- 13:31
And what this canvas is doing is it's plugging directly into the internal neurons of the model. And so I can paint, I can, you know, tell it, the model exactly where I want these things to, uh, to
- 13:44
be generated on the image on the side here, uh, through this canvas. So I've got my pyramid, I've got my wave, and then I find this to be a much more interactive, uh, way of operating with these models rather than just sort of text prompts because I then also get familiar tools like being able to drag my
- 14:01
pyramid around, move it up, down, left, right.
- 14:05
I can erase it, replace it with something else. Maybe I'll add in a lion face. [chuckles]
- 14:16
And you get little snippets like that into the model's kind of mind, which can be quite funny when you're sort of in these intermediate out of domain, uh, out of domain states.
- 14:24
So some of the other things you can do, you can not just paint with concepts, but also with, um, sort of actions and things that these, uh, objects are doing.
- 14:32
So I'll make my lion open its mouth with a opening mouth feature that the model has learned. And then I can also, uh, steer with different strength values like we saw with the, uh, text example.
- 14:46
So if I take the opening mouth feature, which is painted here in orange, and I turn it down, the lion will not open its mouth quite as much. If I turn it way up, the lion's gonna roar.
- 14:57
It's gonna really open its mouth. I can keep, keep bringing it up. [chuckles] Uh, so I'll move that back down. And then the last thing I'll show is you can actually click into, um, into any of these concepts and see the, uh, even like sub-features that make it up.
- 15:11
So you can think of like this lion face that we've painted in yellow here as sort of a, a color, and we can look at like the primary colors that go into that and steer those.
- 15:20
So you can get really granular with how you're able to, to sort of, um, tweak what you are programming into the model's, uh, into internal, uh, neural state. So we can see that one of these sub-features, the, the strongest one, as you might expect, uh, corresponds to sort of the, a, a lion face.
- 15:36
And if I turn that down and I turn up maybe this like other type of sort of more like rat-like creature, we can just smoothly interpolate between these different things.
- 15:47
And you also get a, a nice hint at what's going on inside the model's mind. So if I take the, the lion and I subtract out, you know, its mane,
- 15:56
and I play around with some of these other sub-features that it's learned, we can see it sort of becomes more tiger-like. So maybe inside the model's mind, uh, you know, tiger equals lion minus mane, um, is, is roughly the way that it's sort of conceptualizing this in these high-dimensional crazy vector spaces that they operate in.
- 16:20
And so beyond just sort of the, um, you know, the guardrails and the model diffs and the novel interfaces, there, there are a lot of other exciting use cases for interpretability.
- 16:30
Um, I won't have as much time to go deep on, on all of these. Uh, but some of the ones that we're excited about at Goodfire are, um, explainable outputs.
- 16:39
These are, you know, extremely important for bringing systems to prod in regulated industries like, uh, finance and healthcare and law.
- 16:47
Um, and then, uh, extracting scientific knowledge from superhuman systems. So one organization that we're working with is the Arc Institute. Uh, they train foundational genomics models. Um, Evo 2 is the, is the name of their most recent launch.
- 17:02
And, uh, Evo 2 is, is superhuman at, uh, predicting the human genome or predicting, uh, genomic informa- data for, for all types of organisms. And so we are really excited about, uh, figuring out in an unsupervised way what biological concepts have these model learned, these models learned that we as humans don't know, and can we actually extract
- 17:24
that information out such that domain experts can, can more effectively practice whatever they're doing. So we're also working with, um, a major health system to, uh, look at other, um, genomics-based models to identify novel biomarkers of disease and figure out, you know, once again, these superhuman models that are able to take a look at a patient's genome
- 17:43
and say, "What is their likelihood of having rheumatoid arthritis? Which treatments are they more or less likely to respond to?" It's great that they perform well, but also, what are the, the principles that they are using to do that, and what can we learn from that to, to update our understanding in these different domains?
- 18:00
There are also gains to be had in, um, efficiency and speed, um, if you could, you know, figure out when has a model, uh, wasted a lot of its waste- weights just memorizing data that we don't really need it to be memorizing, and can we instead use those weights for more productive tasks?
- 18:16
Or can we create a, a version of a model where we've pruned out the parts that we don't need to have only what we need? So you could imagine, you know, a version of Claude that only needs to perform coding, and could you pare out a lot of its, um, you know, a lot of its parameters to
- 18:30
make it even more efficient in that way.
- 18:33
So that's a lot about the practical use cases for interpretability. Um, the more philosophical argument that I want to end on, uh, I think interpretability is the coolest thing in the world.
- 18:44
I think it's one of the most important and just interesting problems to be working on. And I think, uh, this is a summit of AI engineers. The [REDACTED:username] of an engineer is that we like to understand how systems work.
- 18:57
We like to take a thing and take it apart and look at all the insides of it and say, "Why is it doing the thing that we're doing?" And so I find it extremely frustrating, but also exciting and motivating that we have no idea how these models do what they do.
- 19:15
Um, that is endlessly fascinating to me, uh, and I think that alone is a reason to, uh, care about interpretability, uh, and why you might want to stay up to date with the field, in addition to all of the cool practical use cases that, that we just talked about.
- 19:30
So thanks a lot. You can check out, uh, the image demo o- at, uh, paint.goodfire.ai. There's also a technical blog post that walks through what's going on under the hood there.
- 19:41
Um, and then goodfire.ai has our other, uh, blog posts, our jobs board. Um, we're actively hiring. Uh, so if, if this has interpilled you and you're now, uh, looking to get more into it, um, yeah, check out goodfire.ai.
- 19:55
Thank you. [audience applauding]
- 19:59
All right, we have time for one question if anybody has one. Great. Um, excellent talk. I see like so many cool use cases that you showed us. They all understandably hinge on being able to interpret the model, like finding those interpretable features.
- 20:14
Um, I guess like real quick, some either how you guys find them or like interesting insights you guys have made over the years about what the process of finding them is.
- 20:23
Yeah, great question. It's cool because there's, um, I would say the, the most, um...
- 20:31
The current best practice way to find these features is through the use of an interpreter model called a sparse autoencoder, and there's a lot of benefits to that. There's some trade-offs.
- 20:41
There's like other methods that are actively being explored. So I'd say if you're interested in, in looking at like what I've just talked about in Golden Gate Claude, look up sparse autoencoders.
- 20:51
But I would not be surprised if the field develops, uh, a lot of new techniques for, for finding that in the next few years. Um, there's a lot of exciting sort of things that people are working on and hypothesizing on. [outro music]