AI Engineer Europe 2026
How Transformers Finally Ate Vision
Read the talk
How Transformers Finally Ate Vision
Vision transformers traded built-in spatial structure for learned representations, then borrowed optimized attention and architecture search to make those representations practical.
From a talk by Isaac Robinson
Before you start: Basic familiarity with neural networks, attention, and pre-training is helpful; the article explains image patches, masked reconstruction, and linear probing as they arise.
A person is still a person in either corner
A person in the upper-left corner of an image is still a person in the bottom-right corner. Convolutional neural networks encode a useful assumption about that problem: the same filter applies across image locations, so a feature detector does not have to learn an independent rule for every position. Stacking these operations builds hierarchical representations, as in ResNets. This shared spatial structure is the starting point for Isaac Robinson’s comparison of convolution and attention.
A transformer starts from tokens and applies a set-to-set operation. Dense attention considers pairs of tokens, making its attention cost quadratic in token count. Constraints can then shape what those tokens may do: an autoregressive transformer, for example, uses a causal mask to prevent attention to future positions. When Robinson describes the transformer as having no inductive bias, the useful distinction is that it lacks convolution’s built-in spatial priors—not that its architecture makes no assumptions at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The seemingly expensive way to process an image
The Vision Transformer, or ViT, adapts that token operation to images with a remarkably small recipe: split the image into patches, add learned positional encodings, and pass the resulting tokens through a transformer. The version described here uses 16×16 patches. Position is supplied explicitly, but the architecture does not enforce the same location-shared responses as convolution.
The scaling comparison needs two different quantities. For a square image with side length N and fixed patch width 16, there are (N/16)² patch tokens. Dense attention considers pairs of those tokens:
Thus attention is quadratic in token count, but quartic in image side length. With fixed kernels, depth, and channel widths, convolution instead scales with image area, or N². The comparison concerns the spatial scaling of these operations, rather than every component of a complete model.
ViT therefore appears to begin with two disadvantages: weaker built-in spatial structure and worse resolution scaling. Moving the person from one corner to another can produce a different activation pattern, rather than reusing a response constrained by shared convolutional filters. Yet ViT is the architecture Robinson presents as the eventual winner.
Large-scale pre-training and optimized attention change what makes an architecture competitive. Pre-training can teach useful visual structure that the architecture does not hard-code, while investment in LLM infrastructure makes the shared transformer operations faster. The path through Swin, ConvNeXt, and Hiera explores progressively different answers to the same question before returning to the simple, scalable ViT.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Swin makes attention local, then shifts the boundaries
Swin begins by restricting attention to local windows of patch tokens. Each token interacts with a small neighborhood instead of every patch in the image. But repeating the same partition would leave different windows isolated: tokens on opposite sides of a window boundary would never directly exchange information.
The next layer shifts the windows. Tokens previously separated by a boundary can now belong to the same attention group. Alternating the partitions lets information pass between regions while preserving local computation, resembling the overlapping spatial interactions of convolution. If window size stays independent of image resolution, computation grows with the number of windows, restoring N² scaling in image side length. Swin buys efficiency by putting locality back into the architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
ConvNeXt keeps the pattern and changes the mixer
ConvNeXt asks whether the useful lessons from transformers actually require attention. It starts with a 4×4 patchifying stem, rather than the 16×16 patches in the earlier ViT example, and separates two jobs: mixing information across spatial positions and processing the resulting features with a feed-forward operation.
| Design choice | ViT pattern | ConvNeXt adaptation |
|---|---|---|
| Spatial mixing | Self-attention | Convolution |
| Repeated pattern | Attention, feed-forward | Convolution, feed-forward |
| Spatial organization | Patch tokens | Hierarchical features |
Convolution takes over the spatial mixing job without abandoning the alternating mixer/feed-forward pattern. ConvNeXt also retains a hierarchical structure and incorporates LayerNorm and other design lessons. Robinson reports that ConvNeXt beats ViT and Swin on the standard ImageNet reference comparison. That is a result within the stated evaluation context, not a ranking across every training regime or deployment target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Hiera learns structure that architecture used to supply
A better accuracy result does not automatically produce the fastest model. Hiera investigates which architectural machinery is still necessary when pre-training is strong enough. Starting with a transformer rich in visual inductive biases, its ablations remove specialized components incrementally. Simpler operations offer speed gains; pre-training must recover useful structure that those components previously supplied.
The training mechanism is a masked autoencoder, or MAE:
- Divide an image into patches.
- Hide many patches and encode the visible context.
- Reconstruct the missing image content from that context.
The slide shows masked input patches passing through a Hiera encoder and a ViT decoder toward a cat-image target. Like BERT-style masked prediction in language, the reconstruction task forces the model to learn relationships among the pieces it can see and those it must infer. At scale, this teaches visual regularities that would otherwise need stronger architectural support.
Robinson motivates the advantage through patch deletion: a transformer can operate on the remaining tokens, whereas ordinary convolution expects a spatial grid. His claim that MAE cannot be applied to convolution is too categorical. ConvNeXt V2 provides a fully convolutional masked-autoencoder approach. The narrower distinction is that straightforward ViT-style patch deletion requires adaptation for convolution; it is not a prohibition on masked pre-training for CNNs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pre-training yields features, not just useful priors
With DINOv2 and DINOv3, pre-training does more than compensate for missing spatial biases. It produces rich feature maps that are useful before a downstream task is trained. Robinson uses these as examples of powerful ViT pre-training, although the current DINOv3 model collection also includes ConvNeXt backbones: access to this kind of representation learning is not exclusive to transformers.
The DINOv3 demonstration makes those features visible with a principal-component analysis, or PCA, of the feature maps. Different cat paws receive different colors, and corresponding parts remain identifiable across the cats. In the aerial imagery, the features separate regions in a semantically meaningful way. These are visual examples of structure already present in the representation, rather than downstream labels painted onto a raw image.
Linear probing tests how much of that structure is immediately usable. Freeze the backbone, extract its features, and train only the final linear projection for the target task. Robinson describes the resulting performance as approaching fully supervised results. The comparison is qualitative here: the talk does not specify a dataset, metric, backbone variant, or supervised baseline. The important mechanism is that the feature extractor stays frozen while only the small readout learns from the downstream data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimized attention changes the practical comparison
Rich features do not remove dense attention’s unfavorable resolution scaling. The next piece is FlashAttention, an attention optimization that vision models can borrow from the infrastructure developed around LLMs. Robinson points to Hiera’s reported speed advantage over ViT at matched accuracy, then asks what happens when the ViT implementation benefits from optimized attention.
Hiera’s paper excluded attention accelerators for all compared models. Its appendix describes measurements on an A100 40GB GPU, using FP16, 224×224 images, and optimal batch sizes. Robinson interprets the omission as a reason that adding FlashAttention could undermine Hiera’s advantage; the paper does not report an accelerated comparison establishing that the advantage disappears. Implementation speed and asymptotic complexity are separate questions. Faster attention can make ViT competitive at practical input sizes without changing its quartic attention scaling in image side length. Combined with strong pre-training, that shifts the balance back toward a simple backbone supported by widely optimized operations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same cycle appears in Segment Anything
The Segment Anything Model, or SAM, family brings this backbone history into an application setting. SAM begins with an MAE-pre-trained ViT. MobileSAM then substitutes TinyViT, a specialized convolution/transformer hybrid, to make the image encoder smaller. SAM2 uses Hiera with MAE pre-training. Robinson presents SAM3 as a return to relying on a massively pre-trained backbone.
The sequence illustrates changing choices about where to invest: a general transformer, a specialized lightweight encoder, a simplified hierarchical transformer, and then a stronger pre-trained starting point. It does not mean SAM3 abandoned architectural innovation; its design also includes a presence token and a decoupled detector–tracker. The narrower lesson is that the strength of available pre-training can outweigh the appeal of redesigning the backbone alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A strong foundation model is not a deployment budget
Depending on massive pre-training creates a new constraint. Repeating that investment to obtain a different model for every deployment budget is expensive, while reusing a fixed model can leave applications with a one-size-fits-all encoder. The representation may be excellent without fitting the hardware on which it must run.
Robinson cites SAM3 at approximately 800 million parameters and 300 milliseconds per run on a T4 GPU. The talk does not specify the task, input resolution, precision, batch size, or timing boundaries for that runtime, so it should be read as his deployment example rather than a universal latency specification. Those constraints matter particularly in vision, where low-power edge devices and other resource-constrained environments are common targets.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure transfer and deployment together
Roboflow’s response is to turn a fixed foundation model into a flexible family of downstream models. RF100-VL supplies the evaluation setting: diverse object-detection tasks that test how well foundation-model representations transfer beyond their original training distribution. The question becomes how much downstream accuracy can be preserved at a more useful operating cost.
In the presented RF100-VL comparison, Robinson reports about 40× speedup at matched accuracy versus fine-tuning SAM3, or about 15× speedup with a meaningful accuracy improvement. These are the talk’s reported comparison points; model variants, accuracy values, hardware, and execution backend are not specified. The preceding T4 runtime example does not establish the conditions for these ratios.
Combining strong pre-training with methods suitable for constrained hardware also extends beyond detection. Robinson separately reports that RF-DETR meaningfully outperformed the leading real-time instance-segmentation models at its publication. That is a distinct task and comparison from the RF100-VL detection result. The common objective is to retain the foundation model’s representational advantage while making its downstream form practical to deploy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search around the pre-trained starting point
The models along Roboflow’s plotted performance curve share the same underlying foundation model. Neural architecture search modifies that starting point to generate a family of models, rather than treating every speed–accuracy target as a separate foundation-model project.
The architectural choices remain compatible with the existing foundation-model infrastructure. Search combines those choices according to the target data and target hardware, producing different deployment options without discarding the common pre-trained starting point. Robinson does not enumerate the individual knobs in the spoken explanation; the consequential property is their compatibility with the representation already learned.
That completes the progression from architecture to training to deployment: massive ViT pre-training supplies reusable visual representations, LLM-driven attention optimizations improve practical execution, and pre-training-compatible architecture search adapts the result to a hardware budget. The simple backbone succeeds as part of this larger system, not because the costs of attention or the constraints of vision have ceased to exist.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What carries over to video and multimodal training?
The audience’s first question extends the discussion to architectures supporting unified video, image, and text pre-training. Robinson acknowledges work across many modality combinations and offers SAM3 as a partial example. His answer focuses on a specific form of video understanding: tracking objects through a video. SAM3 uses Perception Encoder pre-training for its backbone, followed by extensive downstream training. This describes a staged route into video object tracking, rather than a complete recipe for jointly pre-training all three modalities.
A follow-up asks about JEPA and V-JEPA. Robinson treats them as other foundation-model approaches, but says JEPA does not appear to outperform many alternatives for image-centric pre-training. For V-JEPA, his assessment is explicitly personal: he has not yet seen it used meaningfully for downstream video transfer. That is a limit on the applications he has observed, not evidence that such transfer is impossible or has never been demonstrated. He leaves the question open to future results—the same downstream usefulness test that motivated the deployment discussion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original Vision Transformer paper introducing image patches as transformer inputs.
Architectural ablations showing how masked pretraining supports a simpler hierarchical vision transformer.
Pretrained visual backbones, dense-feature examples and model-loading instructions; checkpoint access requires approval.
Image segmentation and video tracking with text and visual prompts. Includes notebooks and installation instructions; checkpoints require approved access.
A benchmark for evaluating object-detection transfer across diverse application domains.
Robinson and colleagues explain weight-sharing architecture search, deployment tradeoffs and their detection and segmentation evaluations.
Roboflow’s implementation of RF-DETR for object detection, instance segmentation and fine-tuning.
Further reading
- ConvNeXt V2Paper
A fully convolutional masked-autoencoder approach co-designed with an improved ConvNeXt architecture.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, I'm Isaac Robinson.
- 0:15
I'm the research lead [REDACTED:username], and I'm here to talk to you today about how transformers finally ate vision.
- 0:23
Uh, so I'm gonna start off with a brief summary of the competition, then we're gonna go through an overview of the evolution of the transformer,
- 0:35
why that ended up winning out, some consequences of that, and what's next.
- 0:40
So where we started, convolutional neural networks. I'm sure everyone here is aware of how these work, but just to summarize, [chuckles]
- 0:48
they have excellent inductive bias motivated by looking at how the eye works. So you have a, a filter that you convolve against your image, and you have activations that light up the same way regardless of where in the image the thing is happening.
- 1:05
Great inductive bias. A person in an image is a person regardless of whether they're in the upper left or the bottom right. And you-- we build these, uh, interesting hierarchical structures out of these ResNets, et cetera.
- 1:17
This is how we've done vision for a very long time.
- 1:21
Then comes the transformer. Again, I'm sure everyone is aware of how a transformer works, but just to summarize, we, uh, have a just a set of tokens. We run a set-to-set operation.
- 1:34
So there's no inductive bias. This is just an N squared transformation. We inject the inductive biases into the transformer. So for example, a classical, uh, autoregressive transformer, we add a causal mask to the attention matris- matrix, and that gives us a sequential modeling.
- 1:53
Uh, for vision, we have a vision transformer,
- 2:01
and this is very complicated. A lot of engineering went into this. We take our image, we split it into, uh, patches. Sixteen by sixteen was the original.
- 2:14
And we add a learned positional encoding, and then we throw that into a transformer,
- 2:21
and that's it. So transformer is N squared set to set. We've got patches in our image. This is, uh... We've got,
- 2:34
uh, N over sixteen patches for the side length N, and we end up actually with N to the fourth power with the resolution, uh, compute scaling. So we have no inductive bias.
- 2:45
The thing that is in the upper left could have a totally different activation pattern if it's in the bottom right.
- 2:53
And so the question naturally arises: which is better? The high inductive bias N squared convolutional network or the no inductive bias N to the fourth power ViT?
- 3:03
So as everyone would expect, it's the ViT.
- 3:09
So how is this possible? And I'm, I'm going to make the argument that it is because of massive ViT-specific pre-training,
- 3:19
and then we get to borrow a lot of speedups and infrastructure from the fact that LLMs are blowing up.
- 3:27
So to trace this evolution, we're gonna talk about-- We, we just talked about the introduction of the ViT, then how it-- how people tried to say, "Okay, well this, this cannot possibly be the best thing that we can do.
- 3:40
How do we make this better?" So we go to Swin, then back to, uh, a convolutional-based network, ConvNeXt, then to Hiera, um, which I think is a, kind of a-- ha-has some really, really beautiful takeaways.
- 3:54
And then as always happens with machine learning, we come back, bitter lesson, to the simple thing that scales well: the ViT.
- 4:05
So, uh, first we're gonna start with Swin.
- 4:10
So we have this Patchify operation, and we take our, uh, our patches and we split them. Instead of doing global attention across all the patches at once, we just say, "Okay, we're gonna do at- attention in this window."
- 4:24
If we just keep doing attention in this window though, the, the, uh, tokens will not be able to interact with each other. So these two will never see each other.
- 4:33
And so the next layer, in fact, we shift the window a little bit. So we've got these back and forth overlapping windows, and this looks very, very similar to what I described with the convolution.
- 4:43
Yeah, so we've got this, uh, similar-looking operation that is happening on these sections of the image and, uh, then we end up with like overlap between the filters, the locations that they get applied on, and this is how we proceed.
- 4:58
And this actually gets us down to N squared if your window size is independent of your resolution, and it adds a locality inductive bias following the convolution of that.
- 5:08
So that, that seems logical. That makes sense. Um,
- 5:14
then we go to the, uh... Someone said, [chuckles] "Okay, look, there's this transformer operation that has no inherent relationship with vision. Let's go back to the convolutional network. Let's take all the learnings that we've had from the vision transformers and just spit them into a convolution network and see what happens."
- 5:32
So ConvNeXt says, "Okay, we're gonna do a Patchify operation." We're gonna do a, a-- I think it was a...
- 5:38
It was a four-by-four patch instead of a sixteen-by-sixteen patch. And we're gonna say our ViT was, as all transformers, a self-attention, feed forward, self-attention, feed forward, et cetera.
- 5:53
And that self-attention is mixing your spatial information. Okay? So for the convolutional network, what if we just say, "Okay, we're gonna have the convolution mix the spatial information. We're gonna do the same pattern."
- 6:05
Uh, mixer- Feed forward, mixer feed forward onwards. And we're gonna borrow the same hierarchical structure that everyone has been using for these convolution networks,
- 6:15
and, uh, also throw in Layer Norm and a couple other innovations, and that's it. We're gonna try that. Turns out that beats ViT and Swin when you apply it on the, uh, standard ImageNet reference.
- 6:30
That's great. Finally, we have something that makes a little bit of sense.
- 6:35
Um, turns out that's not super fast. So someone, uh, Meta decided, okay, what, what are, what are the actual important things here? The Conv, ConvNeXt has a bunch of these beautiful inductive biases.
- 6:50
It's following this formula that we got from the transformer.
- 6:55
Let's look at what those inductive biases are actually useful for. So we're gonna take a really, really good inductively biased transformer model. We're gonna strip out the biases one at a time.
- 7:06
We're gonna get a speed up because we don't have all this specialized equipment anymore for the inductive bias, and we're gonna use pre-training to learn the bias instead. So this is, I think, a really, really great example of the balance between pre-training and inherent inductive bias, which ends up being how transformers ultimately win out.
- 7:25
Here we're using, uh, uh, MAE, masked autoencoder. Um, for those of you who are not familiar, you take your image, you take your patches, you drop a bunch of the patches, and you ask the, the model to reconstruct what would've been in the patches just based on the context.
- 7:44
Very, very similar to BERT for those of you who come from the language space. Uh, you do this at scale, and it turns out the model actually learns back the inductive biases.
- 7:55
But you can't actually apply MAE to a convolutional network. How do you drop out a patch when you're doing this convolution that's invariant across patches? So it's a ViT specific, uh, pre-training technique that adds inductive bias that would otherwise be missing from the structure.
- 8:14
Um, so that's great. That's super interesting. That, that works nicely.
- 8:20
Turns out it doesn't-- You can take that to an extreme and you throw in DINOv2, DINOv3, these really,
- 8:29
again, ViT specific pre-training techniques, and you at the end of it don't just have these inductive biases towards how to process an image, you actually have really, really, really fit, uh, rich feature maps out of the box.
- 8:43
So for example, this is a PCA decomposition of the feature maps produced by a DINOv3 pre-trained ViT, and we see that the paws of the cat have different colors, and that it's tracing the paws correctly for each of the different cats.
- 8:59
The satellite imagery is decomposed in a way that is semantically meaningful. And, uh, in fact, the self-supervised learning objective is, is catching up with the best that we have from supervised learning, and this is via linear probe.
- 9:12
So you have your frozen features. You're just probing into them. You're not training anything specifically on your data, except for that linear projection at the end,
- 9:21
and you're getting very, very, very close to the best that we know how to do with self su- with, uh, fully supervised learning.
- 9:29
Okay, but what about the speed issue? This is still N to the fourth power.
- 9:33
Uh, it turns out that people are-- care a lot about attention. So in the LLM world, we start introducing these tools, uh, FlashAttention, the biggest one, and, uh, Hiera explicitly showed a speed up for the same accuracy versus ViT.
- 9:54
But they have a note in their paper where they, where they say, "Okay, we see the speed up,
- 10:01
and we're not gonna measure with FlashAttention." [audience laughing] So then you add back in FlashAttention, suddenly it doesn't really matter. You've got- You're back to this very silly Ns- N to the fourth power thing that benefits from this ViT specific pre-training method, and that's it.
- 10:19
We're, uh, we've kind of-- We're, we're-- we've kind of won.
- 10:26
So this is a talk about th- uh, this is a, an evolution of the backbones, how it is, how it is that the N to the fourth thing ended up beating out everyone that tried to beat it.
- 10:38
What does this mean in practice and application? So SAM is a very famous series of models.
- 10:47
Again, if I- I don't know where people come from, but SAM was, uh, w- from my perspective, one of the most important, uh, foundation model series in vision, period.
- 10:59
And we actually see the same pattern. So SAM to Mobile SAM to SAM two to SAM three. If you look at the backbones that are underlying it, it's a ViT trained with MAE.
- 11:10
Then someone says, "Okay, well, surely this cannot be the best that we can do." So they, uh, Mobile SAM actually uses a specialized convolutional transformer hybrid, uh, called TinyViT and replaces the, the ViT backbone.
- 11:23
And then SAM two actually uses Hiera with, uh, this MAE pre-training, and then SAM three just gives up on the architecture ablation and just says, "Okay, well, we've got this massively pre-trained backbone.
- 11:37
Let's just stick it in. That's the best that we can do."
- 11:41
Um, so that's all good and fine, but where does that actually leave us? Um,
- 11:50
this is really expensive if we're relying on these huge pre-training strategies in order to recover the performance that is lost due to the fact that our architecture is not biased towards the subject at all.
- 12:02
Uh, that means we have to spend a huge amount of money every time we wanna do a deployment.
- 12:07
So no deployment flexibility means that we have these one size fits all models. So SAM three Is this very, very powerful thing, but is also 800 million parameters. It takes 300 milliseconds to run on a T4 GPU.
- 12:23
It's not actually usable in a lot of cases, uh, especially because Vision historically has been focused on these very low power edge devices, these, uh, resco- resource constrained deployment scenarios.
- 12:36
So, uh, [REDACTED:username], what we have done is we've attempted to say,
- 12:44
how do we actually take these fixed foundation models and transform them into something that has flexibility? So we introduced a data set, RF 100 VL, that measures how well, uh, uh, foundation models transfer to downstream diverse tasks with respect to object detection, which is one of the
- 13:09
canonical vision-centric tasks. And we see about a 40x speed up for the same s- accuracy versus fine-tuning SAM3. Uh, and for merely a 15x speed up, we get a, a meaningful improvement.
- 13:27
Um, so this combination of this huge advancement in foundation model pre-training and these strong deployment methods, combined with, uh,
- 13:40
an actual ability to deploy these in hardware constrained environments, ends up being the, uh, final nail in the coffin for these classical convolutional based methods. Uh, these-- At the time of our publication of RF-DETR, these were the best convolutional--
- 14:00
uh, th- these were the best real-time instance segmentation models, and we, uh, we, we, we, we outperformed them in a, in a meaningful way. Uh [laughs]
- 14:10
Maybe. [laughs]
- 14:12
Um, and so this is-- All those models on that line use actually the same foundation model. We just modify the foundation model using neural architecture search, such that we generate an entire family of, uh, high-performance models in, in one go.
- 14:31
To do this, we actually introduce a bunch of, uh, flexible mobs. All of these are drop-in compatible with the existing foundation model infrastructure. And by mixing and matching them in a way that is dependent on target data and target hardware, we can, uh, resolve the issue that,
- 14:52
uh, these foundation models do not have deployment flexibility.
- 14:57
Um, so massive ViT specific pre-training plus speed ups from LLMs, plus, uh, pre-training compatible neural architecture search, and, uh,
- 15:11
that's it. [laughs] [audience clapping] Yay.
- 15:22
Do we have a- architectures that support like unified video plus image plus text pre-training already? Do-- Is someone working on that?
- 15:33
So there are a lot of people working on like a huge amount of different combinations of things. I actually think SAM3 is a good example of that. They do- [coughs] Well, i- in terms of,
- 15:46
in terms of vision specific video processing, it does video processing in the-- in-- from the perspective of tracking objects through video.
- 15:55
Mm-hmm.
- 15:55
So they do massive scale pre-training. They do the perception encoder pre-training for their backbone, and then they do a huge amount of downstream pre-training.
- 16:04
Or I guess you would just call it training at that point. Um, and-
- 16:08
What about JePA? Like the video-
- 16:10
Yeah, the video-- Yeah, JePA and the ViJePA, those are, uh...
- 16:15
I mean, yeah, those are another variety of foundation model. I think in terms of single, uh, like image centric pre-training,
- 16:23
JePA doesn't seem to outperform a lot of the other ones. Video JePA,
- 16:30
I haven't seen anyone use it meaningfully in a video context for downstream transfer yet, but, you know, we'll see.
- 16:39
Yeah.
- 16:40
Yeah. Any other questions? Cool. Okay.
- 16:48
That's it. [audience clapping] [outro jingle]