Vision AI in 2025 — Peter Robicheaux, Roboflow

[upbeat music] I'm gonna be giving a sh- quick presentation about the state of the union regarding AI vision.

Um, so I'm Peter Robicheaux. I'm the ML lead at Roboflow, which is a platform for building and deploying vision models. Um, so a lot of people are really interested in LLMs these days, so I'm trying to pitch why computer vision matters.

Uh, so if you think about systems that interact with the real world, they have to use vision as one of their primary inputs because the w- the built world is sort of built around vision as a fundamental primitive.

Um, and there's a big gap between where human vision is and where computer vision is, uh, I, I would argue a bigger gap than exists currently for human speech and, uh, computer speech.

Uh, computer vision has like, uh, its own set of problems that are very distinct from the problems that need to be solved by LLMs. Latency usually matters. You need to-- If you want to perceive motion, you have to be running your process multiple frames per second.

Uh, you usually wanna run at the edge. You can't have, like, one big hub where you do all of your computation because you would introduce too much latency to make decisions based off that computation.

Um, and so I sort of gave, uh, a version of this talk, uh, at Latent Space podcast at NeurIPS. Um, and retrospectively, I think we identified a few problems with, uh, the field of vision in twenty twenty-four, one of them being evals are saturated.

So vision evals like ImageNet and COCO, they're mostly like pattern matching. They measure your ability to match patterns and sort of like don't require much visual intelligence to solve.

Uh, consequently, I think vision models don't leverage big pre-training the way that, uh, language models do. So right now you can take a language model, unleash it on the internet and get something incredibly smart.

Some of the best vision models are cert-- are moving in that direction. But because you don't need that level of like knowledge and intelligence to solve the evals, there's kind of no incentive to do so.

Uh, and I think that part of that-- So there's sort of two dimensions here. One is that vision doesn't leverage big pre-training. So you can think of, like, if you're building an application with language right now, you probably wanna use the smartest model to get an embedding that works really well for you.

And right now we don't have-- Like, there are downstream applications that make really good use of the pre-training and the embeddings that they get from large language models. But there aren't really good vision models that can leverage these embeddings.

And the, the corollary to this is that th-there are-- the quality of big pre-trained models just isn't the same in vision as it is in language. And so my, my underlying conclusion is vision models aren't smart.

That's the, the takeaway. And I can prove it to you. So last year when Claude Three point five was happening, you could give it an image of a watch and it just guesses-- You ask it what time it is, and it'll just guess a random time.

And that's because this model, it has a good conceptual abstract idea of what a clock is or what a watch is, but when it comes to actually identifying the location of watch hands and finding the numbers on the watch, it's hopeless.

Uh, and updated for Claude Four, still has no idea what time it is. And this is even like a, a, an egregious failure because ten ten is like the stock time on like all watches.

So the fact that it couldn't even get like the most common time is, uh, pretty telling.

Uh, there's-- So there's this really cool d-- uh, dataset that's trying to measure this inability of LLMs to see, uh, called MMVP, which basically you can see an example here where they ask this question that seems incredibly obvious.

And the model-- So in this case, they ask the model, which is like ChatGPT Four, uh, which direction the school bus is facing. Are we seeing the front or the back of the school bus?

And the model com-- gets it completely wrong and then hallucinates details to support its claim. And again, I think this is evidence that large language models, which are maybe the most intelligent models that we have, like cannot see.

And that is due to a lack of visual features that they can perceive with. And so the way that th-this dataset was created is they went and they found pairs of images that were close in CLIP space, but far in DINOv2 space.

So CLIP is a, is a vision-language model that was c- sort of contrastively trained on the whole internet. DINOv2 is just a pure vision model that was, uh, trained in a self-supervised way on the whole internet, right?

And so what this is showing is that CLIP is not discriminative enough to tell these two images apart, right? So according to CLIP, these two images basically look the same.

And what that's pointing to is like a failure in vision-language pre-training. And so the way CLIP is trained is basically you come up with a big dataset of captioned images, and you, you ask the model to-- You scramble the captions and scramble the images and ask the model to pair the image with the caption.

But the thing is, is if you go back and look at these two images, what is a caption that would distinguish these two images, right? It's like the peculiar pose of the dog.

In one, in one image it's facing the camera, in one it's facing away. But these are sort of details that aren't included in the caption. So if your loss function can't tell these two images apart, then why would your model be able to, right?

So vision-only pre-training kind of works, is, is the claim. So DINOv2 is this really cool model that-- So what you're seeing right now is a visualization of its PCA'd features that have been self-discovered by pre-training on the whole internet.

Um, so what's really cool is not only, like, does it find the mask of the dog, obviously, that's sort of easy 'cause it's highly contrasted with the green background, but it also finds the segments of the dog, and it finds even analogous segments.

So if you look at the, these principle components, you compare the legs of a dog, it'll be in the same sort of feature space as the legs of a, of a human.

And so the, eh, there's sort of this big open question, which is like, how do we get vision features that are well-aligned with language features and usable by VLMs that don't suck [chuckles] and like have visual fidelity?

Um, cool. So, so that's part of the story. The other part of this, the question that needs to be answered is, given that we have some sort of semi-working large pre-training, uh, of vision models, why aren't we leveraging these vision models?

And I would answer that, at least in the object detection space, the answer is mostly in the distinction between convolutional models and transformers. So this is from LW-DETR, which is one of the top-performing, uh, detection transformers that currently exists.

Uh, if you look at this graph, if you look at YOLOv8, YOLOv8n, which is a convolutional object detector on the edge, with and without pre-training on Objects365, it gains like 0.2 mAP, which is like the main accuracy metric for object detectors.

So Objects365, which is a big million, 1.6 million image dataset, uh, pre-training on it leads almost no performance improvements on, on COCO. Whereas for LW-DETR, which is a transformer-based model, you can see that without...

If you look at this column mAP without pre-training, and you look at the column mAP with pre-training, you can see that you're getting like five mAP improvements across the board, sometimes even seven mAP improvements, which is like a gigantic amount, right?

And so basically, the-- while the language world knows that transformers are able to leverage big pre-trainings and, and yield decent results, the vision world is sort of just now catching up.

Uh, and you can see this from the scale of the big pre-training. In the image world, pre-training on Objects365 with 1.6 million images is considered a large pre-training. That would be like a tiny like challenge dataset for like, for like undergrads in the LLM world.

So [chuckles] I wanna announce Roboflow's special new model called R- RF-DETR, which leverages the DINOv2 pre-trained backbone and, uh, perform or uses it in a real-time object detection context. So this is sort of our answer to the, the, the hole that we see in the field of like why aren't we leveraging big pre-trainings for visual models.

Um, and so here are some of the metrics. You can see that, um,

basically what we did is we took the LW-DETR backbone, and we like kinda swapped it out with the DINOv2 backbone, and we get like a decent improvement on COCO.

Um, and we're still not SOTA, uh, compared to on COCO compared to DeFine, which is the current SOTA. We're like second SOTA. Um, but I think what's really interesting is there's this other dataset called RF100-VL, which we created to measure the sort of domain adaptability of this model.

And you can see massive yields from using the DINOv2 pre-trained backbone, which basically is pointing to the fact that, number one, COCO is too easily solvable. Uh, it basically has common classes like humans and like coffee cups and stuff like this, so it's not a good measure of the intelligence of your model.

More so the way that you optimize COCO is by like really nailing the precise location of a bounding box or something, really having good iterative refinement of your like locations that you're guessing.

Um, whereas we posit RF100-VL, this new dataset, is a better measure of the intelligence of a visual model. Um, so we're introducing a new dataset, RF100-VL, which is a collection of 100 different object detec- detection datasets that were pulled from our open source collection of datasets.

We have something like, I don't know, it's something like 750,000 datasets or whatever on Roboflow Universe, and we hand-curated the 100 best, I guess, by some metrics. So like we sorted by community engagement, and we tried to find very difficult domains.

So you'll notice, for instance, we have different, uh, camera poses that are common fr- in COCO. So we have, uh, like aerial camera, camera positioning, uh, which requires your model to sort of understand different views of an object in order to, to do well.

We have different visual imaging domains. Like you can see like microscopes and X-rays and all this sort of things. Uh, so yeah, we think that this dataset can measure the, the richness of features that are learned by object detectors in a much more comprehensive way than COCO.

Uh, and h- here's-- So the, the other fun thing about this is that it is a visual language model, so we are able to benchmark a bunch of different models on RF100-VL, being able to ask them things like using, contextualizing the class name in the context of this dataset, where, where is this action happening, for instance.

So for-- So if you look at the top left, we have this class which is block, which is representing an action, a volleyball block, but you have to be smart enough to contextualize this like word embedding of block within the context of volleyball to be able to detect that.

Same thing with this thunderbolt-type, uh, defect in this cable here. If you just ask a, a, a dumb vision-language model to detect thunderbolts in the image, it will find nothing.

But if it contextualizes it in the context of a cable defect, then it'll be able to find more things. And it also increases the breadth of classes. So if you only look at COCO, you're basically asking your model, "Hey, can you find a dog?

Can you find a cat?" But like, "Can you find fibrosis?" Now your model needs to have like a lot more information around the world, about the world to solve that problem.

Same thing with different imaging domains. Um, so it is a vision-language benchmark, so it, we also have, um, visual descriptions, uh, and sort of instructions on how to find, uh, the objects that are present in this image.

And basically, what we found is like you take a COCO or you take a YOLOv8 model and you train it on like 10 examples per class, it does better than like QwenV2 72- Qwen 2.5VL 72B, like state-of-the-art gigantic vision-language model.

So the vision-language models are really good right now at generalizing out of distribution in the vis- in the linguistic domain, but absolutely hopeless when it comes to generalizing in the visual domain.

And so we hope that this benchmark can sort of drive that part of the, the research and make sure that the visual parts of VLMs don't get left behind.

Uh, and yeah, basically by leveraging like stronger embeddings, uh, a DETR model does much, much better on RF100VL than just leveraging embeddings learned on Objects365, which makes sense. And that's my talk.

Thank you. Yes.

Can you, can you fine-tune it to run it at the edge?

Fine-tune Qwen on the edge?

On the edge. You know, run it yourself.

Oh, yeah, yeah, yeah. It, it's, it's like 20 million parameters at the small size, yeah. Cool. Any other questions? Thanks, Wicks.

Yeah. Is the dataset, uh, publicly available now?

Yeah, it's publicly available. It's on ... Maybe I can-- If you go to rf100vl.org, you can find our archive paper as well as the code utilities to help download the dataset.

It's also like on Hugging Face somewhere. Yeah.

Is that including the [inaudible]

Yeah. So Roboflow kinda has a pretty unique strategy when it comes to our platform. So we make our platform freely available to all researchers basically. And so we have like a ton of people who use our platform to label medical data and biological data for their own papers and their own research, and then our only ask is

that they then contribute that data back to the community and make it open source. And so a lot of this data comes from like papers cited in Nature and stuff like that.

For natural images, there's not a between image pattern matching versus like distortion, tiger type of thing in ImageNet. For like a lot of the images in biology and medical images, a lot of the distortion are kind of not necessarily down to the pattern, but it's physical.

Even though a lot of those are not just kind of natural-

Yeah. So the dataset is kind of measuring the performance of like a bunch of different imaging modalities or predictive modalities, I guess. So, so I think the most interesting tract of the dataset is the few-shot tract.

So basically, we've constructed like, uh, canonical 10-shot splits. So we provide the model the class name, uh, annotator instructions on how to find that class, as well as 10 visual examples per class.

And if a model-- Basically, no model exists that can leverage those three things and get higher mAP than if you just deleted one of those like options. I th- I see that as one of the big shortcomings of vision-language models right now.

In terms of multi modality, there's like a more generous kind of thing like a model that is like can be more branded versus more specialist like a SAM using-

Yeah

... very similar like Grounding DINO.

Yeah.

So then what do you think about the like using like models in terms of these are two directions, which do you think is more important?

Yeah. So, so currently the specialists are by far the best.

Yes.

Uh, we benchmarked Grounding DINO specifically-

Okay

... both zero-shot and fine-tune. So zero-shot Grounding DINO got like 19 mAP average on RF100VL, which is like kinda good, kinda bad. So if you take like a YOLOv8 Nano and you train it from scratch on the 10-shot examples, which is not a lot of data obviously, it gets something like 25 mAP.

So like to, to be worse than fine-tuning a YOLO from scratch is, is sort of bad. But if you then fine-tune the Grounding DINO with Federated Loss, that's the highest performing model we have on the dataset.

However, that being said, like I think that the point of the dataset should be, hey, like you should be able to leverage these annotator instructions, the 10-shot examples, and the class names, and come up with something more accurate, which requires a generalist model.

But okay, I think I'm super over time. So yeah, thanks for the questions.

Yeah. Thanks a lot.

Cool. Thanks everyone. [outro music]