Vision AI in 2025 — Peter Robicheaux, Roboflow

How can a system interact with the physical world if it cannot reliably see what is happening? Built environments depend on visual information. Peter Robicheaux, introducing himself as Roboflow’s ML lead, argues that the gap between human and computer vision remains larger than the corresponding gap in speech.

Vision also imposes a different engineering constraint: perception has to arrive in time to affect a decision. Recognizing motion requires processing successive frames, multiple times per second. Sending every frame to a central computing hub can add enough latency to undermine the resulting decisions. That makes edge execution a practical requirement for many vision systems, rather than merely a deployment preference.

Looking back at a version of this discussion at NeurIPS, Robicheaux identifies saturated evaluations as a central problem. In his diagnosis, ImageNet and COCO largely reward pattern matching. If an evaluation can be solved without richer visual understanding, it supplies little incentive to build that understanding through larger pretraining. Language provides the motivating comparison: broad pretraining produces representations that many downstream applications can reuse.

There are two separate limitations here. Representation quality concerns what a pretrained vision model actually preserves about an image. Downstream transfer concerns whether another model can use those features effectively. Better embeddings alone do not guarantee a better detector; a more capable detector architecture cannot recover information its embeddings never retained. Robicheaux’s criticism addresses both sides of that relationship.

A watch makes the distinction concrete. In Robicheaux’s reported Claude 3.5 example, the model recognizes the kind of object it is looking at but supplies an incorrect time. Reading a watch requires a more precise chain of perception: locate the hands, distinguish their positions, and relate those positions to the dial’s numbers. Conceptual knowledge of watches does not establish that spatial fidelity.

His updated Claude 4 example also fails. He identifies the displayed time as 10:10 and points out that this is a common arrangement in watch imagery. These examples illustrate a failure on the presented inputs, rather than a comprehensive evaluation of either model: fluent knowledge about an object can coexist with an inability to read its visible state.

The MMVP benchmark tests similarly basic visual distinctions. In the school-bus example, GPT-4 is asked whether the image shows the front or back of the bus. The displayed image shows the front, but the response claims that the bus faces away and invents details to support that answer. The explanation sounds coherent even though its visual premise is wrong.

The construction of MMVP helps explain why this can happen. Robicheaux describes selecting image pairs that are close in CLIP embedding space but far apart in DINOv2 space. CLIP learns through contrastive image–text training; DINOv2 learns visual representations through self-supervision. A pair that CLIP treats as similar but DINOv2 separates exposes a distinction that the two representations preserve differently.

In the talk’s simplified account of CLIP training, the model must match images with their corresponding captions. But consider the dog examples: one dog faces the camera, while another faces away. An ordinary caption may describe both without mentioning orientation. If the training objective does not reward that distinction, the model has less reason to preserve it. Caption alignment can succeed while detailed visual discrimination fails. This explains a possible weakness in image–text supervision without requiring every vision-language model to use the same encoder.

DINOv2 offers evidence that visual pretraining can discover useful structure. Robicheaux presents a principal-component analysis, or PCA, visualization of its learned features. He highlights not only the dog’s separation from a green background, but also distinctions between parts of its body. The visualization exposes structure in pretrained features; it is not itself a supervised segmentation result.

He also points to analogous features for dog and human legs. The interesting possibility is a representation that preserves meaningful parts across different subjects. That leads to the unresolved problem: how can those detailed visual features become aligned with language and usable by a vision-language model without losing their fidelity?

Even when useful visual features exist, downstream models must be able to exploit them. Object detection makes that second limitation visible. Robicheaux uses a comparison from LW-DETR to contrast the convolutional YOLOv8n detector with transformer-based detectors. The question is how much each benefits from pretraining before evaluation on COCO.

The talk reports the following gains in mean average precision, or mAP, for the comparison shown:

The YOLOv8n figure is a point increase, not a relative percentage increase. These results concern the displayed models and training recipes; they do not establish that every convolutional detector is unable to benefit from pretraining.

The architectural implication is that pretraining value depends on the model receiving it. Robicheaux describes Objects365 pretraining here as using about 1.6 million images—already considered large in this detection setting, but modest compared with the scale associated with language pretraining. The opportunity is both to improve visual representations and to build detectors that can make better use of them.

RF-DETR is Roboflow’s response to that opportunity. Robicheaux describes taking the LW-DETR detector design and replacing its backbone with a pretrained DINOv2 backbone. The goal is to carry the benefits of visual pretraining into real-time object detection.

The resulting COCO improvement does not put RF-DETR first in the comparison presented: Robicheaux places it behind D-FINE. That is the historical comparison in this talk; the maintained RF-DETR repository now includes later releases. The more interesting result for his argument is the larger benefit on RF100-VL, a benchmark designed to test adaptation across domains.

COCO’s familiar classes, such as people and coffee cups, can make recognition relatively easy while leaving precise bounding-box placement as the main source of improvement. Better iterative refinement of box coordinates can raise a detector’s score without demonstrating much broader visual knowledge. RF100-VL is intended to make the richness of the underlying features matter more.

RF100-VL collects 100 object-detection datasets from Roboflow Universe. Robicheaux loosely estimates that the source collection contained about 750,000 datasets at the time. Curation combined community engagement with a search for difficult domains, rather than simply selecting more examples of familiar natural-image categories.

The resulting diversity changes what a detector must handle:

• Camera viewpoint: Aerial images require recognizing objects from positions uncommon in COCO.
• Imaging modality: Microscopy and X-rays introduce visual appearances unlike ordinary photographs.

Robicheaux proposes this breadth as a more comprehensive test of learned features. It is a benchmark for broader transfer, not an established universal measure of visual intelligence.

RF100-VL also tests the connection between language and localization. A class name is not always a self-contained description of what to detect. In a volleyball dataset, block denotes an action; recognizing it requires interpreting the word in the context of the sport.

Other examples extend the same requirement:

• Cable defects: thunderbolt names a defect appearance. Searching for literal lightning would miss the intended target.
• Scientific concepts: fibrosis requires specialized knowledge beyond familiar categories such as dogs and cats.
• Imaging context: The appearance of a target depends on the modality in which it is observed.

The benchmark therefore supplies visual descriptions and annotator instructions for finding the objects. This is its vision-language component: language helps specify the detection task within a domain. It does not mean that the RF-DETR architecture introduced earlier is itself a general conversational VLM.

Robicheaux reports that YOLOv8 trained on about ten examples per class outperforms Qwen2.5-VL-72B in the RF100-VL setting discussed. The striking comparison is between a detector adapted with a small amount of visual supervision and a much larger vision-language model. It should not be read as a comparison under identical training or prompting conditions.

His interpretation separates linguistic generalization from visual generalization. A VLM may handle unfamiliar language well while struggling when the images themselves depart from familiar domains. RF100-VL is intended to make that weakness visible and encourage research on the visual side of VLMs. The prepared talk closes by returning to pretrained features: Robicheaux reports that stronger embeddings improve DETR’s RF100-VL performance beyond what embeddings learned on Objects365 alone provide.

The first audience question returns to edge deployment. After briefly asking whether the question concerns Qwen, Robicheaux understands it as running the detector locally and answers affirmatively. He cites roughly 20 million parameters for the small detector. That size is not a latency measurement or a hardware prescription, and it does not describe the large Qwen model.

The dataset is publicly available. Robicheaux points readers to the RF100-VL project site for the paper and download utilities, and also mentions availability on Hugging Face.

Medical and biological datasets enter the collection through research use of Roboflow. Robicheaux describes offering researchers free access to labeling tools in exchange for contributing their data openly. He says some contributions come from research associated with papers cited in Nature. This explains the community contribution model behind the collection, rather than establishing the provenance or licensing of each constituent dataset.

An audience member raises the physical differences between natural images and biological or medical imaging. Robicheaux responds by emphasizing the benchmark’s range of imaging and prediction modalities, then identifies the few-shot track as especially interesting. Its canonical ten-shot splits supply three kinds of information: a class name, annotator instructions explaining how to find it, and visual examples. The original paper defines the ten-shot budget as ten object instances per class, not necessarily ten distinct images.

The desired behavior is that these inputs complement one another. The class name supplies a concept, the instructions clarify its meaning in the dataset, and the examples show its visual appearance. Robicheaux’s talk-time assessment is that no model they had found could combine all three to achieve higher mAP than it could obtain with one of those inputs removed. The weakness is not simply missing language or missing images; it is failure to turn their combination into better detection.

The final question contrasts generalist models with specialists such as SAM and Grounding DINO. Robicheaux says specialists currently perform best and describes evaluations of Grounding DINO both without adaptation and after fine-tuning.

His rounded Q&A figures and the original paper’s results differ:

The original paper averages across 100 datasets using pycocotools with maxDets=500. These are different reported results, not interchangeable rounded versions of the same scores. In the spoken comparison, Robicheaux describes YOLOv8n as trained from scratch on the ten-shot examples.

Robicheaux then reports that fine-tuning Grounding DINO with Federated Loss produces their highest-performing model on the dataset. The zero-shot score above does not describe that fine-tuned result. Yet specialist performance is only part of the goal. A generalist should be able to use annotator instructions, ten-shot examples, and class names together—and become more accurate because each contributes information the others lack. That is the capability the benchmark is meant to demand.