Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

A camera application needs to keep up with incoming frames. Sending every image to a vision language model makes that requirement difficult, especially on modest hardware. Merve Noyan, a computer vision practitioner at Hugging Face who has written a book on VLMs, opens with an intentionally provocative request: stop calling them directly for everything. Her target is a vision application that runs end to end, with a model suited to its runtime job.

The practical goal is roughly 30–40 frames per second on what she jokingly calls a toaster, whether the task is classification or instance segmentation. A task-specific model such as RF-DETR is her preferred route. This is a deployment argument about speed and task fit; the recording’s examples do not establish a universal performance advantage across every VLM, device, and vision problem.

Model selection also includes the license. Noyan uses developers’ recurring requests for YOLO as an example of popularity outrunning attention to deployment terms, and favors Apache 2.0 models. Her warning is to inspect the particular model’s license before choosing it; the talk’s brief license discussion does not establish payment obligations for every model sold or distributed under the YOLO name.

VibeVision, the toolkit shown on Noyan’s slide and published in her vision-intern repository, starts from a useful division of labor: an agent can call a specialized vision model as a tool. Noyan describes this as distilling her own model-selection habits into something a coding agent can use. The agent may write good software while remaining, in her words, a “clueless computer vision engineer.” Supplying tools and task knowledge reduces the number of vision decisions it has to invent.

Her selection criteria are deliberately practical:

• License: Check the terms before adopting the model, with Apache 2.0 and MIT among her preferred choices.
• Performance for the size: Compare benchmark results in light of model size and architecture, rather than choosing the largest model by default.
• Vibes: Keep room for the practitioner’s judgment about which models are worth using.

The more involved half is “vibe training.” With labeled images, a developer can go straight to training. With images alone, there are two additional jobs: create annotations and evaluate them. A VLM becomes the labeler, other VLMs become judges, and the retained annotations become training data for the detector. The expensive general-purpose visual reasoning moves into dataset preparation.

That makes the coding agent’s job a long-running workflow: oversee labeling, judging, and training rather than produce a single script and disappear. It can run locally or use Hugging Face infrastructure. Jobs handle one-off batch processing and training; Inference Providers route serverless model calls; buckets hold intermediate data alongside dataset and model repositories. Smaller, more capable vision models make the annotation stages cheaper, while the agent keeps the stages moving.

The pipeline labels an image dataset with a nine-billion-parameter VLM, then sends the annotations to two smaller judges. Their decisions are merged before training RF-DETR medium or large. Noyan avoids asking the judges for numerical quality scores: in the research she discusses, scores from differently sized models do not provide a useful common scale. The decision is instead whether to approve an annotation.

The important representation change happens between labeling and judging. The labeler emits bounding-box coordinates as tokens. Those coordinates are used to draw boxes over the original image, and the judges receive that image with labels and label descriptions. This lets the judge inspect the proposed region against the visible object and the intended class description. It does not have to infer the spatial relationship from a coordinate list alone. 7:17

What actually reaches the detector’s training set? The flow below separates the visual review input from the annotations used for training. Each judge sees the same proposed boxes and descriptions; approval from either judge is enough to retain an example. The coding agent generates the descriptions, but a human approves them before they guide the judges.

An already labeled dataset can skip annotation and proceed to training. The licensing preference also has a stated exception: Noyan describes one judge as having revenue-dependent commercial terms. Even within a curated pipeline, permission to use a model remains a per-model decision.

Noyan reports about three to four dollars for an entire experimental pipeline run. She uses serverless inference for labeling, Hugging Face Jobs for judging, and an L4 for training. That figure belongs to her experiments: without a dataset size or complete workload specification, it is not a general price estimate. Hardware choice also reflects impatience and a preference for a large batch size; she says the small detector can be trained locally or on other hardware.

Road-sign detection provides an external check on the generated labels because the dataset already has ground-truth annotations. Noyan also runs the labeling VLM on the test set and compares against both its pseudo-annotations and the ground truth. She reports a good mean average precision, with a gap between the comparisons. The metric wording does not support a precise score or threshold, but the distinction matters: agreement with the labeler and agreement with ground truth measure different things.

Document parsing asks a different question: can the detector learn to locate images, tables, and signatures from generated annotations? Noyan repurposes a document question-answering dataset for this extraction task, so she lacks equivalent ground-truth evaluation. The result is a qualitative example of generalization rather than a measured document-parsing accuracy.

The signature example makes the change concrete. The training workflow first creates document-region labels, filters them through judges, and trains RF-DETR. On a test document, the labeling VLM misses a signature, while the trained detector finds it. Learning from generated labels has produced a detector that can recognize a region its teacher misses on that image. Noyan credits RF-DETR’s backbone for this behavior; the example illustrates why noisy supervision need not make the student reproduce every individual teacher error. 10:57

Two judges do not automatically make a balanced review system. One rejects many more examples than the other, depending on the task. Requiring both judges to approve would leave too little training data in Noyan’s experiments, weakening generalization. Her minimum-agreement rule keeps an example when either judge says yes. It trades stricter filtering for more surviving examples, and the observed imbalance is smaller for document parsing.

The recommendation remains task-dependent. Noyan suggests observing how the judges behave on the particular task. Her captioned aside connecting consensus to recall is ambiguous; it does not establish a filtering policy to follow. Judge behavior discussion

The generated judge prompt still needs a person. A coding agent can draft class descriptions and review instructions, but someone must look at the dataset and decide whether those instructions match the intended task. “There’s no escaping that.” Automating the workflow does not remove the need to define what a correct label means.

Training code introduces another kind of mistake: an image transformation can invalidate its label. Noyan catches two examples:

• Horizontal flips of traffic signs: Mirroring can change directional meaning while leaving the original class label attached.
• Color jitter on traffic lights: Changing the color can change the state the model is supposed to recognize.

The coding agent applies familiar augmentation techniques without understanding those consequences. Noyan patches the workflow so augmentation can be explicitly allowed or disabled. The useful question is whether a transformation preserves the answer for this task.

The toolkit’s other half exposes selected models as agent tools, spanning depth estimation through zero-shot segmentation. The choices draw on benchmark comparisons across model sizes and Noyan’s reading of computer vision papers. This gives the agent a curated set of capabilities rather than expecting a single VLM to perform every visual operation.

Falcon Perception illustrates why capability matters as much as size. The request “segment this red car” identifies an object by an attribute. A request for the red car next to the orange car that is next to the blue car requires resolving relationships before selecting the region. Noyan presents Falcon Perception as able to handle this open-ended referring segmentation, at 600 million parameters with an Apache 2.0 license.

Other tools cover distinct jobs:

• Human-centered analysis: Pose and related models support human keypoint detection and depth tasks.
• Zero-shot detection: Multiple detector sizes let developers choose a smaller alternative when hardware or speed requires it.
• OCR: Models are selected from OCR benchmark results in different sizes.
• Depth estimation: License terms are checked alongside capability; a model family should not be assumed to share one license across sizes.
• Tracking: Supervision and tracker support from Roboflow help follow instances and bounding boxes.

Industrial objects expose a limit of language-based labeling: some parts are difficult to describe in words. Noyan proposes image-guided detection as a possible next step. A reference image shows the object to find, and the detector searches for that object across other images. The example becomes the query, avoiding the need to invent a verbal description precise enough to distinguish the part. This is a planned direction, not a demonstrated industrial result.

A second proposed change would make judging spatial. Instead of asking a judge to accept or reject the labeler’s box, ask it to produce its own box and compare the two using intersection over union: the area shared by the boxes divided by the area covered by either box. That would expose agreement about location directly. Noyan also says segmentation support is in progress.

For further implementation work, Noyan points to a small vision repository covering fine-tuning, quantization, and multimodal models; Transformers task guides with tutorials; and Hugging Face Skills covering computer vision and infrastructure. These serve different needs: learning a model workflow, following a task tutorial, and giving an agent the instructions needed to run training.

The Q&A keeps the priority concrete. Training VLMs themselves in a self-improvement loop sounds exciting to Noyan, but first she wants developers to train task-specific models and deploy them at the edge. She also explains why the coding agent drafts judge prompts: it already has the workflow’s context. That context helps generate instructions; the earlier human review step still determines whether they describe the task correctly.