# Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

Merve Noyan, Hugging Face

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/dKcTBQzR7jI-skill-issue-stop-deploying-vision-language-models) · [Video](https://www.youtube.com/watch?v=dKcTBQzR7jI)

## Skill issue: stop deploying vision language models, use them with Skills

Merve Noyan’s agent-assisted workflow uses vision language models to create and judge training labels, then trains a smaller detector for deployment. The hard parts are deciding which annotations to keep, which prompts to trust, and which image transformations preserve the task.

### Start with the camera’s frame rate

[0:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=13s)

A camera application needs to keep up with incoming frames. Sending every image to a vision language model makes that requirement difficult, especially on modest hardware. [Merve Noyan](https://hf.co/merve), a computer vision practitioner at Hugging Face who has written a book on VLMs, opens with an intentionally provocative request: stop calling them directly for everything. Her target is a vision application that runs end to end, with a model suited to its runtime job.

![Source frame: Start with the camera’s frame rate](https://ai.engineer/api/talk-essay-media/7bbbe2d386bd2ba236ce3c8a565d30e574874da8b08c0cf3e71185938669affe)

Source frame: Start with the camera’s frame rate

The practical goal is roughly 30–40 frames per second on what she jokingly calls a toaster, whether the task is classification or instance segmentation. A task-specific model such as **RF-DETR** is her preferred route. This is a deployment argument about speed and task fit; the recording’s examples do not establish a universal performance advantage across every VLM, device, and vision problem.

Model selection also includes the license. Noyan uses developers’ recurring requests for YOLO as an example of popularity outrunning attention to deployment terms, and favors Apache 2.0 models. Her warning is to inspect the particular model’s license before choosing it; the talk’s brief license discussion does not establish payment obligations for every model sold or distributed under the YOLO name.

### Give the coding agent computer vision judgment

[2:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=144s)

VibeVision, the toolkit shown on Noyan’s slide and published in her `vision-intern` repository, starts from a useful division of labor: an agent can call a specialized vision model as a tool. Noyan describes this as distilling her own model-selection habits into something a coding agent can use. The agent may write good software while remaining, in her words, a “clueless computer vision engineer.” Supplying tools and task knowledge reduces the number of vision decisions it has to invent.

![Source frame: Give the coding agent computer vision judgment](https://ai.engineer/api/talk-essay-media/ff4b096513ffb3469c43397eea90c3bc363d6e1f22fac4234eb5ecc14ab53869)

Source frame: Give the coding agent computer vision judgment

Her selection criteria are deliberately practical:

- **License:** Check the terms before adopting the model, with Apache 2.0 and MIT among her preferred choices.
- **Performance for the size:** Compare benchmark results in light of model size and architecture, rather than choosing the largest model by default.
- **Vibes:** Keep room for the practitioner’s judgment about which models are worth using.

The more involved half is “vibe training.” With labeled images, a developer can go straight to training. With images alone, there are two additional jobs: create annotations and evaluate them. A VLM becomes the labeler, other VLMs become judges, and the retained annotations become training data for the detector. The expensive general-purpose visual reasoning moves into dataset preparation.

That makes the coding agent’s job a long-running workflow: oversee labeling, judging, and training rather than produce a single script and disappear. It can run locally or use Hugging Face infrastructure. Jobs handle one-off batch processing and training; Inference Providers route serverless model calls; buckets hold intermediate data alongside dataset and model repositories. Smaller, more capable vision models make the annotation stages cheaper, while the agent keeps the stages moving.

### Judge the box on the image

[5:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=347s)

The pipeline labels an image dataset with a nine-billion-parameter VLM, then sends the annotations to two smaller judges. Their decisions are merged before training RF-DETR medium or large. Noyan avoids asking the judges for numerical quality scores: in the research she discusses, scores from differently sized models do not provide a useful common scale. The decision is instead whether to approve an annotation.

![Source frame: Judge the box on the image](https://ai.engineer/api/talk-essay-media/8a37f3911e72d502ac4d278288b24f88f9cf63cd43acd985eb4631521044dec7)

Source frame: Judge the box on the image

The important representation change happens between labeling and judging. The labeler emits bounding-box coordinates as tokens. Those coordinates are used to draw boxes over the original image, and the judges receive that image with labels and label descriptions. This lets the judge inspect the proposed region against the visible object and the intended class description. It does not have to infer the spatial relationship from a coordinate list alone. [7:17](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=437)

What actually reaches the detector’s training set? The flow below separates the visual review input from the annotations used for training. Each judge sees the same proposed boxes and descriptions; approval from either judge is enough to retain an example. The coding agent generates the descriptions, but a human approves them before they guide the judges.

An already labeled dataset can skip annotation and proceed to training. The licensing preference also has a stated exception: Noyan describes one judge as having revenue-dependent commercial terms. Even within a curated pipeline, permission to use a model remains a per-model decision.

From unlabeled images to detector training data
The judges review rendered boxes. Their approval decisions filter the proposed annotations before RF-DETR training.
- Image dataset — Images without task labels
- VLM labeler — Proposes labels and bounding boxes
- Render the proposed boxes — Overlay boxes on the image and attach label descriptions
- Human-approved descriptions — Coding agent drafts what each label means
- Smaller judge A — Approve or reject
- Smaller judge B — Approve or reject
- Retain if either approves — Minimum agreement preserves more training examples
- Train RF-DETR — Medium or large detector learns from retained annotations
- Image dataset → VLM labeler (Label)
- VLM labeler → Render the proposed boxes (Draw emitted coordinates)
- VLM labeler → Retain if either approves (Proposed labels and box coordinates to filter)
- Human-approved descriptions → Render the proposed boxes (Add class meaning)
- Render the proposed boxes → Smaller judge A (Visual review)
- Render the proposed boxes → Smaller judge B (Visual review)
- Smaller judge A → Retain if either approves (Verdict)
- Smaller judge B → Retain if either approves (Verdict)
- Retain if either approves → Train RF-DETR (Filtered annotations)

### A few dollars, then two different tests

[8:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=520s)

Noyan reports about three to four dollars for an entire experimental pipeline run. She uses serverless inference for labeling, Hugging Face Jobs for judging, and an L4 for training. That figure belongs to her experiments: without a dataset size or complete workload specification, it is not a general price estimate. Hardware choice also reflects impatience and a preference for a large batch size; she says the small detector can be trained locally or on other hardware.

![Source frame: A few dollars, then two different tests](https://ai.engineer/api/talk-essay-media/36a0c37e2b62fc47ea584d704e7dfd8f1ae7a6bc62a6cfd37a4a419cde20fc1a)

Source frame: A few dollars, then two different tests

Road-sign detection provides an external check on the generated labels because the dataset already has ground-truth annotations. Noyan also runs the labeling VLM on the test set and compares against both its pseudo-annotations and the ground truth. She reports a good mean average precision, with a gap between the comparisons. The metric wording does not support a precise score or threshold, but the distinction matters: agreement with the labeler and agreement with ground truth measure different things.

Document parsing asks a different question: can the detector learn to locate images, tables, and signatures from generated annotations? Noyan repurposes a document question-answering dataset for this extraction task, so she lacks equivalent ground-truth evaluation. The result is a qualitative example of generalization rather than a measured document-parsing accuracy.

The signature example makes the change concrete. The training workflow first creates document-region labels, filters them through judges, and trains RF-DETR. On a test document, the labeling VLM misses a signature, while the trained detector finds it. Learning from generated labels has produced a detector that can recognize a region its teacher misses on that image. Noyan credits RF-DETR’s backbone for this behavior; the example illustrates why noisy supervision need not make the student reproduce every individual teacher error. [10:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=657)

### Filtering can starve the model; augmentation can change the answer

[11:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=688s)

Two judges do not automatically make a balanced review system. One rejects many more examples than the other, depending on the task. Requiring both judges to approve would leave too little training data in Noyan’s experiments, weakening generalization. Her minimum-agreement rule keeps an example when either judge says yes. It trades stricter filtering for more surviving examples, and the observed imbalance is smaller for document parsing.

![Source frame: Filtering can starve the model; augmentation can change the answer](https://ai.engineer/api/talk-essay-media/8509a511e15430dee5b6cca42b6a749db83e34da38a77c81d98662712d4f64e2)

Source frame: Filtering can starve the model; augmentation can change the answer

The recommendation remains task-dependent. Noyan suggests observing how the judges behave on the particular task. Her captioned aside connecting consensus to recall is ambiguous; it does not establish a filtering policy to follow. [Judge behavior discussion](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=727)

The generated judge prompt still needs a person. A coding agent can draft class descriptions and review instructions, but someone must look at the dataset and decide whether those instructions match the intended task. “There’s no escaping that.” Automating the workflow does not remove the need to define what a correct label means.

Training code introduces another kind of mistake: an image transformation can invalidate its label. Noyan catches two examples:

- **Horizontal flips of traffic signs:** Mirroring can change directional meaning while leaving the original class label attached.
- **Color jitter on traffic lights:** Changing the color can change the state the model is supposed to recognize.

The coding agent applies familiar augmentation techniques without understanding those consequences. Noyan patches the workflow so augmentation can be explicitly allowed or disabled. The useful question is whether a transformation preserves the answer for this task.

### Choose tools for the visual question and the hardware

[13:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=810s)

The toolkit’s other half exposes selected models as agent tools, spanning depth estimation through zero-shot segmentation. The choices draw on benchmark comparisons across model sizes and Noyan’s reading of computer vision papers. This gives the agent a curated set of capabilities rather than expecting a single VLM to perform every visual operation.

![Source frame: Choose tools for the visual question and the hardware](https://ai.engineer/api/talk-essay-media/bc209d098b269d62ecec155667dcc1fa00b85815d8ff2b3f7dd22e4a136e1dcf)

Source frame: Choose tools for the visual question and the hardware

**Falcon Perception** illustrates why capability matters as much as size. The request “segment this red car” identifies an object by an attribute. A request for the red car next to the orange car that is next to the blue car requires resolving relationships before selecting the region. Noyan presents Falcon Perception as able to handle this open-ended referring segmentation, at 600 million parameters with an Apache 2.0 license.

Other tools cover distinct jobs:

- **Human-centered analysis:** Pose and related models support human keypoint detection and depth tasks.
- **Zero-shot detection:** Multiple detector sizes let developers choose a smaller alternative when hardware or speed requires it.
- **OCR:** Models are selected from OCR benchmark results in different sizes.
- **Depth estimation:** License terms are checked alongside capability; a model family should not be assumed to share one license across sizes.
- **Tracking:** Supervision and tracker support from Roboflow help follow instances and bounding boxes.

### When words cannot describe the part

[16:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=960s)

Industrial objects expose a limit of language-based labeling: some parts are difficult to describe in words. Noyan proposes image-guided detection as a possible next step. A reference image shows the object to find, and the detector searches for that object across other images. The example becomes the query, avoiding the need to invent a verbal description precise enough to distinguish the part. This is a planned direction, not a demonstrated industrial result.

![Source frame: When words cannot describe the part](https://ai.engineer/api/talk-essay-media/639f156f2fdca3da2d009e53c4e9e24bebf98003589053c71de8d38b78f962f8)

Source frame: When words cannot describe the part

A second proposed change would make judging spatial. Instead of asking a judge to accept or reject the labeler’s box, ask it to produce its own box and compare the two using intersection over union: the area shared by the boxes divided by the area covered by either box. That would expose agreement about location directly. Noyan also says segmentation support is in progress.

For further implementation work, Noyan points to a small vision repository covering fine-tuning, quantization, and multimodal models; Transformers task guides with tutorials; and Hugging Face Skills covering computer vision and infrastructure. These serve different needs: learning a model workflow, following a task tutorial, and giving an agent the instructions needed to run training.

The Q&A keeps the priority concrete. Training VLMs themselves in a self-improvement loop sounds exciting to Noyan, but first she wants developers to train task-specific models and deploy them at the edge. She also explains why the coding agent drafts judge prompts: it already has the workflow’s context. That context helps generate instructions; the earlier human review step still determines whether they describe the task correctly.

## Key takeaways

- Use VLMs to create and review supervision, then train a task-specific detector for the runtime application.
- Overlaying proposed boxes on images lets judges inspect the region alongside its label description. Keeping an example when either judge approves preserved training data in Noyan’s imbalanced-judge experiments.
- Human judgment remains necessary for class descriptions and augmentation choices: a mirrored sign or recolored traffic light can invalidate the original label.
- The document detector finding a signature missed by its labeling VLM illustrates generalization beyond an individual teacher prediction; it is a qualitative result without equivalent ground-truth evaluation.

## Resources

- [VibeVision / vision-intern](https://github.com/merveenoyan/vision-intern) — Shown on the closing resource slide.
- [Smol Vision](https://github.com/merveenoyan/smol-vision) — Shown on the closing resource slide.
- [Hugging Face Skills](https://github.com/huggingface/skills) — Shown on the closing resource slide.

## Slides

### Shortly about me

Merve Noyan wants developers to build vision apps using agents, while avoiding direct use of vision language models for simpler tasks. She offers the talk as a baseline to build on.

[0:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=27s) [The speaker introduces her computer vision work, a vision language models book, and her goal of helping developers build vision apps with agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/9161ef664a7ade51555b82fbff1f32345951e8a7e22fd5aeafd93810228c2933.webp)

Shortly about me

• I work on computer vision/multimodal (for past 3 years, since IDEFICS/Llava times) and lately on on-device/agents at Hugging Face
• I love vision language models so much that I even wrote a book on it (find me to get a copy!)
• But I don’t want people to directly use vision language models for simpler tasks
• I want every single developer to start building vision apps using agents
• Good baseline → build on it, get your hands dirty

Book cover:
O’REILLY
Vision Language Models
Building VLMs with Hugging Face
Merve Noyan, Miquel Farré, Andrés Marafioti & Orr Zohar
Foreword by Lucas Beyer

### Common behavior I observe

Noyan argues that VLMs can be costly and too slow for real-time, on-device tasks. She contrasts them with a model trained for a specific use case, alongside an RF-DETR demo detecting a person.

[1:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=69s) [The slide describes limitations of using vision language models for every task, alongside an RF-DETR person-detection example.](https://assets.ai.engineer/sites/aiecode2025/sha256/eb0d769fd12defdd52fbf5f4795f1e2b4e9b08e66a90a7a3cccab8ad2088762c.webp)

Common behavior I observe

• they use VLMs for everything because natural language is eez: VLMs are too large and costly for what you want to solve 99% of the time
• VLMs don’t give you real-time, low cost, low latency: you can’t process 30–40 fps on-device (toasters) (memory-bound)
• they’re not as robust as a model actually trained on a specific use case either

RF-DETR

Interface text:
LIVE STREAM
streaming
#0 person 0.96
Boxes
Masks
Confidence thr. 0.40
Start streaming
Stop
Streaming
22.9 fps
16.0s left

### Common pitfalls I observe with developers

Noyan points to repeated YOLO questions as a sign that developers may overlook model licenses, and urges them to consider Apache 2.0 alternatives.

[1:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=113s) [The speaker urges developers to check model licenses, with social media replies discussing object detection models.](https://assets.ai.engineer/sites/aiecode2025/sha256/03e130bfbdfbed7bdc6f8775eeba930127065e96fd724b5e5974ca8253800b2e.webp)

Common pitfalls I observe with developers

devs don’t check licenses: so many people use YOLO etc (lovely models but NC), if you want to use it you have to pay for it, we have more alternatives, easy to use with transformers
just because a model is open-source doesn’t mean you can use it!

Replying to @skalskip92 @GoogleAI and 2 others
Interesting. Is it better than fine tuned usual detection model? (e.g. YOLO)

Replying to @mervenoyann and @huggingface
Wow, I don't know that RF-DETR exist in HF. I want to learn my custom dataset learn on RF-DETR and compare with YOLO

Interface text:
1  2  262

### VibeVision

Noyan introduces VibeVision, which she built around letting agents code and ship computer vision projects.

[2:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=142s) [VibeVision title slide introduces building vision projects with agents, beside a camera-themed Hugging Face illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/4454f975d6608dd386579a532d26541bfa4a820c9c299877f551815523616a0e.webp)

VibeVision

// agentic computer vision

Build vision projects the agentic way: let the agents code and ship.

Hugging Face

detec[illegible]
agent

### Inspiration

A post inspired VibeVision: Gemma 4 calls SAM 3.1 to segment vehicles in a parking lot, finding 64 vehicles and then 23 white ones. The post says both models run locally on a MacBook.

[2:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=148s) [A social media post describes Gemma 4 calling SAM 3.1 to segment vehicles locally, illustrated by a parking lot with colored vehicle masks.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d6c4828829a3f74b92d303a46efa69fc442e4439f2615754f35e08604708bec.webp)

Inspiration

Maziyar PANAHI
@MaziyarPanahi

Gemma 4 looks at a parking lot. Decides what to ask. Calls SAM 3.1.

"Segment all vehicles." 64 found.
"Now just the white ones." 23 found.

One model reasoning and orchestrating. One model executing.

Both running locally on a MacBook. MLX. No cloud. No API.

Interface text:
0:10

### VibeVision: Distilling myself

Noyan selects models as tools for an agent, prioritizing licenses that allow commercial use. She also weighs performance on unsaturated benchmarks against model size—and checks the vibes.

[2:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=167s) [The speaker lists criteria for selecting models as tools: commercial-use licensing, benchmark performance relative to model size, and vibes.](https://assets.ai.engineer/sites/aiecode2025/sha256/43682c2083fe526acbf0b8a96b403db8654b1a7fb07baa680812531a3dc332d4.webp)

VibeVision: Distilling myself

• my favorite models as tools for different tasks
• I check following
  ◦ License (biggest prio): doesn’t have to be A2.0 and MIT, has to allow commercial use
  ◦ Performance on unsaturated benchmarks vs model size
  ◦ Vibes

### Building an object detection model

With labelled images, a developer or agent can train directly. With images alone, Noyan proposes using a VLM to label them, judging the labels, then training the model.

[3:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=214s) [VibeVision slide contrasts training from labeled images with labeling and judging unlabeled images before training.](https://assets.ai.engineer/sites/aiecode2025/sha256/43fa2a23ef394b3a47171caaa7370abd392bbea71e349f5f317b4205b2c099dd.webp)

VibeVision

I put myself in developers’ shoes, what is the easiest way to build an object detection model?

• If I have labelled images → just train (yourself or agent), easy
• If I have images only → label with VLM, judge labels, train model. How does the pipeline look like?

### VibeVision workflow and infrastructure

VibeVision uses VLMs to label images and judge labels before training a model. The workflow supports local inference and training, remote jobs, inference providers, and intermediate data storage.

[4:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=267s) [Slide outlines VLM labeling, VLM judging, training, and supporting infrastructure.](https://assets.ai.engineer/sites/aiecode2025/sha256/7529016dc1c619b4f2a2d65844fe0d1579084a9cc50de0d2b6cabd627b51ba66.webp)

VibeVision: Distilling myself

• I built a generic model training, agent-friendly workflow, works on different domains (for baseline to get you started!)
  ◦ VLM for labelling → VLMs-as-judges → training
• infra support
  ◦ local model inference/training
  ◦ Jobs for remote batch processing/training
  ◦ Inference Providers for labelling/judging
  ◦ Buckets to dump intermediate data

### What enables this work?

Long-horizon agents can oversee labeling and training, while smaller vision-language models reduce labeling costs. RF-DETR and refactored Transformers v5 support the workflow; RF-DETR-Seg is a work in progress.

[5:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=310s) [Slide lists long-horizon agents, RF-DETR, smaller vision-language models, and Transformers improvements.](https://assets.ai.engineer/sites/aiecode2025/sha256/886ba70c7e7638b2c869b6ece41bf9be5d6fd161579fc16ea3f877a1bab51839.webp)

What enables this work?

• We have better agents for long horizon tasks
• We owe this to RF-DETR, and for segmentation [WIP] RF-DETR-Seg
• Smaller yet more capable vision language models for labelling so we don’t pay a ton
• transformers DETRs are refactored + v5 is more robust (RF-DETR family is now available as well as many other models)

### VLM-labeled detection training pipeline

Qwen labels images, then Gemma and LFM judge the detections. A vote keeps agreed labels for RF-DETR training, while a coding agent supervises each stage.

[5:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=346s) [Diagram shows a coding agent orchestrating Qwen labeling, two VLM judges, merged judgments, and RF-DETR training.](https://assets.ai.engineer/sites/aiecode2025/sha256/a51726aa71cb95325520312255e7f0339cb17f65f186a520cc5f62f77fd1a420.webp)

vision-agent — VLM-labeled detection training pipeline

ORCHESTRATOR · SUPERVISES EVERY STAGE
Any coding agent of your choice
Drives & babysits the run — never labels, never judges.

Input data
HF dataset · DocVQA
Local dir · COCO

INFERENCE PROVIDERS · remote
1 Label · detect
label_qwen.py
LABELER · VLM
Qwen/Qwen3.5-9B
Qwen · 9.65B
Detects objects → boxes + class labels.

Labeled dataset
+ detections

HF JOBS · managed compute
2 Judge A · JUDGE
judge_one.py
google/gemma-4-E4B-it
Google · 8.0B

VLM-AS-JUDGE ENSEMBLE · SCORES 0-1

2 Judge B · JUDGE
judge_one.py
LiquidAI/LFM2.5-VL-1.6B
Liquid AI · 1.6B

3 Merge · ensemble
merge_judges.py
MODEL
— ensemble vote logic
cpu-upgrade
Vote across judges → keep agreed detections.

Judged dataset
filtered + verdicts

4 Train · fine-tune
train_rfdetr_job.py
DETECTOR
Roboflow/rf-detr-base
RF-DETR
GPU
Fine-tunes the object detector on kept labels.

Trained RF-DETR checkpoint

Labelling with Qwen3.5-9B
LFM2.5-VL and Gemma-4 as judges
merge judgements
train RF-DETR

### Vibe-training pipeline

A coding agent can run the labeling, judging and training pipeline from a dataset prompt. The example pauses for approval of the generated judge descriptions; already-labeled data can go straight to training.

[6:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=412s) [Slide explains prompting a coding agent to run annotation and training, with a sample exchange about detecting soda bottles.](https://assets.ai.engineer/sites/aiecode2025/sha256/433baf5f23d58c757fa7f222cd80d817db39b23b66482db5fa6b104c1e1fac08.webp)

Vibe-training Pipeline

• Just clone the repository and prompt “Run the annotation and training in this project on username/objects dataset to detect objects” to your coding agent

› can you run e2e pipeline in this repository on this dataset
  where you need to detect soda bottles? Francesco/soda-bottles
  use recommended setup

• I'll run the e2e pipeline using the dedicated vision-e2e-runner agent, which handles the label → judge → train flow and halts at the one approval gate (the generated judge descriptions).

### Models used in the pipeline

Noyan uses Qwen3.5-9B for one-off labelling and smaller models as judges. RF-DETR handles detection; the agent workflow was built with Opus 4.8 and then run with GLM-5.2.

[8:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=482s) [Slide lists the labelling, judging, detection and agent models used in the pipeline.](https://assets.ai.engineer/sites/aiecode2025/sha256/637f53e4908dc5b7a43e974e5ff1ab4ff786d550c7dc1473fb38de2b9db04d16.webp)

Models used in pipeline

• Multiple smaller (than labeller) judges perform well (based on research and common sense)
• Qwen3.5-9B: 9B model, ~19 GB for one-off labelling
• Gemma-4-E4B-it: 8B params, 15.1 GB at BF16
• LFM2.5-VL-1.6B: edge focused dense model ~1.6B params, takes 2.34 GB in bf16
• RF-DETR-medium: 33.7M params, sota real-time detection model by Roboflow, comes in various sizes, DINOv2 backbone
• For agents, I initially built with Opus 4.8 and then ran the workflow with GLM-5.2

*All Apache 2.0 licensed except LFM2.5-VL

### Infrastructure and costs

Noyan estimates ~$3 to process 1.4k images end to end: DeepInfra labels them for $0.16, parallel HF Jobs run the judges, and L4 training costs about $1. Together is faster but costs more.

[8:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=518s) [Slide summarizes infrastructure and estimated costs for labelling, judging and training, with an inference provider comparison.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fce8b7bc6f8197893d3dfffc6f7d540b9b7f7d16aa9dffb7ed7e804c1fe5f47.webp)

Infra

• note: I work at HF, have a ton of creds and I’m impatient, but it costs ~$3 to run e2e (1.4k images)
• Qwen3.5-9B on Inference Providers, serverless
  ◦ Labelling 1.4k images: $0.16 (with DeepInfra, if you want faster use Together)
• Gemma-4 and LFM2.5-VL on parallel HF Jobs (batch inference charged hourly)
  ◦ I put Gemma-4 to L40S (~$1, ~$1.8/hr) (wallclock, would fit to L4 with bsz = 8)
  ◦ LFM to L4 ($0.47, ~$0.8/hr)
• Training (again, I’m impatient): l4 (~$1)

Model | Provider | Input $/1M | Output $/1M | Context | Latency(s)
Qwen/Qwen3.5-9B | together (fastest) | $0.17 | $0.25 | 262,144 | 0.35
Qwen/Qwen3.5-9B | ovhcloud | $0.12 | $0.18 | 262,144 | 0.40
Qwen/Qwen3.5-9B | deepinfra (cheapest) | $0.10 | $0.15 | 262,144 | 1.95

### VibeVision test problems

Noyan compared road-sign detection against existing ground-truth labels. For document parsing, she wanted to extract images, tables and signatures from DocVQA documents.

[9:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=578s) [Slide identifies road sign detection and document parsing as test problems.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e881f8ef16cad558613678735947221250beb4f638aa86048d6f93865b85b0b.webp)

VibeVision

Tested on various problems
• Roadsign detection: ground-truth labels available, compared against ground truth
• Document parsing: extracting non-text media from DocVQA dataset

### First road sign detection results

The road-sign detector scores 0.9693 mAP@50 against Qwen pseudo-labels and 0.8028 against human annotations. Its ROC-AUC against human ground truth is 0.987.

[10:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=613s) [Three road sign detection examples accompany mAP at 50 and ROC-AUC results.](https://assets.ai.engineer/sites/aiecode2025/sha256/da919f4133d68d1054f652da1ec9fd673d61d8554af9c9f0d421f446687e6cbb.webp)

First run results: it works!

railway_crossing 0.73
T_intersection | 0.53
do_not_turn | 0.50
do_not_turn | 0.53

tldr; mAP is “how good is the detector”, ROC-AUC measures "do the model's confidence scores rank true detections above false ones"

• pseudo-labels by Qwen (judged): map@50=0.9693
• human GT annotations: map@50=0.8028
• ROC-AUC (human GT): 0.987

### Document image detection comparison

The model’s three image boxes closely match the Qwen3.5 annotations on the same apparel page, including the two photographs below the main fashion image.

[11:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=678s) [Side-by-side document pages compare annotation boxes with model output boxes around photographs.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ad5afffcbcdcec18b9daae0c608f0d90c1075bbd1332796e134e26d6e6c67eb.webp)

Qwen3.5 annotation

Lifestyle Apparel
WILLS LIFESTYLE

Synonymous with elegance and style that is effortless and chic, Wills Lifestyle presents a premium fashion wardrobe for men and women. With a tempting choice of Wills Classic formal wear, Wills Sport relaxed wear, Wills Clublife evening wear and Wills Signature designer wear, Wills Lifestyle offers a truly delightful shopping experience for the discerning customer.

WILLS CLASSIC
WILLS SPORT
WILLS clublife
WILLS SIGNATURE

johnplayers jeans

johnplayers jeans is the new language for the fashion-forward youth, featuring an innovative collection of denims and casual wear in edgy styling, bold patterns and contemporary fits.

JOHN PLAYERS

John Players embodies the spirit of the modern youth that is playful, fashionable and cool. John Players presents a complete wardrobe incorporating the most contemporary trends in an exciting mix of colours, styling and fits.

Model Output

Lifestyle Apparel
WILLS LIFESTYLE

Synonymous with elegance and style that is effortless and chic, Wills Lifestyle presents a premium fashion wardrobe for men and women. With a tempting choice of Wills Classic formal wear, Wills Sport relaxed wear, Wills Clublife evening wear and Wills Signature designer wear, Wills Lifestyle offers a truly delightful shopping experience for the discerning customer.

WILLS CLASSIC
WILLS SPORT
WILLS clublife
WILLS SIGNATURE

johnplayers jeans

johnplayers jeans is the new language for the fashion-forward youth, featuring an innovative collection of denims and casual wear in edgy styling, bold patterns and contemporary fits.

JOHN PLAYERS

John Players embodies the spirit of the modern youth that is playful, fashionable and cool. John Players presents a complete wardrobe incorporating the most contemporary trends in an exciting mix of colours, styling and fits.

Detection box labels: [illegible]

### Judging prompt generation

Generated judge descriptions need human approval: prompts for simple labels can be too long, while descriptions of specific things can fall short. The slide shows a verbose soda-bottle description.

[12:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=739s) [A slide describes limitations of generated judge descriptions and shows an example approval gate.](https://assets.ai.engineer/sites/aiecode2025/sha256/ac816291e6ac00ffbc61c9eb1314eeb86e77aa9da7eeffa660439027afe518c2.webp)

Judging Prompt Generation (Metaprompt)

• Asking LLM to generate labels for judges can sometimes backfire depending on class labels in the dataset repository
• We leave it to human to approve label descriptions, e.g. for simple things prompt gets too long, for specific things it falls short

Approval gate: generated judge description
{
  "soda bottle": "a glass or plastic beverage container featuring a cylindrical body that tapers into a narrower neck, typically topped with a metal or plastic cap and often wrapped with a paper or plastic label displaying brand text and logos"
}

### Augmentation

A coding agent flipped road signs and applied color jitter to traffic lights, risking corrupted training data. Noyan patched the augmentation behavior later.

[12:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=766s) [A slide warns that flipping road-sign images and applying color jitter to traffic lights can damage training data.](https://assets.ai.engineer/sites/aiecode2025/sha256/3a692a1ce894bab4d6dec63532ef7f91acb90dc3730aec44e9b8aa114e0e2495.webp)

Augmentation

• Sometimes coding agents augment the dataset where it shouldn’t, I have patched this later
• Example: roadsigns 🚦 🚧
  ◦ the training pipeline flips the images horizontally/vertically which damages “turn right/left”
  ◦ applies color jitter to traffic lights 😁

### Falcon-Perception for segmentation

Noyan recommends 600M-parameter Falcon-Perception for segmentation, saying it can follow positional descriptions such as a red box next to an orange box that SAM cannot.

[14:15](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=855s) [A slide presents Falcon-Perception and compares its segmentation examples with SAM3 across three box prompts.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2374d8aa4229e43178931119c3260f350f401db1a6755a61d4dfdab6ef23c8d.webp)

My preferred models as tools

• SAM cannot do open-ended referral expression segmentation, e.g. it can segment “red box” but cannot do “red box on the left next to orange box”
• Enter Falcon-Perception, 600M model with A2.0 license

Comparison columns: Box; Purple box; ACME box
Comparison rows: Falcon Perception; SAM3

### Future plans and disclaimers

The pipeline provides a baseline, but industry objects can be hard to describe in natural language. Image-guided detection could help: the illustration uses one visual example to find three matches.

[15:58](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=958s) [Pipeline limitations alongside an illustration of image-guided detection using a visual example.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e852723d630a3f623825ac5e40263e5d4e4feaf9cb0550eea6988e30383d084.webp)

Future plans and disclaimers

• This pipeline creates a good baseline, no hyperparameter sweep, industry use cases are hard to tackle with natural language
• This pipeline can fall short for industry use cases with judges where object is hard to describe, less known
• Image guided detection could help with this

image-guided detection
1 example → 3 matches
visual prompt
example
guides
target image
match 0.97
match 0.93
match 0.86
one-shot · detect by visual example, no labels needed

### Thanks for listening

Noyan points to smol-vision for model fine-tuning and quantization, Transformers task guides, Hugging Face skills, and the vision-intern repository.

[17:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1030s) [Closing slide listing project and documentation resources beside a smol vision illustration.](https://assets.ai.engineer/sites/aiecode2025/sha256/17df357142aee4da1cc3f2c239bc411335e6d0c93a474c6651701463574ae818.webp)

Thanks for listening!

More resources (for humans and agents):
• github: merveenoyan/smol-vision
• hf.co/docs/transformers/en/tasks/object_detection (and other task recipes)
• github: huggingface/skills
• @mervenoyann on X
• repo is at merveenoyan/vision-intern

smol vision

### AI Engineer World's Fair

[19:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1143s) [AI Engineer World's Fair title card in white on black.](https://assets.ai.engineer/sites/aiecode2025/sha256/7322b35589cf668e9a862fbaf40cd3386acc3b0b1d926a952438049203a9508f.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=13s) Hello and welcome to this talk on skill

[0:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=16s) issue. It's actually no longer a skill

[0:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=19s) issue. So uh by the end of this talk you

[0:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=22s) will be able to build a lot with the

[0:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=25s) vision models if you are already not

[0:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=26s) doing it. So shortly about me, I'm Marv.

[0:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=30s) Uh I've been working on computer vision

[0:33](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=33s) since Lava in the fix times. Uh and

[0:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=37s) lately I work on more agents and

[0:39](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=39s) ondevice stuff because I'm a bit

[0:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=40s) fascinated and I love vision language

[0:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=43s) models so much that I even wrote a book

[0:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=45s) on it. But I don't want developers to

[0:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=47s) directly use vision language models

[0:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=49s) anymore. And uh I want every single

[0:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=52s) developer to start uh building vision

[0:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=56s) langu vision applications end to end.

[0:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=59s) And this talk is going to give you like

[1:01](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=61s) a good baseline to do so.

[1:04](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=64s) And common behaviors I observe with the

[1:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=67s) developers is the fact that they try to

[1:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=69s) use vision language models for

[1:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=70s) everything but they you will never get

[1:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=74s) real time. And when I mentioned real

[1:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=76s) time, it's like you have a toaster and

[1:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=79s) you get like 30 40 fps uh on it like

[1:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=82s) whatever whether you are building like

[1:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=84s) image classification, instance

[1:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=86s) segmentation or whatever.

[1:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=89s) And um they are not super robust in

[1:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=92s) terms of like if you were to train like

[1:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=94s) an RFDTR which Joseph spoke in the I

[1:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=97s) spoke about in the first talk um it will

[1:41](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=101s) always outperform your vision language

[1:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=103s) model and I'm going to prove it today.

[1:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=107s) And uh on the right hand side you can

[1:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=109s) actually see me uh doing stuff with

[1:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=111s) RFDTR.

[1:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=113s) Um and another thing is that they don't

[1:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=116s) read the licenses. Everybody's like

[1:58](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=118s) whenever I post something about object

[2:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=120s) detection, they always ask me about

[2:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=122s) YOLO. Like YOLO is a good model. Uh but

[2:04](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=124s) it's a it has AGPL 3.0 license I think.

[2:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=128s) And I could swear on my life that there

[2:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=130s) is like some developers that actually

[2:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=131s) deploy it without knowing that they have

[2:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=133s) to pay for it. Um so yeah like uh I want

[2:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=138s) you to migrate to Apache 2.0 models

[2:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=142s) today.

[2:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=144s) And for this I have built something

[2:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=146s) called web vision and it's kind of

[2:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=148s) inspired by this post by Mazar.

[2:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=150s) Basically what he does is that he gives

[2:33](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=153s) um SAM 3.1 model as a tool to Gemma 4 to

[2:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=157s) call and I find it super impressive. Um

[2:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=160s) and today I built like a tool kit uh

[2:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=164s) where you can do this with like even

[2:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=166s) more things.

[2:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=169s) So I'm kind of distilling myself. Um

[2:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=172s) first off uh this toolkit has my

[2:55](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=175s) favorite models as tools uh so that you

[2:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=177s) can give it to your agent because your

[2:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=179s) coding agent is a bit of a clueless

[3:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=182s) computer vision engineer basically. Um

[3:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=185s) and when I distill myself basically like

[3:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=187s) whenever I pick a model I always check

[3:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=189s) the following. So first up the license

[3:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=191s) is the biggest priority for me. It has

[3:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=194s) to be Apache 2.2 MIT or something

[3:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=196s) non-commercial. Uh secondly uh the

[3:20](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=200s) performance has to be on par uh

[3:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=202s) depending on its size or architectural

[3:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=204s) choices. So I check from the benchmarks

[3:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=208s) uh whenever a model from a computer

[3:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=210s) vision conference comes out and third

[3:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=212s) vibes obviously

[3:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=215s) and so this toolkit has a second part to

[3:39](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=219s) it and it's a bit like a vibe training

[3:41](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=221s) part which is the most exciting part. So

[3:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=224s) I will go through it first. So I put

[3:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=227s) myself in developers shoes to build like

[3:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=229s) a vision application. Um if I have

[3:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=232s) labeled images easy I can just train a

[3:55](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=235s) model or like if I I can give my

[3:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=237s) computer computer vision agent some

[4:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=240s) tutorials uh to do that because it's all

[4:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=242s) out there like we built transformers for

[4:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=245s) it. Uh but if I have images only I have

[4:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=248s) to annotate uh and then uh evaluate the

[4:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=251s) annotations and then just train a model.

[4:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=254s) But how can you do it on scale? You can

[4:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=256s) actually use a vision language model as

[4:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=258s) a labeler and the vision language models

[4:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=261s) as judge and then train what you want.

[4:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=264s) But how does this pipeline look like?

[4:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=268s) So basically I built this and it has

[4:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=272s) like it has VLM for labeling VLM as a

[4:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=275s) judge and then training. Uh it's a bit

[4:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=278s) of a long horizon tasks for coding

[4:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=280s) agents and it has infra many

[4:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=282s) infrastructure support. You can do this

[4:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=284s) locally. You can do this uh remotely. So

[4:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=287s) basically it runs on hugging face

[4:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=288s) infrastructure. We have jobs that allow

[4:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=291s) you to do oneoff batch processing or

[4:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=294s) training. We also have like a serverless

[4:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=297s) routing system called inference

[4:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=299s) providers where you can use multiple

[5:01](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=301s) providers and you we also have like

[5:04](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=304s) buckets to dump intermediate data on top

[5:06](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=306s) of the data set repositories, model

[5:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=308s) repositories and so on.

[5:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=311s) Um but what enables this work? First

[5:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=314s) off, my favorite model, RFDTR, RFDTR

[5:17](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=317s) segmentation. I'm currently working on

[5:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=319s) segmentation at the moment. Um, we have

[5:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=322s) better agents for long horizon tasks.

[5:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=325s) Uh, that, you know, you have to babysit

[5:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=327s) the labeling process, the training

[5:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=330s) process, etc. Uh, and smaller yet more

[5:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=334s) capable vision models enable you to

[5:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=336s) label stuff very cheaply. And also with

[5:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=340s) transformers we went for V5 refactoring

[5:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=343s) etc. So like it performs better for

[5:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=345s) vision models at the moment.

[5:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=347s) And this is how the pipeline actually

[5:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=349s) looks like. So first up I label the data

[5:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=353s) set like I take an image data set like

[5:55](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=355s) any image data set. I label that image

[5:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=357s) data set with Q1 3.59B

[6:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=360s) and then I pass the labeled data set to

[6:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=363s) two judges. First one is Gemma 4 E4B

[6:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=367s) which is like an 8B judge and then the

[6:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=370s) second one is LFM 2.5VL which is nearly

[6:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=373s) 2B uh it's relatively smaller basically

[6:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=376s) I checked the research and it's better

[6:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=378s) to have like an ensemble smaller judges

[6:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=381s) and on top of it I merged the judgments

[6:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=384s) so I also checked the research around it

[6:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=387s) and most of the people ask the VLM or

[6:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=390s) LLM to assign some score to it but those

[6:33](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=393s) scores absolutely absolutely don't work

[6:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=395s) especially if your models are of

[6:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=396s) different size with judges. Um and then

[6:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=400s) I pass it to train RFDTR medium or

[6:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=403s) large. So I I chatted a bit with Rob

[6:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=406s) people and they encouraged to use that

[6:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=409s) and I'm going it it actually works. I'm

[6:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=411s) going to show you very shortly. U but

[6:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=414s) how does it work? So you take the

[6:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=416s) repository and then you just ask it okay

[6:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=419s) can you train it? Can you do the

[7:01](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=421s) training on this data set on hub? And

[7:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=425s) then it will start like if if the data

[7:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=428s) set has labels like you can actually

[7:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=430s) just get to training but if it doesn't

[7:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=433s) have it you can just uh start annotating

[7:17](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=437s) and I basically like the trick is I I

[7:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=441s) pass the overlaid bounding boxes on

[7:23](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=443s) images to the judge. So basically Qan

[7:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=446s) technically outputs bounding boxes as

[7:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=448s) tokens. I don't pass them. I just

[7:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=450s) overlay the bounding boxes and I pass

[7:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=452s) that image on top of some labels and

[7:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=455s) label descriptions and I say okay if

[7:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=458s) this label description has bounding box

[7:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=460s) on it then just um tell me if you

[7:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=464s) approve or not and then the judge I I

[7:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=467s) merge the judge verdicts over minimum

[7:50](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=470s) agreement and not consensus which I will

[7:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=473s) come to why I did that that way and

[7:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=476s) these label descriptions are also

[7:58](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=478s) generated by coding agents. and you just

[8:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=480s) approve it as a human being.

[8:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=483s) And the the models that I used in this

[8:06](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=486s) pipeline all have Apache 2.0 of licenses

[8:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=489s) except for LFM model which has type of

[8:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=493s) license where you are kind of um if you

[8:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=496s) have like certain amount of revenue

[8:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=499s) after that you pay for it but you can

[8:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=501s) comfortably use it it's large and for

[8:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=505s) the coding agents that are babysitting

[8:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=506s) this pipeline uh I initially built with

[8:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=509s) oppus 4.6 uh 8 and then ran the workflow

[8:33](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=513s) with the G gl GLM 5.2 2 which is doing a

[8:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=516s) good job on long horizon tasks to be

[8:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=518s) honest

[8:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=520s) and for infra I actually work at hugging

[8:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=522s) face I have a lot of compute credits and

[8:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=525s) I'm super impatient in life so like uh I

[8:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=529s) use a good amount of hardware for

[8:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=531s) experimentations but I benchmarked it

[8:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=534s) and overall it takes like three four

[8:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=537s) dollars if you want to run this entire

[8:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=539s) pipeline to train models which to me is

[9:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=542s) crazy initially for Q1 3.5 5 I used

[9:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=545s) serverless because I was like okay this

[9:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=548s) is convenient and it's super cheap so I

[9:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=550s) use deep infra which is super super

[9:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=553s) cheap if you were to use together it's

[9:15](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=555s) better if you do batch processing over

[9:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=558s) jobs and then for the judging I used

[9:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=561s) hugging face jobs which cost less and

[9:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=564s) then for training again I used like an

[9:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=567s) L4 but the model is super small like

[9:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=569s) RFDTR is super small and you can just

[9:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=572s) use something else you can do it locally

[9:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=574s) if you wanted to. I'm just impatient. I

[9:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=576s) want a big batch size. So yeah,

[9:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=580s) and I tested in two problems. First of

[9:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=582s) the road sign detection. Uh secondly,

[9:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=586s) document parsing. For the road sign

[9:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=588s) detection one, I already have the

[9:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=589s) labels. So like I actually compared

[9:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=592s) against the ground truth annotations if

[9:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=594s) my pipeline works or not. And for

[9:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=596s) document parsing, I actually uh couldn't

[9:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=599s) do it because basically I use like a

[10:01](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=601s) docqa data set and I the problem is like

[10:04](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=604s) I want to extract the images, tables, I

[10:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=607s) don't know signatures and stuff. So it's

[10:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=609s) like a novel task and I wanted to see if

[10:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=611s) RFDTR can actually learn it.

[10:15](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=615s) So first result it works. So yay. um we

[10:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=619s) have like a good uh mean average

[10:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=621s) precision over um 50 um and I compare it

[10:26](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=626s) against the basically I have a test set

[10:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=629s) and I take that test set and pass

[10:31](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=631s) through QN and then I compare against

[10:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=634s) the pseudo notations and the ground root

[10:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=637s) annotations of that test set there is a

[10:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=640s) bit of a gap but it's kind of expected

[10:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=642s) because it's learned from QN and also uh

[10:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=645s) rock oak is also like a good value to be

[10:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=648s) frank for like such use case. Um and for

[10:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=653s) the document parsing it actually

[10:55](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=655s) generalizes which to me is crazy.

[10:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=657s) Basically uh here the trained model

[11:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=660s) output you can see that it detected the

[11:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=663s) signature. Meanwhile the QN annotation

[11:06](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=666s) of that test set missed it. So I would

[11:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=669s) like to say it actually generalizes very

[11:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=671s) well as well. uh but we owe this to like

[11:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=674s) the how good RFDTR is as a backbone in a

[11:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=678s) way

[11:20](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=680s) and here you can also see how it uh

[11:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=682s) captures the images technically

[11:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=685s) um and it's like oneonone

[11:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=688s) and while I was building this I actually

[11:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=690s) noticed that I was clueless about

[11:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=692s) building with vision agents um so I have

[11:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=695s) bunch of findings around it um so first

[11:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=698s) up there is a huge judge imbalance so

[11:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=700s) for depending on the problem. LFM tends

[11:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=704s) to reject a lot. That's why I couldn't

[11:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=706s) take the consensus because if I were to

[11:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=708s) if I were to eliminate everything that

[11:50](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=710s) both LFM and Gemma agreed to remove, um

[11:55](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=715s) I would left with very very little

[11:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=717s) number of examples which would leave me

[11:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=719s) with very poor generalization. So what I

[12:01](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=721s) did was that okay, if one of them says

[12:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=723s) yes, I'm going to take that example and

[12:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=725s) nevertheless it worked well. If you have

[12:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=727s) a large data set and if you care about

[12:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=729s) the recall, I suggest that you take the

[12:13](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=733s) consensus or like just observe for

[12:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=736s) document parsing the gap isn't as big.

[12:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=741s) And secondly, uh the prompt generation

[12:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=744s) is a bit hard. So like this is the only

[12:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=747s) part where as a human you have to

[12:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=749s) approve uh okay the model generates the

[12:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=752s) prompts for you for the judge uh and

[12:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=755s) then you will say okay this I I approve

[12:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=757s) this because you need to to take it

[12:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=760s) still take a look at it uh take a look

[12:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=762s) at your data set a little there's no

[12:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=764s) escaping that

[12:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=766s) um and thirdly and this is super

[12:50](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=770s) interesting because your co your coding

[12:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=773s) agent despite no matter how uh good it

[12:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=776s) is. Like you take OPUS 4.8 which is like

[12:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=779s) a very good coding agent. Um it's

[13:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=782s) clueless as a computer vision engineer

[13:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=785s) as well as it misses common sense. For

[13:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=787s) instance like it was doing like

[13:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=788s) horizontal flip over the traffic signs

[13:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=791s) or it was doing like jitter over the

[13:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=794s) traffic lights which will definitely

[13:17](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=797s) corrupt your data set and break it. So I

[13:20](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=800s) patched this later. So you can just say

[13:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=802s) okay I don't want to augment or I can

[13:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=805s) augment. So Angular coding agent will

[13:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=807s) help you with that.

[13:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=810s) And lastly the second part of this

[13:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=812s) toolkit is my preferred models as tools.

[13:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=815s) Um so this repository covers my favorite

[13:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=818s) models from depth estimation to zero

[13:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=820s) shot segmentation.

[13:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=822s) Uh and this is partially powered by

[13:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=824s) first of hugging face benchmarks uh

[13:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=826s) which we recently rolled out maybe a few

[13:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=828s) months ago. Basically every single

[13:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=831s) basically we have like a benchmark

[13:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=834s) leaderboard and over there you have the

[13:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=836s) open models as well as their evaluation

[13:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=839s) results and you can compare different

[14:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=842s) models of different sizes. So um I keep

[14:06](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=846s) it up to date as well but like also it's

[14:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=849s) partially powered by me who likes to

[14:11](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=851s) read the um computer vision conference

[14:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=854s) papers.

[14:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=856s) Um, so I would like to give a shout out

[14:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=859s) to this model because not a lot of

[14:20](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=860s) people know about this. So basically Sam

[14:23](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=863s) cannot do open-ended reference se uh

[14:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=865s) segmentation like you can do like okay

[14:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=869s) segment this red car and it will do it.

[14:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=872s) But if you say okay the red car next to

[14:34](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=874s) the orange car that is next to the blue

[14:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=876s) car it will not do that. and Falcon

[14:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=878s) perception which is a model by TIA um

[14:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=882s) can actually do it and it's only like

[14:44](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=884s) 600 million parameters with Apache 2.0

[14:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=886s) license. So this one does the zero shot

[14:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=889s) segmentation for me and this is like a

[14:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=893s) non-exhaustive list. So for posing we

[14:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=896s) have the sapiance family um for the

[14:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=899s) human centric tasks where you need to do

[15:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=902s) human koid detection human um depth

[15:04](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=904s) estimation and so on and for zero shot

[15:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=908s) detection uh I have moon dream 3 and mm

[15:12](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=912s) grounding dynino which is like a model

[15:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=914s) with apache 2.0 license is also very

[15:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=916s) good. It's very small compared to Moon

[15:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=918s) Dream. I give you the multiple models in

[15:22](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=922s) multiple sizes depending on your

[15:23](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=923s) hardware that you can pick like if you

[15:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=925s) want to go fast just pick the tiny

[15:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=928s) alternative.

[15:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=929s) Um for OCR I actually took them from the

[15:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=932s) OCR benchmark in different sizes and for

[15:36](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=936s) depth estimation I actually uh

[15:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=938s) discovered that the large model doesn't

[15:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=940s) have a non-commercial license and the

[15:42](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=942s) rest of them have it. So you can

[15:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=943s) actually use it. That one has like an

[15:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=945s) Apache 2.0 license. And it also comes

[15:48](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=948s) with um supervision and tracker support.

[15:50](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=950s) They are both uh libraries from Roboflow

[15:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=952s) that um allow you to do tracking of

[15:56](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=956s) instances, bounding boxes and so on

[16:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=960s) and future plans. So first off, I I

[16:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=963s) could hear you say, okay, this will

[16:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=965s) definitely not work for the industry use

[16:07](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=967s) cases because industry use cases have

[16:10](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=970s) different parts. it has like

[16:12](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=972s) nondescribable parts like natural

[16:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=974s) language is not a good gateway to it. Um

[16:18](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=978s) so I think in that sense image guided

[16:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=981s) detection could help like if you don't

[16:23](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=983s) know about image guided detection you

[16:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=984s) basically have like instance of an image

[16:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=987s) like a haggi here as an example and then

[16:30](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=990s) you ask the model okay detect this

[16:33](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=993s) object in this image um across all of

[16:37](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=997s) the images. I feel like it could

[16:39](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=999s) actually somewhat help in the industry

[16:40](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1000s) use cases where you cannot describe it

[16:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1003s) by like um natural language.

[16:46](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1006s) Um as well as I want to try um

[16:51](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1011s) intersection over union merger sort of

[16:54](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1014s) basically you have like labeled boxes

[16:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1017s) and then judges boxes like you ask the

[17:00](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1020s) judge to actually generate a box and

[17:02](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1022s) then you take the intersection over

[17:03](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1023s) union instead of asking judge to reject

[17:06](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1026s) or accept and I'm working currently

[17:09](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1029s) working on the segmentation support

[17:12](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1032s) and thanks for listening if you want to

[17:14](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1034s) actually learn more so basically I We

[17:16](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1036s) have a small vision repository. It has

[17:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1039s) everything about like fine-tuning

[17:21](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1041s) models, quantizing models, multimodel

[17:24](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1044s) models, everything around vision. Uh as

[17:27](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1047s) well as transformers task guides. We

[17:29](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1049s) keep them up to date has many um

[17:32](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1052s) tutorials. Um, we also have hugging face

[17:35](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1055s) skills that actually have computer

[17:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1058s) vision specific skills as well as the

[17:41](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1061s) infras skills that you can just do like

[17:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1063s) a one prompt training. Again, um, and

[17:47](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1067s) this is my Twitter profile and this repo

[17:50](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1070s) is actually at GitHub Marvin uh, vision

[17:53](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1073s) intern. I think I have qu um, time for

[17:57](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1077s) one question.

[17:59](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1079s) Thank you so much.

[18:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1085s) Yes,

[18:05](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1085s) >> you have plans for training.

[18:08](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1088s) >> Um, he's asking if I have uh plans to

[18:12](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1092s) train train VLMs themselves like a

[18:15](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1095s) self-improvement type of thing. That

[18:17](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1097s) would be super exciting. But first, I

[18:19](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1099s) want to solve this thing of like

[18:20](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1100s) developers actually training task

[18:23](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1103s) specific models and then deploying on

[18:25](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1105s) edge and then that could come perhaps.

[18:28](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1108s) Maybe one more. Yes.

[18:38](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1118s) >> Um, not really. I don't think so. I just

[18:41](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1121s) use the because I wanted because a

[18:43](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1123s) coding agent actually has the context. I

[18:45](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1125s) wanted it to generate the prompt.

[18:49](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1129s) Maybe one more.

[18:52](https://www.youtube.com/watch?v=dKcTBQzR7jI&t=1132s) Okay. Thank you so much.