Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

Hello and welcome to this talk on skill

issue. It's actually no longer a skill

issue. So uh by the end of this talk you

will be able to build a lot with the

vision models if you are already not

doing it. So shortly about me, I'm Marv.

Uh I've been working on computer vision

since Lava in the fix times. Uh and

lately I work on more agents and

ondevice stuff because I'm a bit

fascinated and I love vision language

models so much that I even wrote a book

on it. But I don't want developers to

directly use vision language models

anymore. And uh I want every single

developer to start uh building vision

langu vision applications end to end.

And this talk is going to give you like

a good baseline to do so.

And common behaviors I observe with the

developers is the fact that they try to

use vision language models for

everything but they you will never get

real time. And when I mentioned real

time, it's like you have a toaster and

you get like 30 40 fps uh on it like

whatever whether you are building like

image classification, instance

segmentation or whatever.

And um they are not super robust in

terms of like if you were to train like

an RFDTR which Joseph spoke in the I

spoke about in the first talk um it will

always outperform your vision language

model and I'm going to prove it today.

And uh on the right hand side you can

actually see me uh doing stuff with

RFDTR.

Um and another thing is that they don't

read the licenses. Everybody's like

whenever I post something about object

detection, they always ask me about

YOLO. Like YOLO is a good model. Uh but

it's a it has AGPL 3.0 license I think.

And I could swear on my life that there

is like some developers that actually

deploy it without knowing that they have

to pay for it. Um so yeah like uh I want

you to migrate to Apache 2.0 models

today.

And for this I have built something

called web vision and it's kind of

inspired by this post by Mazar.

Basically what he does is that he gives

um SAM 3.1 model as a tool to Gemma 4 to

call and I find it super impressive. Um

and today I built like a tool kit uh

where you can do this with like even

more things.

So I'm kind of distilling myself. Um

first off uh this toolkit has my

favorite models as tools uh so that you

can give it to your agent because your

coding agent is a bit of a clueless

computer vision engineer basically. Um

and when I distill myself basically like

whenever I pick a model I always check

the following. So first up the license

is the biggest priority for me. It has

to be Apache 2.2 MIT or something

non-commercial. Uh secondly uh the

performance has to be on par uh

depending on its size or architectural

choices. So I check from the benchmarks

uh whenever a model from a computer

vision conference comes out and third

vibes obviously

and so this toolkit has a second part to

it and it's a bit like a vibe training

part which is the most exciting part. So

I will go through it first. So I put

myself in developers shoes to build like

a vision application. Um if I have

labeled images easy I can just train a

model or like if I I can give my

computer computer vision agent some

tutorials uh to do that because it's all

out there like we built transformers for

it. Uh but if I have images only I have

to annotate uh and then uh evaluate the

annotations and then just train a model.

But how can you do it on scale? You can

actually use a vision language model as

a labeler and the vision language models

as judge and then train what you want.

But how does this pipeline look like?

So basically I built this and it has

like it has VLM for labeling VLM as a

judge and then training. Uh it's a bit

of a long horizon tasks for coding

agents and it has infra many

infrastructure support. You can do this

locally. You can do this uh remotely. So

basically it runs on hugging face

infrastructure. We have jobs that allow

you to do oneoff batch processing or

training. We also have like a serverless

routing system called inference

providers where you can use multiple

providers and you we also have like

buckets to dump intermediate data on top

of the data set repositories, model

repositories and so on.

Um but what enables this work? First

off, my favorite model, RFDTR, RFDTR

segmentation. I'm currently working on

segmentation at the moment. Um, we have

better agents for long horizon tasks.

Uh, that, you know, you have to babysit

the labeling process, the training

process, etc. Uh, and smaller yet more

capable vision models enable you to

label stuff very cheaply. And also with

transformers we went for V5 refactoring

etc. So like it performs better for

vision models at the moment.

And this is how the pipeline actually

looks like. So first up I label the data

set like I take an image data set like

any image data set. I label that image

data set with Q1 3.59B

and then I pass the labeled data set to

two judges. First one is Gemma 4 E4B

which is like an 8B judge and then the

second one is LFM 2.5VL which is nearly

2B uh it's relatively smaller basically

I checked the research and it's better

to have like an ensemble smaller judges

and on top of it I merged the judgments

so I also checked the research around it

and most of the people ask the VLM or

LLM to assign some score to it but those

scores absolutely absolutely don't work

especially if your models are of

different size with judges. Um and then

I pass it to train RFDTR medium or

large. So I I chatted a bit with Rob

people and they encouraged to use that

and I'm going it it actually works. I'm

going to show you very shortly. U but

how does it work? So you take the

repository and then you just ask it okay

can you train it? Can you do the

training on this data set on hub? And

then it will start like if if the data

set has labels like you can actually

just get to training but if it doesn't

have it you can just uh start annotating

and I basically like the trick is I I

pass the overlaid bounding boxes on

images to the judge. So basically Qan

technically outputs bounding boxes as

tokens. I don't pass them. I just

overlay the bounding boxes and I pass

that image on top of some labels and

label descriptions and I say okay if

this label description has bounding box

on it then just um tell me if you

approve or not and then the judge I I

merge the judge verdicts over minimum

agreement and not consensus which I will

come to why I did that that way and

these label descriptions are also

generated by coding agents. and you just

approve it as a human being.

And the the models that I used in this

pipeline all have Apache 2.0 of licenses

except for LFM model which has type of

license where you are kind of um if you

have like certain amount of revenue

after that you pay for it but you can

comfortably use it it's large and for

the coding agents that are babysitting

this pipeline uh I initially built with

oppus 4.6 uh 8 and then ran the workflow

with the G gl GLM 5.2 2 which is doing a

good job on long horizon tasks to be

honest

and for infra I actually work at hugging

face I have a lot of compute credits and

I'm super impatient in life so like uh I

use a good amount of hardware for

experimentations but I benchmarked it

and overall it takes like three four

dollars if you want to run this entire

pipeline to train models which to me is

crazy initially for Q1 3.5 5 I used

serverless because I was like okay this

is convenient and it's super cheap so I

use deep infra which is super super

cheap if you were to use together it's

better if you do batch processing over

jobs and then for the judging I used

hugging face jobs which cost less and

then for training again I used like an

L4 but the model is super small like

RFDTR is super small and you can just

use something else you can do it locally

if you wanted to. I'm just impatient. I

want a big batch size. So yeah,

and I tested in two problems. First of

the road sign detection. Uh secondly,

document parsing. For the road sign

detection one, I already have the

labels. So like I actually compared

against the ground truth annotations if

my pipeline works or not. And for

document parsing, I actually uh couldn't

do it because basically I use like a

docqa data set and I the problem is like

I want to extract the images, tables, I

don't know signatures and stuff. So it's

like a novel task and I wanted to see if

RFDTR can actually learn it.

So first result it works. So yay. um we

have like a good uh mean average

precision over um 50 um and I compare it

against the basically I have a test set

and I take that test set and pass

through QN and then I compare against

the pseudo notations and the ground root

annotations of that test set there is a

bit of a gap but it's kind of expected

because it's learned from QN and also uh

rock oak is also like a good value to be

frank for like such use case. Um and for

the document parsing it actually

generalizes which to me is crazy.

Basically uh here the trained model

output you can see that it detected the

signature. Meanwhile the QN annotation

of that test set missed it. So I would

like to say it actually generalizes very

well as well. uh but we owe this to like

the how good RFDTR is as a backbone in a

way

and here you can also see how it uh

captures the images technically

um and it's like oneonone

and while I was building this I actually

noticed that I was clueless about

building with vision agents um so I have

bunch of findings around it um so first

up there is a huge judge imbalance so

for depending on the problem. LFM tends

to reject a lot. That's why I couldn't

take the consensus because if I were to

if I were to eliminate everything that

both LFM and Gemma agreed to remove, um

I would left with very very little

number of examples which would leave me

with very poor generalization. So what I

did was that okay, if one of them says

yes, I'm going to take that example and

nevertheless it worked well. If you have

a large data set and if you care about

the recall, I suggest that you take the

consensus or like just observe for

document parsing the gap isn't as big.

And secondly, uh the prompt generation

is a bit hard. So like this is the only

part where as a human you have to

approve uh okay the model generates the

prompts for you for the judge uh and

then you will say okay this I I approve

this because you need to to take it

still take a look at it uh take a look

at your data set a little there's no

escaping that

um and thirdly and this is super

interesting because your co your coding

agent despite no matter how uh good it

is. Like you take OPUS 4.8 which is like

a very good coding agent. Um it's

clueless as a computer vision engineer

as well as it misses common sense. For

instance like it was doing like

horizontal flip over the traffic signs

or it was doing like jitter over the

traffic lights which will definitely

corrupt your data set and break it. So I

patched this later. So you can just say

okay I don't want to augment or I can

augment. So Angular coding agent will

help you with that.

And lastly the second part of this

toolkit is my preferred models as tools.

Um so this repository covers my favorite

models from depth estimation to zero

shot segmentation.

Uh and this is partially powered by

first of hugging face benchmarks uh

which we recently rolled out maybe a few

months ago. Basically every single

basically we have like a benchmark

leaderboard and over there you have the

open models as well as their evaluation

results and you can compare different

models of different sizes. So um I keep

it up to date as well but like also it's

partially powered by me who likes to

read the um computer vision conference

papers.

Um, so I would like to give a shout out

to this model because not a lot of

people know about this. So basically Sam

cannot do open-ended reference se uh

segmentation like you can do like okay

segment this red car and it will do it.

But if you say okay the red car next to

the orange car that is next to the blue

car it will not do that. and Falcon

perception which is a model by TIA um

can actually do it and it's only like

600 million parameters with Apache 2.0

license. So this one does the zero shot

segmentation for me and this is like a

non-exhaustive list. So for posing we

have the sapiance family um for the

human centric tasks where you need to do

human koid detection human um depth

estimation and so on and for zero shot

detection uh I have moon dream 3 and mm

grounding dynino which is like a model

with apache 2.0 license is also very

good. It's very small compared to Moon

Dream. I give you the multiple models in

multiple sizes depending on your

hardware that you can pick like if you

want to go fast just pick the tiny

alternative.

Um for OCR I actually took them from the

OCR benchmark in different sizes and for

depth estimation I actually uh

discovered that the large model doesn't

have a non-commercial license and the

rest of them have it. So you can

actually use it. That one has like an

Apache 2.0 license. And it also comes

with um supervision and tracker support.

They are both uh libraries from Roboflow

that um allow you to do tracking of

instances, bounding boxes and so on

and future plans. So first off, I I

could hear you say, okay, this will

definitely not work for the industry use

cases because industry use cases have

different parts. it has like

nondescribable parts like natural

language is not a good gateway to it. Um

so I think in that sense image guided

detection could help like if you don't

know about image guided detection you

basically have like instance of an image

like a haggi here as an example and then

you ask the model okay detect this

object in this image um across all of

the images. I feel like it could

actually somewhat help in the industry

use cases where you cannot describe it

by like um natural language.

Um as well as I want to try um

intersection over union merger sort of

basically you have like labeled boxes

and then judges boxes like you ask the

judge to actually generate a box and

then you take the intersection over

union instead of asking judge to reject

or accept and I'm working currently

working on the segmentation support

and thanks for listening if you want to

actually learn more so basically I We

have a small vision repository. It has

everything about like fine-tuning

models, quantizing models, multimodel

models, everything around vision. Uh as

well as transformers task guides. We

keep them up to date has many um

tutorials. Um, we also have hugging face

skills that actually have computer

vision specific skills as well as the

infras skills that you can just do like

a one prompt training. Again, um, and

this is my Twitter profile and this repo

is actually at GitHub Marvin uh, vision

intern. I think I have qu um, time for

one question.

Thank you so much.

Yes,

>> you have plans for training.

>> Um, he's asking if I have uh plans to

train train VLMs themselves like a

self-improvement type of thing. That

would be super exciting. But first, I

want to solve this thing of like

developers actually training task

specific models and then deploying on

edge and then that could come perhaps.

Maybe one more. Yes.

>> Um, not really. I don't think so. I just

use the because I wanted because a

coding agent actually has the context. I

wanted it to generate the prompt.

Maybe one more.

Okay. Thank you so much.