[
  {
    "startMs": 13519,
    "text": "Hello and welcome to this talk on skill"
  },
  {
    "startMs": 16880,
    "text": "issue. It's actually no longer a skill"
  },
  {
    "startMs": 19439,
    "text": "issue. So uh by the end of this talk you"
  },
  {
    "startMs": 22880,
    "text": "will be able to build a lot with the"
  },
  {
    "startMs": 25039,
    "text": "vision models if you are already not"
  },
  {
    "startMs": 26960,
    "text": "doing it. So shortly about me, I'm Marv."
  },
  {
    "startMs": 30960,
    "text": "Uh I've been working on computer vision"
  },
  {
    "startMs": 33360,
    "text": "since Lava in the fix times. Uh and"
  },
  {
    "startMs": 37040,
    "text": "lately I work on more agents and"
  },
  {
    "startMs": 39120,
    "text": "ondevice stuff because I'm a bit"
  },
  {
    "startMs": 40719,
    "text": "fascinated and I love vision language"
  },
  {
    "startMs": 43440,
    "text": "models so much that I even wrote a book"
  },
  {
    "startMs": 45280,
    "text": "on it. But I don't want developers to"
  },
  {
    "startMs": 47680,
    "text": "directly use vision language models"
  },
  {
    "startMs": 49760,
    "text": "anymore. And uh I want every single"
  },
  {
    "startMs": 52960,
    "text": "developer to start uh building vision"
  },
  {
    "startMs": 56079,
    "text": "langu vision applications end to end."
  },
  {
    "startMs": 59520,
    "text": "And this talk is going to give you like"
  },
  {
    "startMs": 61760,
    "text": "a good baseline to do so."
  },
  {
    "startMs": 64960,
    "text": "And common behaviors I observe with the"
  },
  {
    "startMs": 67280,
    "text": "developers is the fact that they try to"
  },
  {
    "startMs": 69280,
    "text": "use vision language models for"
  },
  {
    "startMs": 70960,
    "text": "everything but they you will never get"
  },
  {
    "startMs": 74400,
    "text": "real time. And when I mentioned real"
  },
  {
    "startMs": 76400,
    "text": "time, it's like you have a toaster and"
  },
  {
    "startMs": 79040,
    "text": "you get like 30 40 fps uh on it like"
  },
  {
    "startMs": 82880,
    "text": "whatever whether you are building like"
  },
  {
    "startMs": 84560,
    "text": "image classification, instance"
  },
  {
    "startMs": 86080,
    "text": "segmentation or whatever."
  },
  {
    "startMs": 89040,
    "text": "And um they are not super robust in"
  },
  {
    "startMs": 92720,
    "text": "terms of like if you were to train like"
  },
  {
    "startMs": 94560,
    "text": "an RFDTR which Joseph spoke in the I"
  },
  {
    "startMs": 97759,
    "text": "spoke about in the first talk um it will"
  },
  {
    "startMs": 101200,
    "text": "always outperform your vision language"
  },
  {
    "startMs": 103040,
    "text": "model and I'm going to prove it today."
  },
  {
    "startMs": 107360,
    "text": "And uh on the right hand side you can"
  },
  {
    "startMs": 109439,
    "text": "actually see me uh doing stuff with"
  },
  {
    "startMs": 111759,
    "text": "RFDTR."
  },
  {
    "startMs": 113360,
    "text": "Um and another thing is that they don't"
  },
  {
    "startMs": 116320,
    "text": "read the licenses. Everybody's like"
  },
  {
    "startMs": 118479,
    "text": "whenever I post something about object"
  },
  {
    "startMs": 120399,
    "text": "detection, they always ask me about"
  },
  {
    "startMs": 122079,
    "text": "YOLO. Like YOLO is a good model. Uh but"
  },
  {
    "startMs": 124960,
    "text": "it's a it has AGPL 3.0 license I think."
  },
  {
    "startMs": 128399,
    "text": "And I could swear on my life that there"
  },
  {
    "startMs": 130160,
    "text": "is like some developers that actually"
  },
  {
    "startMs": 131680,
    "text": "deploy it without knowing that they have"
  },
  {
    "startMs": 133680,
    "text": "to pay for it. Um so yeah like uh I want"
  },
  {
    "startMs": 138720,
    "text": "you to migrate to Apache 2.0 models"
  },
  {
    "startMs": 142160,
    "text": "today."
  },
  {
    "startMs": 144000,
    "text": "And for this I have built something"
  },
  {
    "startMs": 146000,
    "text": "called web vision and it's kind of"
  },
  {
    "startMs": 148400,
    "text": "inspired by this post by Mazar."
  },
  {
    "startMs": 150959,
    "text": "Basically what he does is that he gives"
  },
  {
    "startMs": 153680,
    "text": "um SAM 3.1 model as a tool to Gemma 4 to"
  },
  {
    "startMs": 157599,
    "text": "call and I find it super impressive. Um"
  },
  {
    "startMs": 160959,
    "text": "and today I built like a tool kit uh"
  },
  {
    "startMs": 164080,
    "text": "where you can do this with like even"
  },
  {
    "startMs": 166560,
    "text": "more things."
  },
  {
    "startMs": 169599,
    "text": "So I'm kind of distilling myself. Um"
  },
  {
    "startMs": 172720,
    "text": "first off uh this toolkit has my"
  },
  {
    "startMs": 175360,
    "text": "favorite models as tools uh so that you"
  },
  {
    "startMs": 177840,
    "text": "can give it to your agent because your"
  },
  {
    "startMs": 179680,
    "text": "coding agent is a bit of a clueless"
  },
  {
    "startMs": 182480,
    "text": "computer vision engineer basically. Um"
  },
  {
    "startMs": 185280,
    "text": "and when I distill myself basically like"
  },
  {
    "startMs": 187280,
    "text": "whenever I pick a model I always check"
  },
  {
    "startMs": 189519,
    "text": "the following. So first up the license"
  },
  {
    "startMs": 191920,
    "text": "is the biggest priority for me. It has"
  },
  {
    "startMs": 194480,
    "text": "to be Apache 2.2 MIT or something"
  },
  {
    "startMs": 196879,
    "text": "non-commercial. Uh secondly uh the"
  },
  {
    "startMs": 200400,
    "text": "performance has to be on par uh"
  },
  {
    "startMs": 202959,
    "text": "depending on its size or architectural"
  },
  {
    "startMs": 204959,
    "text": "choices. So I check from the benchmarks"
  },
  {
    "startMs": 208080,
    "text": "uh whenever a model from a computer"
  },
  {
    "startMs": 210400,
    "text": "vision conference comes out and third"
  },
  {
    "startMs": 212959,
    "text": "vibes obviously"
  },
  {
    "startMs": 215920,
    "text": "and so this toolkit has a second part to"
  },
  {
    "startMs": 219360,
    "text": "it and it's a bit like a vibe training"
  },
  {
    "startMs": 221840,
    "text": "part which is the most exciting part. So"
  },
  {
    "startMs": 224959,
    "text": "I will go through it first. So I put"
  },
  {
    "startMs": 227440,
    "text": "myself in developers shoes to build like"
  },
  {
    "startMs": 229760,
    "text": "a vision application. Um if I have"
  },
  {
    "startMs": 232560,
    "text": "labeled images easy I can just train a"
  },
  {
    "startMs": 235360,
    "text": "model or like if I I can give my"
  },
  {
    "startMs": 237920,
    "text": "computer computer vision agent some"
  },
  {
    "startMs": 240319,
    "text": "tutorials uh to do that because it's all"
  },
  {
    "startMs": 242879,
    "text": "out there like we built transformers for"
  },
  {
    "startMs": 245439,
    "text": "it. Uh but if I have images only I have"
  },
  {
    "startMs": 248480,
    "text": "to annotate uh and then uh evaluate the"
  },
  {
    "startMs": 251519,
    "text": "annotations and then just train a model."
  },
  {
    "startMs": 254400,
    "text": "But how can you do it on scale? You can"
  },
  {
    "startMs": 256639,
    "text": "actually use a vision language model as"
  },
  {
    "startMs": 258880,
    "text": "a labeler and the vision language models"
  },
  {
    "startMs": 261120,
    "text": "as judge and then train what you want."
  },
  {
    "startMs": 264800,
    "text": "But how does this pipeline look like?"
  },
  {
    "startMs": 268160,
    "text": "So basically I built this and it has"
  },
  {
    "startMs": 272560,
    "text": "like it has VLM for labeling VLM as a"
  },
  {
    "startMs": 275440,
    "text": "judge and then training. Uh it's a bit"
  },
  {
    "startMs": 278240,
    "text": "of a long horizon tasks for coding"
  },
  {
    "startMs": 280160,
    "text": "agents and it has infra many"
  },
  {
    "startMs": 282720,
    "text": "infrastructure support. You can do this"
  },
  {
    "startMs": 284479,
    "text": "locally. You can do this uh remotely. So"
  },
  {
    "startMs": 287440,
    "text": "basically it runs on hugging face"
  },
  {
    "startMs": 288960,
    "text": "infrastructure. We have jobs that allow"
  },
  {
    "startMs": 291600,
    "text": "you to do oneoff batch processing or"
  },
  {
    "startMs": 294240,
    "text": "training. We also have like a serverless"
  },
  {
    "startMs": 297199,
    "text": "routing system called inference"
  },
  {
    "startMs": 299199,
    "text": "providers where you can use multiple"
  },
  {
    "startMs": 301120,
    "text": "providers and you we also have like"
  },
  {
    "startMs": 304000,
    "text": "buckets to dump intermediate data on top"
  },
  {
    "startMs": 306800,
    "text": "of the data set repositories, model"
  },
  {
    "startMs": 308720,
    "text": "repositories and so on."
  },
  {
    "startMs": 311840,
    "text": "Um but what enables this work? First"
  },
  {
    "startMs": 314000,
    "text": "off, my favorite model, RFDTR, RFDTR"
  },
  {
    "startMs": 317520,
    "text": "segmentation. I'm currently working on"
  },
  {
    "startMs": 319120,
    "text": "segmentation at the moment. Um, we have"
  },
  {
    "startMs": 322240,
    "text": "better agents for long horizon tasks."
  },
  {
    "startMs": 325440,
    "text": "Uh, that, you know, you have to babysit"
  },
  {
    "startMs": 327600,
    "text": "the labeling process, the training"
  },
  {
    "startMs": 330080,
    "text": "process, etc. Uh, and smaller yet more"
  },
  {
    "startMs": 334000,
    "text": "capable vision models enable you to"
  },
  {
    "startMs": 336160,
    "text": "label stuff very cheaply. And also with"
  },
  {
    "startMs": 340160,
    "text": "transformers we went for V5 refactoring"
  },
  {
    "startMs": 343280,
    "text": "etc. So like it performs better for"
  },
  {
    "startMs": 345199,
    "text": "vision models at the moment."
  },
  {
    "startMs": 347919,
    "text": "And this is how the pipeline actually"
  },
  {
    "startMs": 349759,
    "text": "looks like. So first up I label the data"
  },
  {
    "startMs": 353680,
    "text": "set like I take an image data set like"
  },
  {
    "startMs": 355440,
    "text": "any image data set. I label that image"
  },
  {
    "startMs": 357759,
    "text": "data set with Q1 3.59B"
  },
  {
    "startMs": 360960,
    "text": "and then I pass the labeled data set to"
  },
  {
    "startMs": 363759,
    "text": "two judges. First one is Gemma 4 E4B"
  },
  {
    "startMs": 367199,
    "text": "which is like an 8B judge and then the"
  },
  {
    "startMs": 370080,
    "text": "second one is LFM 2.5VL which is nearly"
  },
  {
    "startMs": 373360,
    "text": "2B uh it's relatively smaller basically"
  },
  {
    "startMs": 376319,
    "text": "I checked the research and it's better"
  },
  {
    "startMs": 378639,
    "text": "to have like an ensemble smaller judges"
  },
  {
    "startMs": 381919,
    "text": "and on top of it I merged the judgments"
  },
  {
    "startMs": 384960,
    "text": "so I also checked the research around it"
  },
  {
    "startMs": 387919,
    "text": "and most of the people ask the VLM or"
  },
  {
    "startMs": 390560,
    "text": "LLM to assign some score to it but those"
  },
  {
    "startMs": 393520,
    "text": "scores absolutely absolutely don't work"
  },
  {
    "startMs": 395360,
    "text": "especially if your models are of"
  },
  {
    "startMs": 396960,
    "text": "different size with judges. Um and then"
  },
  {
    "startMs": 400000,
    "text": "I pass it to train RFDTR medium or"
  },
  {
    "startMs": 403199,
    "text": "large. So I I chatted a bit with Rob"
  },
  {
    "startMs": 406080,
    "text": "people and they encouraged to use that"
  },
  {
    "startMs": 409039,
    "text": "and I'm going it it actually works. I'm"
  },
  {
    "startMs": 411600,
    "text": "going to show you very shortly. U but"
  },
  {
    "startMs": 414800,
    "text": "how does it work? So you take the"
  },
  {
    "startMs": 416479,
    "text": "repository and then you just ask it okay"
  },
  {
    "startMs": 419440,
    "text": "can you train it? Can you do the"
  },
  {
    "startMs": 421919,
    "text": "training on this data set on hub? And"
  },
  {
    "startMs": 425840,
    "text": "then it will start like if if the data"
  },
  {
    "startMs": 428000,
    "text": "set has labels like you can actually"
  },
  {
    "startMs": 430800,
    "text": "just get to training but if it doesn't"
  },
  {
    "startMs": 433039,
    "text": "have it you can just uh start annotating"
  },
  {
    "startMs": 437919,
    "text": "and I basically like the trick is I I"
  },
  {
    "startMs": 441360,
    "text": "pass the overlaid bounding boxes on"
  },
  {
    "startMs": 443599,
    "text": "images to the judge. So basically Qan"
  },
  {
    "startMs": 446319,
    "text": "technically outputs bounding boxes as"
  },
  {
    "startMs": 448720,
    "text": "tokens. I don't pass them. I just"
  },
  {
    "startMs": 450720,
    "text": "overlay the bounding boxes and I pass"
  },
  {
    "startMs": 452560,
    "text": "that image on top of some labels and"
  },
  {
    "startMs": 455759,
    "text": "label descriptions and I say okay if"
  },
  {
    "startMs": 458240,
    "text": "this label description has bounding box"
  },
  {
    "startMs": 460639,
    "text": "on it then just um tell me if you"
  },
  {
    "startMs": 464160,
    "text": "approve or not and then the judge I I"
  },
  {
    "startMs": 467840,
    "text": "merge the judge verdicts over minimum"
  },
  {
    "startMs": 470639,
    "text": "agreement and not consensus which I will"
  },
  {
    "startMs": 473360,
    "text": "come to why I did that that way and"
  },
  {
    "startMs": 476639,
    "text": "these label descriptions are also"
  },
  {
    "startMs": 478560,
    "text": "generated by coding agents. and you just"
  },
  {
    "startMs": 480720,
    "text": "approve it as a human being."
  },
  {
    "startMs": 483919,
    "text": "And the the models that I used in this"
  },
  {
    "startMs": 486479,
    "text": "pipeline all have Apache 2.0 of licenses"
  },
  {
    "startMs": 489520,
    "text": "except for LFM model which has type of"
  },
  {
    "startMs": 493120,
    "text": "license where you are kind of um if you"
  },
  {
    "startMs": 496560,
    "text": "have like certain amount of revenue"
  },
  {
    "startMs": 499199,
    "text": "after that you pay for it but you can"
  },
  {
    "startMs": 501680,
    "text": "comfortably use it it's large and for"
  },
  {
    "startMs": 505039,
    "text": "the coding agents that are babysitting"
  },
  {
    "startMs": 506879,
    "text": "this pipeline uh I initially built with"
  },
  {
    "startMs": 509280,
    "text": "oppus 4.6 uh 8 and then ran the workflow"
  },
  {
    "startMs": 513360,
    "text": "with the G gl GLM 5.2 2 which is doing a"
  },
  {
    "startMs": 516240,
    "text": "good job on long horizon tasks to be"
  },
  {
    "startMs": 518399,
    "text": "honest"
  },
  {
    "startMs": 520399,
    "text": "and for infra I actually work at hugging"
  },
  {
    "startMs": 522800,
    "text": "face I have a lot of compute credits and"
  },
  {
    "startMs": 525360,
    "text": "I'm super impatient in life so like uh I"
  },
  {
    "startMs": 529120,
    "text": "use a good amount of hardware for"
  },
  {
    "startMs": 531680,
    "text": "experimentations but I benchmarked it"
  },
  {
    "startMs": 534240,
    "text": "and overall it takes like three four"
  },
  {
    "startMs": 537440,
    "text": "dollars if you want to run this entire"
  },
  {
    "startMs": 539839,
    "text": "pipeline to train models which to me is"
  },
  {
    "startMs": 542240,
    "text": "crazy initially for Q1 3.5 5 I used"
  },
  {
    "startMs": 545760,
    "text": "serverless because I was like okay this"
  },
  {
    "startMs": 548480,
    "text": "is convenient and it's super cheap so I"
  },
  {
    "startMs": 550640,
    "text": "use deep infra which is super super"
  },
  {
    "startMs": 553040,
    "text": "cheap if you were to use together it's"
  },
  {
    "startMs": 555600,
    "text": "better if you do batch processing over"
  },
  {
    "startMs": 558399,
    "text": "jobs and then for the judging I used"
  },
  {
    "startMs": 561279,
    "text": "hugging face jobs which cost less and"
  },
  {
    "startMs": 564959,
    "text": "then for training again I used like an"
  },
  {
    "startMs": 567040,
    "text": "L4 but the model is super small like"
  },
  {
    "startMs": 569040,
    "text": "RFDTR is super small and you can just"
  },
  {
    "startMs": 572800,
    "text": "use something else you can do it locally"
  },
  {
    "startMs": 574480,
    "text": "if you wanted to. I'm just impatient. I"
  },
  {
    "startMs": 576320,
    "text": "want a big batch size. So yeah,"
  },
  {
    "startMs": 580000,
    "text": "and I tested in two problems. First of"
  },
  {
    "startMs": 582880,
    "text": "the road sign detection. Uh secondly,"
  },
  {
    "startMs": 586160,
    "text": "document parsing. For the road sign"
  },
  {
    "startMs": 588320,
    "text": "detection one, I already have the"
  },
  {
    "startMs": 589839,
    "text": "labels. So like I actually compared"
  },
  {
    "startMs": 592399,
    "text": "against the ground truth annotations if"
  },
  {
    "startMs": 594160,
    "text": "my pipeline works or not. And for"
  },
  {
    "startMs": 596640,
    "text": "document parsing, I actually uh couldn't"
  },
  {
    "startMs": 599040,
    "text": "do it because basically I use like a"
  },
  {
    "startMs": 601600,
    "text": "docqa data set and I the problem is like"
  },
  {
    "startMs": 604160,
    "text": "I want to extract the images, tables, I"
  },
  {
    "startMs": 607279,
    "text": "don't know signatures and stuff. So it's"
  },
  {
    "startMs": 609279,
    "text": "like a novel task and I wanted to see if"
  },
  {
    "startMs": 611920,
    "text": "RFDTR can actually learn it."
  },
  {
    "startMs": 615200,
    "text": "So first result it works. So yay. um we"
  },
  {
    "startMs": 619519,
    "text": "have like a good uh mean average"
  },
  {
    "startMs": 621440,
    "text": "precision over um 50 um and I compare it"
  },
  {
    "startMs": 626800,
    "text": "against the basically I have a test set"
  },
  {
    "startMs": 629440,
    "text": "and I take that test set and pass"
  },
  {
    "startMs": 631360,
    "text": "through QN and then I compare against"
  },
  {
    "startMs": 634000,
    "text": "the pseudo notations and the ground root"
  },
  {
    "startMs": 637680,
    "text": "annotations of that test set there is a"
  },
  {
    "startMs": 640240,
    "text": "bit of a gap but it's kind of expected"
  },
  {
    "startMs": 642320,
    "text": "because it's learned from QN and also uh"
  },
  {
    "startMs": 645839,
    "text": "rock oak is also like a good value to be"
  },
  {
    "startMs": 648720,
    "text": "frank for like such use case. Um and for"
  },
  {
    "startMs": 653279,
    "text": "the document parsing it actually"
  },
  {
    "startMs": 655360,
    "text": "generalizes which to me is crazy."
  },
  {
    "startMs": 657519,
    "text": "Basically uh here the trained model"
  },
  {
    "startMs": 660720,
    "text": "output you can see that it detected the"
  },
  {
    "startMs": 663200,
    "text": "signature. Meanwhile the QN annotation"
  },
  {
    "startMs": 666640,
    "text": "of that test set missed it. So I would"
  },
  {
    "startMs": 669600,
    "text": "like to say it actually generalizes very"
  },
  {
    "startMs": 671600,
    "text": "well as well. uh but we owe this to like"
  },
  {
    "startMs": 674800,
    "text": "the how good RFDTR is as a backbone in a"
  },
  {
    "startMs": 678320,
    "text": "way"
  },
  {
    "startMs": 680160,
    "text": "and here you can also see how it uh"
  },
  {
    "startMs": 682399,
    "text": "captures the images technically"
  },
  {
    "startMs": 685680,
    "text": "um and it's like oneonone"
  },
  {
    "startMs": 688320,
    "text": "and while I was building this I actually"
  },
  {
    "startMs": 690240,
    "text": "noticed that I was clueless about"
  },
  {
    "startMs": 692160,
    "text": "building with vision agents um so I have"
  },
  {
    "startMs": 695040,
    "text": "bunch of findings around it um so first"
  },
  {
    "startMs": 698160,
    "text": "up there is a huge judge imbalance so"
  },
  {
    "startMs": 700640,
    "text": "for depending on the problem. LFM tends"
  },
  {
    "startMs": 704000,
    "text": "to reject a lot. That's why I couldn't"
  },
  {
    "startMs": 706320,
    "text": "take the consensus because if I were to"
  },
  {
    "startMs": 708480,
    "text": "if I were to eliminate everything that"
  },
  {
    "startMs": 710880,
    "text": "both LFM and Gemma agreed to remove, um"
  },
  {
    "startMs": 715120,
    "text": "I would left with very very little"
  },
  {
    "startMs": 717120,
    "text": "number of examples which would leave me"
  },
  {
    "startMs": 719360,
    "text": "with very poor generalization. So what I"
  },
  {
    "startMs": 721920,
    "text": "did was that okay, if one of them says"
  },
  {
    "startMs": 723680,
    "text": "yes, I'm going to take that example and"
  },
  {
    "startMs": 725839,
    "text": "nevertheless it worked well. If you have"
  },
  {
    "startMs": 727680,
    "text": "a large data set and if you care about"
  },
  {
    "startMs": 729839,
    "text": "the recall, I suggest that you take the"
  },
  {
    "startMs": 733440,
    "text": "consensus or like just observe for"
  },
  {
    "startMs": 736240,
    "text": "document parsing the gap isn't as big."
  },
  {
    "startMs": 741440,
    "text": "And secondly, uh the prompt generation"
  },
  {
    "startMs": 744399,
    "text": "is a bit hard. So like this is the only"
  },
  {
    "startMs": 747360,
    "text": "part where as a human you have to"
  },
  {
    "startMs": 749200,
    "text": "approve uh okay the model generates the"
  },
  {
    "startMs": 752639,
    "text": "prompts for you for the judge uh and"
  },
  {
    "startMs": 755519,
    "text": "then you will say okay this I I approve"
  },
  {
    "startMs": 757920,
    "text": "this because you need to to take it"
  },
  {
    "startMs": 760240,
    "text": "still take a look at it uh take a look"
  },
  {
    "startMs": 762639,
    "text": "at your data set a little there's no"
  },
  {
    "startMs": 764720,
    "text": "escaping that"
  },
  {
    "startMs": 766800,
    "text": "um and thirdly and this is super"
  },
  {
    "startMs": 770560,
    "text": "interesting because your co your coding"
  },
  {
    "startMs": 773440,
    "text": "agent despite no matter how uh good it"
  },
  {
    "startMs": 776639,
    "text": "is. Like you take OPUS 4.8 which is like"
  },
  {
    "startMs": 779440,
    "text": "a very good coding agent. Um it's"
  },
  {
    "startMs": 782880,
    "text": "clueless as a computer vision engineer"
  },
  {
    "startMs": 785040,
    "text": "as well as it misses common sense. For"
  },
  {
    "startMs": 787040,
    "text": "instance like it was doing like"
  },
  {
    "startMs": 788720,
    "text": "horizontal flip over the traffic signs"
  },
  {
    "startMs": 791519,
    "text": "or it was doing like jitter over the"
  },
  {
    "startMs": 794880,
    "text": "traffic lights which will definitely"
  },
  {
    "startMs": 797040,
    "text": "corrupt your data set and break it. So I"
  },
  {
    "startMs": 800560,
    "text": "patched this later. So you can just say"
  },
  {
    "startMs": 802639,
    "text": "okay I don't want to augment or I can"
  },
  {
    "startMs": 805279,
    "text": "augment. So Angular coding agent will"
  },
  {
    "startMs": 807680,
    "text": "help you with that."
  },
  {
    "startMs": 810320,
    "text": "And lastly the second part of this"
  },
  {
    "startMs": 812160,
    "text": "toolkit is my preferred models as tools."
  },
  {
    "startMs": 815519,
    "text": "Um so this repository covers my favorite"
  },
  {
    "startMs": 818320,
    "text": "models from depth estimation to zero"
  },
  {
    "startMs": 820800,
    "text": "shot segmentation."
  },
  {
    "startMs": 822639,
    "text": "Uh and this is partially powered by"
  },
  {
    "startMs": 824560,
    "text": "first of hugging face benchmarks uh"
  },
  {
    "startMs": 826720,
    "text": "which we recently rolled out maybe a few"
  },
  {
    "startMs": 828880,
    "text": "months ago. Basically every single"
  },
  {
    "startMs": 831839,
    "text": "basically we have like a benchmark"
  },
  {
    "startMs": 834399,
    "text": "leaderboard and over there you have the"
  },
  {
    "startMs": 836480,
    "text": "open models as well as their evaluation"
  },
  {
    "startMs": 839360,
    "text": "results and you can compare different"
  },
  {
    "startMs": 842000,
    "text": "models of different sizes. So um I keep"
  },
  {
    "startMs": 846160,
    "text": "it up to date as well but like also it's"
  },
  {
    "startMs": 849040,
    "text": "partially powered by me who likes to"
  },
  {
    "startMs": 851199,
    "text": "read the um computer vision conference"
  },
  {
    "startMs": 854639,
    "text": "papers."
  },
  {
    "startMs": 856560,
    "text": "Um, so I would like to give a shout out"
  },
  {
    "startMs": 859199,
    "text": "to this model because not a lot of"
  },
  {
    "startMs": 860800,
    "text": "people know about this. So basically Sam"
  },
  {
    "startMs": 863120,
    "text": "cannot do open-ended reference se uh"
  },
  {
    "startMs": 865519,
    "text": "segmentation like you can do like okay"
  },
  {
    "startMs": 869360,
    "text": "segment this red car and it will do it."
  },
  {
    "startMs": 872000,
    "text": "But if you say okay the red car next to"
  },
  {
    "startMs": 874079,
    "text": "the orange car that is next to the blue"
  },
  {
    "startMs": 876480,
    "text": "car it will not do that. and Falcon"
  },
  {
    "startMs": 878800,
    "text": "perception which is a model by TIA um"
  },
  {
    "startMs": 882000,
    "text": "can actually do it and it's only like"
  },
  {
    "startMs": 884000,
    "text": "600 million parameters with Apache 2.0"
  },
  {
    "startMs": 886959,
    "text": "license. So this one does the zero shot"
  },
  {
    "startMs": 889839,
    "text": "segmentation for me and this is like a"
  },
  {
    "startMs": 893680,
    "text": "non-exhaustive list. So for posing we"
  },
  {
    "startMs": 896399,
    "text": "have the sapiance family um for the"
  },
  {
    "startMs": 899519,
    "text": "human centric tasks where you need to do"
  },
  {
    "startMs": 902000,
    "text": "human koid detection human um depth"
  },
  {
    "startMs": 904720,
    "text": "estimation and so on and for zero shot"
  },
  {
    "startMs": 908720,
    "text": "detection uh I have moon dream 3 and mm"
  },
  {
    "startMs": 912079,
    "text": "grounding dynino which is like a model"
  },
  {
    "startMs": 914399,
    "text": "with apache 2.0 license is also very"
  },
  {
    "startMs": 916639,
    "text": "good. It's very small compared to Moon"
  },
  {
    "startMs": 918720,
    "text": "Dream. I give you the multiple models in"
  },
  {
    "startMs": 922000,
    "text": "multiple sizes depending on your"
  },
  {
    "startMs": 923600,
    "text": "hardware that you can pick like if you"
  },
  {
    "startMs": 925760,
    "text": "want to go fast just pick the tiny"
  },
  {
    "startMs": 928079,
    "text": "alternative."
  },
  {
    "startMs": 929760,
    "text": "Um for OCR I actually took them from the"
  },
  {
    "startMs": 932880,
    "text": "OCR benchmark in different sizes and for"
  },
  {
    "startMs": 936480,
    "text": "depth estimation I actually uh"
  },
  {
    "startMs": 938560,
    "text": "discovered that the large model doesn't"
  },
  {
    "startMs": 940560,
    "text": "have a non-commercial license and the"
  },
  {
    "startMs": 942320,
    "text": "rest of them have it. So you can"
  },
  {
    "startMs": 943760,
    "text": "actually use it. That one has like an"
  },
  {
    "startMs": 945519,
    "text": "Apache 2.0 license. And it also comes"
  },
  {
    "startMs": 948240,
    "text": "with um supervision and tracker support."
  },
  {
    "startMs": 950480,
    "text": "They are both uh libraries from Roboflow"
  },
  {
    "startMs": 952880,
    "text": "that um allow you to do tracking of"
  },
  {
    "startMs": 956560,
    "text": "instances, bounding boxes and so on"
  },
  {
    "startMs": 960720,
    "text": "and future plans. So first off, I I"
  },
  {
    "startMs": 963759,
    "text": "could hear you say, okay, this will"
  },
  {
    "startMs": 965440,
    "text": "definitely not work for the industry use"
  },
  {
    "startMs": 967759,
    "text": "cases because industry use cases have"
  },
  {
    "startMs": 970399,
    "text": "different parts. it has like"
  },
  {
    "startMs": 972399,
    "text": "nondescribable parts like natural"
  },
  {
    "startMs": 974880,
    "text": "language is not a good gateway to it. Um"
  },
  {
    "startMs": 978560,
    "text": "so I think in that sense image guided"
  },
  {
    "startMs": 981360,
    "text": "detection could help like if you don't"
  },
  {
    "startMs": 983040,
    "text": "know about image guided detection you"
  },
  {
    "startMs": 984800,
    "text": "basically have like instance of an image"
  },
  {
    "startMs": 987440,
    "text": "like a haggi here as an example and then"
  },
  {
    "startMs": 990480,
    "text": "you ask the model okay detect this"
  },
  {
    "startMs": 993600,
    "text": "object in this image um across all of"
  },
  {
    "startMs": 997199,
    "text": "the images. I feel like it could"
  },
  {
    "startMs": 999040,
    "text": "actually somewhat help in the industry"
  },
  {
    "startMs": 1000880,
    "text": "use cases where you cannot describe it"
  },
  {
    "startMs": 1003040,
    "text": "by like um natural language."
  },
  {
    "startMs": 1006800,
    "text": "Um as well as I want to try um"
  },
  {
    "startMs": 1011360,
    "text": "intersection over union merger sort of"
  },
  {
    "startMs": 1014160,
    "text": "basically you have like labeled boxes"
  },
  {
    "startMs": 1017680,
    "text": "and then judges boxes like you ask the"
  },
  {
    "startMs": 1020240,
    "text": "judge to actually generate a box and"
  },
  {
    "startMs": 1022000,
    "text": "then you take the intersection over"
  },
  {
    "startMs": 1023440,
    "text": "union instead of asking judge to reject"
  },
  {
    "startMs": 1026319,
    "text": "or accept and I'm working currently"
  },
  {
    "startMs": 1029199,
    "text": "working on the segmentation support"
  },
  {
    "startMs": 1032400,
    "text": "and thanks for listening if you want to"
  },
  {
    "startMs": 1034640,
    "text": "actually learn more so basically I We"
  },
  {
    "startMs": 1036480,
    "text": "have a small vision repository. It has"
  },
  {
    "startMs": 1039360,
    "text": "everything about like fine-tuning"
  },
  {
    "startMs": 1041839,
    "text": "models, quantizing models, multimodel"
  },
  {
    "startMs": 1044480,
    "text": "models, everything around vision. Uh as"
  },
  {
    "startMs": 1047600,
    "text": "well as transformers task guides. We"
  },
  {
    "startMs": 1049520,
    "text": "keep them up to date has many um"
  },
  {
    "startMs": 1052320,
    "text": "tutorials. Um, we also have hugging face"
  },
  {
    "startMs": 1055919,
    "text": "skills that actually have computer"
  },
  {
    "startMs": 1058400,
    "text": "vision specific skills as well as the"
  },
  {
    "startMs": 1061280,
    "text": "infras skills that you can just do like"
  },
  {
    "startMs": 1063760,
    "text": "a one prompt training. Again, um, and"
  },
  {
    "startMs": 1067440,
    "text": "this is my Twitter profile and this repo"
  },
  {
    "startMs": 1070080,
    "text": "is actually at GitHub Marvin uh, vision"
  },
  {
    "startMs": 1073679,
    "text": "intern. I think I have qu um, time for"
  },
  {
    "startMs": 1077440,
    "text": "one question."
  },
  {
    "startMs": 1079440,
    "text": "Thank you so much."
  },
  {
    "startMs": 1085120,
    "text": "Yes,"
  },
  {
    "startMs": 1085840,
    "text": ">> you have plans for training."
  },
  {
    "startMs": 1088720,
    "text": ">> Um, he's asking if I have uh plans to"
  },
  {
    "startMs": 1092000,
    "text": "train train VLMs themselves like a"
  },
  {
    "startMs": 1095120,
    "text": "self-improvement type of thing. That"
  },
  {
    "startMs": 1097200,
    "text": "would be super exciting. But first, I"
  },
  {
    "startMs": 1099039,
    "text": "want to solve this thing of like"
  },
  {
    "startMs": 1100960,
    "text": "developers actually training task"
  },
  {
    "startMs": 1103440,
    "text": "specific models and then deploying on"
  },
  {
    "startMs": 1105679,
    "text": "edge and then that could come perhaps."
  },
  {
    "startMs": 1108960,
    "text": "Maybe one more. Yes."
  },
  {
    "startMs": 1118320,
    "text": ">> Um, not really. I don't think so. I just"
  },
  {
    "startMs": 1121039,
    "text": "use the because I wanted because a"
  },
  {
    "startMs": 1123440,
    "text": "coding agent actually has the context. I"
  },
  {
    "startMs": 1125760,
    "text": "wanted it to generate the prompt."
  },
  {
    "startMs": 1129120,
    "text": "Maybe one more."
  },
  {
    "startMs": 1132640,
    "text": "Okay. Thank you so much."
  }
]