Moondream: how does a tiny vision model slap so hard?

Talk transcript

1. 0:00

   [upbeat music] Hi, my name is Vik. I work on a model, an open source vision model called Moondream.

2. 0:17

   Um, a little bit about myself before I dive into Moondream. Uh, I was at A- AWS for about nine years, um, before I started working on this model. Um, looking at where the stock price is going, I'm not sure if that was the right financial decision, but I'm very happy with the work I'm doing.

3. 0:33

   Uh, so let's dive into it. We'll talk about Moondream a little bit. Um, it is a tiny vision language model. It's less than two billion parameters, so it can run anywhere, and it's open source, Apache 2.0, so you can use it to do anything.

4. 0:49

   Um, here's some examples of things you can do with Moondream. You can ask it questions about images, um, you can

5. 0:57

   caption images, uh, it can detect spe-specific objects inside of images. So here I asked it to tell me where the peak is, and it gives me coordinates. Um, it can count stuff.

6. 1:08

   It can do all sorts of things. Um, I had the audacity to title my talk, "How can a tiny vision model slap so hard?" So I have to back things up a little bit.

7. 1:17

   Um, and so here's me doing that. Um, these are two vision, uh, benchmarks, vi-vision qu-question answering benchmarks. One's called VQA v2, the other is called GQA. Um, as you can see, Moondream has been steadily improving over the releases I've made over the last three months.

8. 1:33

   Uh, I've included a reference line over there for LLaVA 1.5, which is a popular seven billion parameter, uh, vision model. So this shows you that, um, Moondream gives you compare-- performance that's comparable to, uh, models that are about four times bigger than it.

9. 1:50

   Um, I didn't really set out to build a vision model, um, so I kinda got roped into it. I was originally trying to build a-an application that required an AI agent, so I needed to be able to see what was going on on the user's screen and, um, have it describe what's on the browser page for QA

10. 2:06

    testing automation. Tried to do this at first with, uh, GPT-4V, but, um, there were too many safety refusals back then. Like, if there was any human being present in the image, it would just refuse to process it.

11. 2:19

    Um, it was also gonna be really slow and expensive, and so I realized if this is a product I'm trying to build, I really need to have control over the model itself.

12. 2:26

    So I figured, you know what? How hard can it be? Let me just go try and, um, build this model myself. Now, the task I was trying to perform here was fairly constrained.

13. 2:34

    Um, I, I just needed to describe s-screens and answer questions about screens, so, um, it, it doesn't need to be generally intelligent. Um, I had a couple of 3090s at home, so I figured I'd train a small version of the model at home and then rent some beefier machines in the cloud to go train a bigger version.

14. 2:52

    Um, and, uh, once I got done training the small version, I was like, "Hey, this actually works pretty well," so I posted it on Twitter. I thought, "You know what?

15. 2:59

    I might get 20 likes off of this, and then I'll move on with my side project at the time." Um, it blew up far beyond expectations. I was a little surprised, pleasantly surprised, but surprised nonetheless.

16. 3:10

    And I immediately started seeing, um, other automated testing companies reach out and be like, "Hey, can I use this to describe browser screens? Um, 'cause this would work really well for us."

17. 3:20

    Um, as well as other companies. Shout out to our friends at Open Interpreter from Seattle, um,

18. 3:26

    that basically told us that they were... I figured, you know what? Like, this is getting a lot of traction. Let me pause on the whole automated testing app for a couple of weeks and focus on Moondream and see where it goes.

19. 3:37

    Um, yeah, so let me dive into a couple of the technical details, um, around what makes the model succeed despite being small. The first thing we did, um, that I think really helped was, um, deciding what problems the model should solve and what it should not solve.

20. 3:55

    So Moondream wants to be a developer tool. We focus on being really accurate and not hallucinate. Um, it doesn't really have a know-- a lot of knowledge about the world.

21. 4:04

    So, um, if you ask it to write a poem, it's probably not gonna help you. It's really focused on answering questions, like giving you... helping you understand images. Um, this is really important because it affects the type of data that you use and the sort of benchmarks that you wanna focus on.

22. 4:20

    Uh, there's a popular vision language model benchmark called MathVista, which measures how good models are at solving math problems. You take a picture of a differential equation, and you see whether the model can solve it.

23. 4:30

    That was, uh, an example of a non-goal for us because we just want the model to be good at looking at images. The, the most we do is probably generate a LaTeX representation of the problem.

24. 4:40

    We don't really wanna even attempt to [chuckles] try and solve calculus. Um, it was not pre-trained from scratch. We use, um, um, we, we fuse a vision encoder called SigLIP from Google, uh, with a pre-trained text model called Phi, uh, 1.5 from Microsoft.

25. 4:58

    Uh, the notable thing over here is Phi 1.5 was also trained on mostly synthetic data, which is very similar to our pipeline, so it works very well. Um,

26. 5:07

    for this sort of task, pre-training from scratch doesn't really make a difference, uh, as opposed to using pre-trained models, and it is cost prohibitive. So unless you wanna get those brownie points for saying you trained it from scratch, uh, it's probably not worth doing.

27. 5:20

    We experimented with a bunch of different other models, uh, as they were released, and nothing really made too much of a difference. What does make a difference, though, is, uh, training data.

28. 5:29

    Um, the latest release of Moondream is trained on around 35 million images, and, uh, the problem is, especially when you're on a budget, like high-quality, multi-modal training data is really hard to come by.

29. 5:43

    Um, there's companies that, um, there's a lot of companies out there that'll annotate data with humans, um, but it, it's really expensive, and I've heard a rumor recently that they won't even talk to you anymore unless you're willing to sign an upfront seven-figure commitment.

30. 5:58

    There's a lot of data on the internet, images, alt text pairs. Um, the problem with this is it's often not in the format you want it to be, and it's really noisy, and the noise is really problematic when you're, uh, when, when you're training small models.

31. 6:13

    Um, and so synthetic data is a way to solve this, uh, where you use that alt text information, process it. It's a bit of an open secret that a lot of people are training on outputs from GPT-4.

32. 6:24

    Um, you probably don't want to do that. Um, besides being questionable in terms of, terms of use, uh, it's often not helpful. GPT-4 is a very powerful model. It has reasoning capabilities and knowledge that your small model is never going to be able to get.

33. 6:39

    And so when you train it on GPT-4 outputs, what it learns instead is to hallucinate. It's gonna generate plausible-sounding outputs that include details that it cannot possibly memorize, uh, and so you end up in trouble.

34. 6:51

    So, um, this is a little important. I'm going to go a little more technically detailed for a couple of minutes to dive into how to do synthetic data, so bear with me for a sec.

35. 6:59

    We'll pop back up. Um, here's an example of how not to, not to do it. Uh, COCO is a dataset. It has around two hundred K images. Each image has five short descriptions and a bunch of, uh, object annotations, uh, with like, "Hey, there's a bicycle at these coordinates," and whatnot.

36. 7:14

    Um, and let's say you wanna take those short descriptions and these an-- these object annotations and generate more detailed captions that include the union of all the information present over here.

37. 7:22

    If you just naively call GPT-4, uh, with this information, uh, it generates this. Not important to read all of it, uh, but there's, uh, two important things to note.

38. 7:31

    The first is that, um, it hallucinates. It says, uh, in the second paragraph, there's a person near the right side of the harbor. I, I think there's, like, a person way back the...

39. 7:43

    There's, like, five pixels there that may be a post. It may be a person. We don't really know. That's because the object annotations were bad, but besides that, like, um, the model is also taking a lot of creative liberties over here, like saying there's five yachts standing out from the rest and whatnot.

40. 7:57

    Um, and so this is, uh... You need to do a little more pre-processing of your data before you feed it to the model. Um, here's another example. There's a dataset from Google called Localized Narratives.

41. 8:09

    Um, the task annotators here, uh, annotators are given here is, um, verbally describe this image, and as you're describing the image, uh, hover your mouse over the part of the image that you're describing.

42. 8:20

    So it's nice in that it encourages people to create really detailed descriptions that capture spatial positioning, um,

43. 8:26

    in the image. So for example, here it says the girl in the front is playing the guitar and whatnot, and spatial reasoning is something that vision language models typically tend to struggle with.

44. 8:35

    Um, I ended up having to build a fairly sophisticated data processing pipeline to get really good results with this. Um, not really important to dive into the details over here, but the important thing to note is, A, it gets really expensive.

45. 8:47

    Um, each image ends up being twenty LLM calls, and the LLM here is Mixtral 8x7B, so it gets pretty expensive. Um,

46. 8:57

    b-but it was necessary. Uh, the training data is the biggest needle mover in terms of model performance, and because of this, uh, I'd say we spent, like, maybe one or two orders of magnitude more compute on generating training data than actually training the model itself.

47. 9:12

    Um, a couple-- So yeah, this particular dataset we've open-sourced. It's, uh, available on Hugging Face. Here's an example of the type of questions it generates for this image. Um,

48. 9:24

    the-- there's an interesting question towards the end: What theory does the kid have about the existence of pleasure in the image? I'll talk about that in a sec. But basically you wanna generate a few distractor questions so the model knows to not always agree with the question that the user is ask-- is asking.

49. 9:40

    Um, so yeah, a couple of the challenges involved in working with synthetic data. Um,

50. 9:45

    there was an interesting incident I had, uh, early on where a user was like, "Hey, I asked a relatively simple question. Why can't the model answer this?" Um, and when I looked at it, it turned out that they didn't capitalize the first letter in their question, and the model had never seen anything like that during training.

51. 9:59

    So I was like, "What do I do over here?" Um, and so it's really important for you to make sure that your training data has the, the same rough distribution as your real-world query.

52. 10:09

    So ended up adding, like, an extra step where we artificially inject, like, capitalization issues and typos and whatnot into the model before training it. Um, there's also this risk of what we call model collapse, where your model has biases inherent to it.

53. 10:23

    So for example, if you try to ask Mixtral to generate distractor questions, "Hey, just, uh, generate a question that's completely irrelevant to the image," it'll always generate something about dinosaurs and aliens.

54. 10:32

    And so if you train your model on that, it'll instead learn, uh, to say, uh, "Hey, if the question is about dinosaurs and aliens, always say no," which doesn't really help.

55. 10:41

    Um, and so there you need to in-inject, like, some entropy into the process of generating synthetic data to avoid this. Uh, in the case of synthetic captioning, you can do something like, "Hey, describe this image," but also consider the alt text on the image, which may be noisy, may be irrelevant, but if it is relevant, use relevant

56. 10:55

    facts from that. Uh, and that tends to help a lot.

57. 11:00

    All right, so popping back up, um, there were a couple of important learnings I had over the last three months, uh, that I would like to share with all of you.

58. 11:12

    Uh, the first was, um, the community was really critical in this whole journey. Seeing that original engagement that we got, uh, from the Moondream release helped me realize that, hey, maybe this is more valuable than that QA testing application that I was working on, 'cause a lot of people have a need for this to build applications like

59. 11:29

    that. Uh, coming from an enterprise-ish company, uh, it's been really valuable. It's been refreshing to be able to just talk to customers directly, like send someone a Twitter DM and be like, "Hey, just saw you're looking for this.

60. 11:43

    What do you think?" Um, but it's also helped us connect with a lot of partners, mentors, and get a lot of support from the community.

61. 11:50

    Being open source was critical. I kinda didn't really have a choice over here because the competition was free, so [chuckles] what am I gonna do? Um, but when you're in the dev tool space, it is pretty important.

62. 12:01

    Uh, open source is important to a lot of developers. They would like to have the ability to run it in different environments. Um-

63. 12:09

    It's also pretty important for a lot of enterprise users. In a lot of cases, they don't really want to run the software themselves, but having the option is very important to them because, uh, they've had-- most enterprises have had situations where a vendor goes out of business or decides to, um,

64. 12:26

    screw them in some other capacity. Um, it's also been really critical for engagement for us. We've had a lot of people in the community help out port it to different platforms, run in-- run the model in the web browser and whatnot.

65. 12:36

    So it's been pretty valuable for us. Um, this one is a little controversial. I'm not sure everyone agrees with this, but I feel pretty strongly that safety guardrails should be implemented at the application layer, not baked into the model itself.

66. 12:54

    Um, this was one of my learnings from my first attempt to build a QA testing application with GPT-4V. Um, it made no sense for that application to reject pictures of any picture that contain a human being.

67. 13:06

    I understand why they felt it was important. Um,

68. 13:10

    dev tools are kind of B to-- B to B, not B to C, so it's important to make it easy for developers to decide what guardrails they want and implement it in their model as opposed to just deciding it for all users.

69. 13:22

    Uh, I'm not saying this is not important at all. Uh, kind of makes sense if you're trying to build an assistant to bake that stuff right directly into the model.

70. 13:28

    But when you're building for developers, makes, makes, makes less sense.

71. 13:33

    Um, yeah, I believe pretty strongly now that tiny models are gonna run the world. Um, in computer vision more so perhaps than in text models, efficiency is really important.

72. 13:47

    Um, in a lot of cases, you're really worried about cost because you're processing video, and thirty frames a second at seven-tenths of a cent per second adds up very quickly and [chuckles] doesn't give you a lot of room to work with.

73. 14:01

    But there's also situations where you're really worried about privacy, uh, or latency, and therefore you wanna run the model really close to where decisions need to be made. Um, which is not to say big models are not useful.

74. 14:13

    I think they're very useful. I just think that we'll mostly be running them in our development environments, maybe for generating training data. Um, but the artifact that you're gonna want to deploy is most likely going to be a smaller model.

75. 14:27

    Um, another thing that was a little surprising to me was looking at the different things people were doing with Moondream. There were a lot of people building net-- new applications that weren't possible to do before because the model can understand language as well as images.

76. 14:47

    But there were also a lot of people doing traditional computer vision things with, um, with the model. It's like, is there a person in the scene or, uh, is there something suspicious going on?

77. 14:57

    Tell me where the, uh, where the, where the bus is in this picture of, uh, from a road camera. Um,

78. 15:05

    all of which was possible to do before we had transformers, like just train a YOLOv3 model or whatnot. The thing that was, uh... Yeah. The lesson I took, I took from this was, uh, prompting is a much better developer experience than having to train a custom model.

79. 15:20

    And so for a lot of developers that would be interested in incorporating vision into their applications, um, before they'd be like, "You know what? It's not worth me spending two weeks learning how to, like, collect data and annotate it and train my own custom model."

80. 15:35

    Um, giving them the option to say, "Hey, for fairly cheap, you can just in English describe what you want extracted from this image," um, makes it something that they actually consider doing it, doing now.

81. 15:48

    All right, um, I think I'm a little ahead of time, so I'm excited to maybe do a live demo if the demo gods smile upon me, but we'll see.

82. 15:55

    Um, in conclusion, uh, yeah, where is Moondream going? We're, we're not AGI people. I'm really focused on making it really easy for developers to build amazing applications with vision.

83. 16:09

    Um, there's a bunch of model improvements that I'm working on right now. Um, I'll talk about some. Um, right now we use seven twenty-nine tokens to represent an image, so you can only really send one image to the model at a time.

84. 16:22

    Uh, we're working on giving users the option to, like, give a more compressed representation to the, to the model, um, which makes sense if you're not trying to read text or something from the image.

85. 16:29

    If you're just trying to do classification or whatnot, that makes the model run a lot faster, which is important, especially if you're on CPU as opposed to GPUs, which can't do as much.

86. 16:37

    Um, CPUs can't do as much parallel compute, and so that sort of thing ends up being really important.

87. 16:43

    Um, we've also just raised a seed round, um, from Physis, Ascend, and also the GitHub Fund, which I forgot to include in the slide. Sorry, GitHub. Um,

88. 16:55

    this means more GPUs, but more importantly, it means I can finally get some sleep because we're able to get a couple more people to join the team. Uh, if you're interested, please reach out.

89. 17:03

    We have a contact email on the website, or just hit me up on Twitter. Uh, we also have an exciting release coming up later this summer that I'm super pumped for, so stay tuned.

90. 17:13

    Um, I think that's about it. So I have a couple of minutes left, I think, so I'm gonna try doing something that may not be the wisest idea, but we'll see how it goes.

91. 17:30

    All right, uh, I've turned the Wi-Fi off. This whole thing is running locally.

92. 17:37

    Um, so what this is gonna do is, like, um, start

93. 17:41

    taking my webcam in, and it's gonna use Moondream in a, in an infinite loop to describe what it sees,

94. 17:48

    and we can ask it different questions. So we'll see how that goes. [audience applauding] [chuckles]

95. 18:06

    And yeah, you can ask it different things. Uh, so let's say, "Is the person wearing glasses?" Um, you do have to tell the model to answer briefly if you want a yes or no.

96. 18:18

    Otherwise, it gives you, like, a, um, answer with a single word. Let's try that.

97. 18:27

    Yes. Okay, I'll take them off and... I can't see it. [laughs] [audience laughing]

98. 18:35

    Let's see. [laughs] Did it get it? [audience laughing] [audience applauding] Can you point it at us? Let's do that. I'll go back to the old prompt.

99. 18:48

    How many people? [laughing] Hey. [audience cheering]

100. 19:03

     All right. Well, that was it for me. Thank you all. [upbeat music]