AI Engineer World's Fair 2024
Moondream: how does a tiny vision model slap so hard?
Read the talk
Moondream: small models, careful supervision
A compact vision model becomes useful through a narrow task boundary, carefully generated training data, and an interface that lets developers ask for what they need.
From a talk by Vikhyat Korrapati
Questions, captions, coordinates and counts
Where is the parrot’s beak? Locating an object is one of the practical jobs for Moondream, alongside answering questions, captioning images and counting. Its creator, Vikhyat Korrapati, began working on the project after roughly nine years at AWS. The version presented here has fewer than two billion parameters and is available under Apache 2.0.
The opening examples show how these tasks require different outputs: a description for captioning, an answer for a visual question, or coordinates for localization. In the object-detection frame, a green bounding box encloses the parrot’s beak.
Across VQA v2 and GQA, Korrapati shows improvements over three months of releases and reports performance comparable to LLaVA 1.5, a model with seven billion parameters. That comparison establishes the opportunity: a much smaller model can be competitive on these visual-question-answering tasks. It does not establish equivalent capability across everything the larger model can do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Browser automation makes model control a requirement
Moondream began inside a different product idea: an agent for automated browser testing. The agent needed to see the user’s screen, describe the page and answer questions about it. Korrapati initially tried GPT-4V, but his early experience included refusals when images contained people. Anticipated latency and cost created further obstacles. For this application, control over the model became a product requirement.
The task was constrained enough to make building a small model plausible. Describing screens and answering questions about them did not require general intelligence. With a couple of RTX 3090s at home, Korrapati planned to train a small prototype, then rent larger cloud machines for a bigger version.
The small version worked better than he expected. He posted it on Twitter expecting modest interest and a quick return to the application. Instead, the response was large enough to change the project’s direction.
Automated-testing companies began asking whether they could use the model to describe browser screens. Other developers were interested too; Korrapati gives Open Interpreter a shout-out. He paused the testing application to concentrate on Moondream and find out how far a reusable vision component could go.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define the perception task, then choose the components
The task boundary determines the data and the evaluation. Moondream is intended as a developer tool for accurate image understanding with fewer hallucinations. Broad world knowledge and creative writing are outside that focus. Asking it to write a poem spends capacity on something the product does not need. Those exclusions make it easier to decide which training examples and benchmarks deserve attention.
Mathematical reasoning provides a concrete boundary. Solving problems in a benchmark such as MathVista is not the goal. For Korrapati’s example of a photographed differential equation, producing a LaTeX representation could be useful: it extracts what is visible. Solving the calculus problem is a separate capability that Moondream does not aim to acquire.
For this task and budget, he found training from scratch cost prohibitive without a meaningful benefit. Experiments with other pretrained models also made relatively little difference. Training data was the larger source of improvement, so that became the focus of the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Synthetic supervision must fit the smaller model
Korrapati reports that the latest release discussed was trained on around 35 million images. Obtaining useful supervision at that scale creates two immediate problems:
- Human annotation: High-quality annotation is expensive. He relays a rumor that some providers require seven-figure upfront commitments before engaging.
- Internet image–alt-text pairs: These are plentiful, but often noisy and formatted differently from the questions and answers the model needs to learn. Korrapati finds that noise particularly damaging when training small models.
Synthetic processing offers a way to reshape existing information into suitable training examples. But a powerful generator is not automatically a good teacher. Korrapati raises terms-of-use concerns about training on GPT-4 outputs, then identifies a technical problem: those outputs can depend on reasoning and world knowledge that the smaller model cannot reproduce.
His warning is that the student may learn to imitate the teacher’s confident language without learning the basis for its answers. The resulting response sounds plausible but includes unsupported details. The useful target is grounded supervision the small model can learn, not simply the most elaborate answer a larger model can produce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A harbor caption shows two ways synthesis goes wrong
The COCO example makes the problem visible. He describes COCO as roughly 200,000 images, with five short descriptions per image and object annotations that include coordinates. The desired transformation sounds straightforward: combine those short descriptions and annotations into a detailed caption containing their collective information.
Naively supplying that material to GPT-4 produces a fluent harbor description, but fluency conceals two different errors.
| Error source | What happens in the harbor caption |
|---|---|
| Unreliable annotation | A tiny feature that might be a person or a post becomes a definite person near the right side of the harbor. |
| Generated embellishment | The caption says that five yachts stand out from the rest. |
The first error turns ambiguous source information into certainty. The second adds an interpretation that the supplied information does not justify. Combining annotations into longer prose therefore needs more than a request for detail: the process must control which facts survive and what the generator is allowed to add.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn spatial narratives into grounded questions
Preprocessing becomes central in the next example, Google’s Localized Narratives dataset. Annotators describe an image aloud while moving the mouse over the region they are describing. This encourages detailed descriptions tied to spatial positions. A description of the girl in front playing a guitar carries both an activity and a relationship within the scene—useful supervision for spatial reasoning, which vision-language models often struggle with.
The processing slide lays out the stages used to turn that material into training questions:
- Correct transcription errors.
- Extract facts.
- Validate those facts.
- Create question–answer pairs.
- Filter unhelpful pairs.
- Generate absurd or irrelevant questions.
Fact extraction and validation give the pipeline explicit opportunities to clean the information before it becomes a question and answer. Filtering then removes unhelpful results, while the final stage introduces questions the image cannot support.
The pipeline used about 20 calls to Mixtral 8x7B per image. Korrapati estimates that generating training data consumed one or two orders of magnitude more compute than training Moondream itself. In his experience, that expense was justified because training-data quality moved model performance more than the other changes he tried.
He released the resulting dataset on Hugging Face. One generated example asks, “What theory does the kid have about the existence of pleasure in the image?” A photograph cannot establish that philosophical belief. These distractor questions teach an essential behavior: the model should not accept a question’s premise merely because the user supplied it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match real queries and break synthetic shortcuts
Even a simple formatting mismatch can expose a gap in synthetic training data. An early user submitted a straightforward question that Moondream could not answer. Korrapati traced the failure to its lowercase first letter: the training questions had not included that variation.
The fix was to add capitalization changes and typos to the training data. The broader requirement is distribution matching. Training inputs need to resemble the queries people actually submit, including their imperfections.
The generator’s own habits create another failure mode, which Korrapati discusses under model collapse. When asked to invent questions irrelevant to an image, Mixtral repeatedly produced questions about dinosaurs and aliens. Training on that pattern risks teaching the student a topic shortcut: reject questions about dinosaurs and aliens. That is different from learning whether the image supports the question.
The generation process therefore needs variation beyond the generator’s habitual choices. For synthetic captions, one option is to provide the image’s alt text as additional context. The instruction must allow that text to be noisy or irrelevant and use only facts that are relevant. This introduces outside information without requiring the model to trust all of it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Community feedback and the option to self-host
The community helped establish that a reusable vision model might be more valuable than the original testing application. Other developers wanted to build their own applications around the same capability. Direct conversations on Twitter gave Korrapati a short path from discovering a need to asking a potential user about it. The community also brought partners, mentors and practical support.
Free competing tools influenced the decision to release Moondream as open source, but the resulting benefits went beyond price:
- Developer portability: Users could run the model in the environments their applications required.
- Enterprise independence: An organization could prefer managed operation while retaining the option to run the software itself if a vendor disappeared or changed the terms of the relationship.
- Community contributions: Contributors helped port the model to additional platforms, including execution in a web browser.
The ability to self-host therefore matters even to customers who do not initially intend to operate the model themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let the application choose its guardrails
For developer tools, Korrapati favors placing safety guardrails at the application layer. His browser-testing experience explains why: refusing to process an image because it contains a person blocked a legitimate task the application needed to perform. A restriction appropriate to one product can prevent another product from working.
In his view, developers building on a model should be able to choose and implement safeguards suited to their applications. He distinguishes this infrastructure role from a consumer assistant, where building safeguards directly into the model can make sense. The position is about where control belongs in the product stack, not whether safety matters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploy small models close to the decision
Vision applications make efficiency especially consequential because a video stream creates a continuing inference workload. To illustrate how continuous video processing accumulates cost, Korrapati uses 30 frames per second at a stated seven-tenths of a cent per second. This is a cost illustration, not a Moondream serving-price quote. Repeated processing can quickly consume the margin available to an application.
Privacy and latency create separate reasons to run inference close to where decisions happen. Korrapati expects smaller models to become the deployed artifacts for many such applications, while larger models remain useful in development—for example, generating the smaller model’s training data. That division gives the larger model a valuable role without requiring every production inference to use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompting makes familiar vision tasks easier to adopt
Some users built applications that depended on understanding language and images together. Others used Moondream for familiar computer-vision requests: determine whether a person is present, ask whether something suspicious is happening, or locate a bus in a road-camera image. The demand was not limited to entirely new capabilities.
Established models such as YOLOv3 could already address tasks such as object detection. The change was the amount of work a developer had to take on before trying the idea.
| Approach | Work required to specify the task |
|---|---|
| Train a custom vision model | Learn the workflow, collect and annotate data, then train a model. |
| Use a promptable vision-language model | Describe in English what to extract from the image. |
Korrapati describes a developer weighing an illustrative two weeks of learning against the value of adding a vision feature. An inexpensive prompt can make that same feature worth trying. Prompting changes the developer experience, even when the underlying task was already technically possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reduce image tokens and expand the team
The roadmap keeps the same product boundary: help developers build vision applications, rather than pursue general intelligence. At the time of the talk, Moondream used 729 tokens to represent an image, which Korrapati links to practical use with only one image at a time.
The planned improvement was an option for a more compressed image representation. The useful tradeoff depends on the task. Reading text requires visual detail that aggressive compression could discard; classification may work with a less detailed representation. Reducing the number of image tokens is intended to speed up inference, particularly on CPUs, which have less parallel compute available than GPUs.
The closing update announces a seed round, with Felicis and Ascend shown on the slide and the GitHub Fund added verbally. Funding would support more GPUs and, more importantly for the founder’s workload, additional people. Korrapati invites potential hires to contact him through the website or Twitter and previews another release planned for later that summer. These are the plans announced at the time of the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An offline webcam loop, redirected by a question
For the live demonstration, Korrapati turns off Wi-Fi and states that the system is running entirely locally. A webcam supplies images to Moondream in a repeating loop, and the model describes what it sees. The question applied to those incoming images can be changed while the demonstration runs.
He changes the task to “Is the person wearing glasses?” Response formatting needs its own instruction: asking the visual question alone does not necessarily produce a short answer. He refines the prompt with “Answer with a single word.” A “Yes” response follows, and he removes his glasses to test the changed scene.
At a request to point the webcam toward the audience, he turns it outward and asks, “How many people?” The closing interaction uses the same local image loop for scene description, an attribute question and a request to count—changing the requested behavior through language.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Project source, examples, and links for running Moondream. Current instructions and variants differ from the 2024 model.
First-party results for the model used as the talk's comparison, distinguishing full fine-tuning from LoRA checkpoints.
Explore synchronized narration and mouse traces, with annotation formats and data-loading resources.
Examples and task definitions for the visual mathematical reasoning benchmark discussed as outside Moondream's intended scope.
Further reading
Original research behind the SigLIP vision component named in the talk; the paper title itself was not spoken.
Background on the small language backbone and its synthetic-data approach.
- LNQA datasetDataset
Korrapati's question-answering dataset derived from Localized Narratives.
- How Korrapati built LNQAArticle
Companion methodology covering fact extraction, distractor diversity, prompting, and filtering.
Read the complete timestamped transcript
- 0:00
[upbeat music] Hi, my name is Vik. I work on a model, an open source vision model called Moondream.
- 0:17
Um, a little bit about myself before I dive into Moondream. Uh, I was at A- AWS for about nine years, um, before I started working on this model. Um, looking at where the stock price is going, I'm not sure if that was the right financial decision, but I'm very happy with the work I'm doing.
- 0:33
Uh, so let's dive into it. We'll talk about Moondream a little bit. Um, it is a tiny vision language model. It's less than two billion parameters, so it can run anywhere, and it's open source, Apache 2.0, so you can use it to do anything.
- 0:49
Um, here's some examples of things you can do with Moondream. You can ask it questions about images, um, you can
- 0:57
caption images, uh, it can detect spe-specific objects inside of images. So here I asked it to tell me where the peak is, and it gives me coordinates. Um, it can count stuff.
- 1:08
It can do all sorts of things. Um, I had the audacity to title my talk, "How can a tiny vision model slap so hard?" So I have to back things up a little bit.
- 1:17
Um, and so here's me doing that. Um, these are two vision, uh, benchmarks, vi-vision qu-question answering benchmarks. One's called VQA v2, the other is called GQA. Um, as you can see, Moondream has been steadily improving over the releases I've made over the last three months.
- 1:33
Uh, I've included a reference line over there for LLaVA 1.5, which is a popular seven billion parameter, uh, vision model. So this shows you that, um, Moondream gives you compare-- performance that's comparable to, uh, models that are about four times bigger than it.
- 1:50
Um, I didn't really set out to build a vision model, um, so I kinda got roped into it. I was originally trying to build a-an application that required an AI agent, so I needed to be able to see what was going on on the user's screen and, um, have it describe what's on the browser page for QA
- 2:06
testing automation. Tried to do this at first with, uh, GPT-4V, but, um, there were too many safety refusals back then. Like, if there was any human being present in the image, it would just refuse to process it.
- 2:19
Um, it was also gonna be really slow and expensive, and so I realized if this is a product I'm trying to build, I really need to have control over the model itself.
- 2:26
So I figured, you know what? How hard can it be? Let me just go try and, um, build this model myself. Now, the task I was trying to perform here was fairly constrained.
- 2:34
Um, I, I just needed to describe s-screens and answer questions about screens, so, um, it, it doesn't need to be generally intelligent. Um, I had a couple of 3090s at home, so I figured I'd train a small version of the model at home and then rent some beefier machines in the cloud to go train a bigger version.
- 2:52
Um, and, uh, once I got done training the small version, I was like, "Hey, this actually works pretty well," so I posted it on Twitter. I thought, "You know what?
- 2:59
I might get 20 likes off of this, and then I'll move on with my side project at the time." Um, it blew up far beyond expectations. I was a little surprised, pleasantly surprised, but surprised nonetheless.
- 3:10
And I immediately started seeing, um, other automated testing companies reach out and be like, "Hey, can I use this to describe browser screens? Um, 'cause this would work really well for us."
- 3:20
Um, as well as other companies. Shout out to our friends at Open Interpreter from Seattle, um,
- 3:26
that basically told us that they were... I figured, you know what? Like, this is getting a lot of traction. Let me pause on the whole automated testing app for a couple of weeks and focus on Moondream and see where it goes.
- 3:37
Um, yeah, so let me dive into a couple of the technical details, um, around what makes the model succeed despite being small. The first thing we did, um, that I think really helped was, um, deciding what problems the model should solve and what it should not solve.
- 3:55
So Moondream wants to be a developer tool. We focus on being really accurate and not hallucinate. Um, it doesn't really have a know-- a lot of knowledge about the world.
- 4:04
So, um, if you ask it to write a poem, it's probably not gonna help you. It's really focused on answering questions, like giving you... helping you understand images. Um, this is really important because it affects the type of data that you use and the sort of benchmarks that you wanna focus on.
- 4:20
Uh, there's a popular vision language model benchmark called MathVista, which measures how good models are at solving math problems. You take a picture of a differential equation, and you see whether the model can solve it.
- 4:30
That was, uh, an example of a non-goal for us because we just want the model to be good at looking at images. The, the most we do is probably generate a LaTeX representation of the problem.
- 4:40
We don't really wanna even attempt to [chuckles] try and solve calculus. Um, it was not pre-trained from scratch. We use, um, um, we, we fuse a vision encoder called SigLIP from Google, uh, with a pre-trained text model called Phi, uh, 1.5 from Microsoft.
- 4:58
Uh, the notable thing over here is Phi 1.5 was also trained on mostly synthetic data, which is very similar to our pipeline, so it works very well. Um,
- 5:07
for this sort of task, pre-training from scratch doesn't really make a difference, uh, as opposed to using pre-trained models, and it is cost prohibitive. So unless you wanna get those brownie points for saying you trained it from scratch, uh, it's probably not worth doing.
- 5:20
We experimented with a bunch of different other models, uh, as they were released, and nothing really made too much of a difference. What does make a difference, though, is, uh, training data.
- 5:29
Um, the latest release of Moondream is trained on around 35 million images, and, uh, the problem is, especially when you're on a budget, like high-quality, multi-modal training data is really hard to come by.
- 5:43
Um, there's companies that, um, there's a lot of companies out there that'll annotate data with humans, um, but it, it's really expensive, and I've heard a rumor recently that they won't even talk to you anymore unless you're willing to sign an upfront seven-figure commitment.
- 5:58
There's a lot of data on the internet, images, alt text pairs. Um, the problem with this is it's often not in the format you want it to be, and it's really noisy, and the noise is really problematic when you're, uh, when, when you're training small models.
- 6:13
Um, and so synthetic data is a way to solve this, uh, where you use that alt text information, process it. It's a bit of an open secret that a lot of people are training on outputs from GPT-4.
- 6:24
Um, you probably don't want to do that. Um, besides being questionable in terms of, terms of use, uh, it's often not helpful. GPT-4 is a very powerful model. It has reasoning capabilities and knowledge that your small model is never going to be able to get.
- 6:39
And so when you train it on GPT-4 outputs, what it learns instead is to hallucinate. It's gonna generate plausible-sounding outputs that include details that it cannot possibly memorize, uh, and so you end up in trouble.
- 6:51
So, um, this is a little important. I'm going to go a little more technically detailed for a couple of minutes to dive into how to do synthetic data, so bear with me for a sec.
- 6:59
We'll pop back up. Um, here's an example of how not to, not to do it. Uh, COCO is a dataset. It has around two hundred K images. Each image has five short descriptions and a bunch of, uh, object annotations, uh, with like, "Hey, there's a bicycle at these coordinates," and whatnot.
- 7:14
Um, and let's say you wanna take those short descriptions and these an-- these object annotations and generate more detailed captions that include the union of all the information present over here.
- 7:22
If you just naively call GPT-4, uh, with this information, uh, it generates this. Not important to read all of it, uh, but there's, uh, two important things to note.
- 7:31
The first is that, um, it hallucinates. It says, uh, in the second paragraph, there's a person near the right side of the harbor. I, I think there's, like, a person way back the...
- 7:43
There's, like, five pixels there that may be a post. It may be a person. We don't really know. That's because the object annotations were bad, but besides that, like, um, the model is also taking a lot of creative liberties over here, like saying there's five yachts standing out from the rest and whatnot.
- 7:57
Um, and so this is, uh... You need to do a little more pre-processing of your data before you feed it to the model. Um, here's another example. There's a dataset from Google called Localized Narratives.
- 8:09
Um, the task annotators here, uh, annotators are given here is, um, verbally describe this image, and as you're describing the image, uh, hover your mouse over the part of the image that you're describing.
- 8:20
So it's nice in that it encourages people to create really detailed descriptions that capture spatial positioning, um,
- 8:26
in the image. So for example, here it says the girl in the front is playing the guitar and whatnot, and spatial reasoning is something that vision language models typically tend to struggle with.
- 8:35
Um, I ended up having to build a fairly sophisticated data processing pipeline to get really good results with this. Um, not really important to dive into the details over here, but the important thing to note is, A, it gets really expensive.
- 8:47
Um, each image ends up being twenty LLM calls, and the LLM here is Mixtral 8x7B, so it gets pretty expensive. Um,
- 8:57
b-but it was necessary. Uh, the training data is the biggest needle mover in terms of model performance, and because of this, uh, I'd say we spent, like, maybe one or two orders of magnitude more compute on generating training data than actually training the model itself.
- 9:12
Um, a couple-- So yeah, this particular dataset we've open-sourced. It's, uh, available on Hugging Face. Here's an example of the type of questions it generates for this image. Um,
- 9:24
the-- there's an interesting question towards the end: What theory does the kid have about the existence of pleasure in the image? I'll talk about that in a sec. But basically you wanna generate a few distractor questions so the model knows to not always agree with the question that the user is ask-- is asking.
- 9:40
Um, so yeah, a couple of the challenges involved in working with synthetic data. Um,
- 9:45
there was an interesting incident I had, uh, early on where a user was like, "Hey, I asked a relatively simple question. Why can't the model answer this?" Um, and when I looked at it, it turned out that they didn't capitalize the first letter in their question, and the model had never seen anything like that during training.
- 9:59
So I was like, "What do I do over here?" Um, and so it's really important for you to make sure that your training data has the, the same rough distribution as your real-world query.
- 10:09
So ended up adding, like, an extra step where we artificially inject, like, capitalization issues and typos and whatnot into the model before training it. Um, there's also this risk of what we call model collapse, where your model has biases inherent to it.
- 10:23
So for example, if you try to ask Mixtral to generate distractor questions, "Hey, just, uh, generate a question that's completely irrelevant to the image," it'll always generate something about dinosaurs and aliens.
- 10:32
And so if you train your model on that, it'll instead learn, uh, to say, uh, "Hey, if the question is about dinosaurs and aliens, always say no," which doesn't really help.
- 10:41
Um, and so there you need to in-inject, like, some entropy into the process of generating synthetic data to avoid this. Uh, in the case of synthetic captioning, you can do something like, "Hey, describe this image," but also consider the alt text on the image, which may be noisy, may be irrelevant, but if it is relevant, use relevant
- 10:55
facts from that. Uh, and that tends to help a lot.
- 11:00
All right, so popping back up, um, there were a couple of important learnings I had over the last three months, uh, that I would like to share with all of you.
- 11:12
Uh, the first was, um, the community was really critical in this whole journey. Seeing that original engagement that we got, uh, from the Moondream release helped me realize that, hey, maybe this is more valuable than that QA testing application that I was working on, 'cause a lot of people have a need for this to build applications like
- 11:29
that. Uh, coming from an enterprise-ish company, uh, it's been really valuable. It's been refreshing to be able to just talk to customers directly, like send someone a Twitter DM and be like, "Hey, just saw you're looking for this.
- 11:43
What do you think?" Um, but it's also helped us connect with a lot of partners, mentors, and get a lot of support from the community.
- 11:50
Being open source was critical. I kinda didn't really have a choice over here because the competition was free, so [chuckles] what am I gonna do? Um, but when you're in the dev tool space, it is pretty important.
- 12:01
Uh, open source is important to a lot of developers. They would like to have the ability to run it in different environments. Um-
- 12:09
It's also pretty important for a lot of enterprise users. In a lot of cases, they don't really want to run the software themselves, but having the option is very important to them because, uh, they've had-- most enterprises have had situations where a vendor goes out of business or decides to, um,
- 12:26
screw them in some other capacity. Um, it's also been really critical for engagement for us. We've had a lot of people in the community help out port it to different platforms, run in-- run the model in the web browser and whatnot.
- 12:36
So it's been pretty valuable for us. Um, this one is a little controversial. I'm not sure everyone agrees with this, but I feel pretty strongly that safety guardrails should be implemented at the application layer, not baked into the model itself.
- 12:54
Um, this was one of my learnings from my first attempt to build a QA testing application with GPT-4V. Um, it made no sense for that application to reject pictures of any picture that contain a human being.
- 13:06
I understand why they felt it was important. Um,
- 13:10
dev tools are kind of B to-- B to B, not B to C, so it's important to make it easy for developers to decide what guardrails they want and implement it in their model as opposed to just deciding it for all users.
- 13:22
Uh, I'm not saying this is not important at all. Uh, kind of makes sense if you're trying to build an assistant to bake that stuff right directly into the model.
- 13:28
But when you're building for developers, makes, makes, makes less sense.
- 13:33
Um, yeah, I believe pretty strongly now that tiny models are gonna run the world. Um, in computer vision more so perhaps than in text models, efficiency is really important.
- 13:47
Um, in a lot of cases, you're really worried about cost because you're processing video, and thirty frames a second at seven-tenths of a cent per second adds up very quickly and [chuckles] doesn't give you a lot of room to work with.
- 14:01
But there's also situations where you're really worried about privacy, uh, or latency, and therefore you wanna run the model really close to where decisions need to be made. Um, which is not to say big models are not useful.
- 14:13
I think they're very useful. I just think that we'll mostly be running them in our development environments, maybe for generating training data. Um, but the artifact that you're gonna want to deploy is most likely going to be a smaller model.
- 14:27
Um, another thing that was a little surprising to me was looking at the different things people were doing with Moondream. There were a lot of people building net-- new applications that weren't possible to do before because the model can understand language as well as images.
- 14:47
But there were also a lot of people doing traditional computer vision things with, um, with the model. It's like, is there a person in the scene or, uh, is there something suspicious going on?
- 14:57
Tell me where the, uh, where the, where the bus is in this picture of, uh, from a road camera. Um,
- 15:05
all of which was possible to do before we had transformers, like just train a YOLOv3 model or whatnot. The thing that was, uh... Yeah. The lesson I took, I took from this was, uh, prompting is a much better developer experience than having to train a custom model.
- 15:20
And so for a lot of developers that would be interested in incorporating vision into their applications, um, before they'd be like, "You know what? It's not worth me spending two weeks learning how to, like, collect data and annotate it and train my own custom model."
- 15:35
Um, giving them the option to say, "Hey, for fairly cheap, you can just in English describe what you want extracted from this image," um, makes it something that they actually consider doing it, doing now.
- 15:48
All right, um, I think I'm a little ahead of time, so I'm excited to maybe do a live demo if the demo gods smile upon me, but we'll see.
- 15:55
Um, in conclusion, uh, yeah, where is Moondream going? We're, we're not AGI people. I'm really focused on making it really easy for developers to build amazing applications with vision.
- 16:09
Um, there's a bunch of model improvements that I'm working on right now. Um, I'll talk about some. Um, right now we use seven twenty-nine tokens to represent an image, so you can only really send one image to the model at a time.
- 16:22
Uh, we're working on giving users the option to, like, give a more compressed representation to the, to the model, um, which makes sense if you're not trying to read text or something from the image.
- 16:29
If you're just trying to do classification or whatnot, that makes the model run a lot faster, which is important, especially if you're on CPU as opposed to GPUs, which can't do as much.
- 16:37
Um, CPUs can't do as much parallel compute, and so that sort of thing ends up being really important.
- 16:43
Um, we've also just raised a seed round, um, from Physis, Ascend, and also the GitHub Fund, which I forgot to include in the slide. Sorry, GitHub. Um,
- 16:55
this means more GPUs, but more importantly, it means I can finally get some sleep because we're able to get a couple more people to join the team. Uh, if you're interested, please reach out.
- 17:03
We have a contact email on the website, or just hit me up on Twitter. Uh, we also have an exciting release coming up later this summer that I'm super pumped for, so stay tuned.
- 17:13
Um, I think that's about it. So I have a couple of minutes left, I think, so I'm gonna try doing something that may not be the wisest idea, but we'll see how it goes.
- 17:30
All right, uh, I've turned the Wi-Fi off. This whole thing is running locally.
- 17:37
Um, so what this is gonna do is, like, um, start
- 17:41
taking my webcam in, and it's gonna use Moondream in a, in an infinite loop to describe what it sees,
- 17:48
and we can ask it different questions. So we'll see how that goes. [audience applauding] [chuckles]
- 18:06
And yeah, you can ask it different things. Uh, so let's say, "Is the person wearing glasses?" Um, you do have to tell the model to answer briefly if you want a yes or no.
- 18:18
Otherwise, it gives you, like, a, um, answer with a single word. Let's try that.
- 18:27
Yes. Okay, I'll take them off and... I can't see it. [laughs] [audience laughing]
- 18:35
Let's see. [laughs] Did it get it? [audience laughing] [audience applauding] Can you point it at us? Let's do that. I'll go back to the old prompt.
- 18:48
How many people? [laughing] Hey. [audience cheering]
- 19:03
All right. Well, that was it for me. Thank you all. [upbeat music]