AI Engineer World's Fair 2026
Evaling Video Slop
Read the talk
Evaling Video Slop: Put Judgment Inside the Generation Loop
Video quality depends on what happens across time. Maor Bril walks through building a fast evaluator, discovering its shortcuts, and repairing the data behind its judgments.
From a talk by Maor Bril
Before you start: Basic familiarity with generative models, evaluation datasets, and model training will help; no prior video-processing experience is required.
Generating a video is easier than deciding it is good
How do you decide whether a generated video is good enough to use? Kling, Seedance, Veo, and Sora have made generation increasingly capable, but evaluation often still means watching the result and making a judgment by eye. Maor Bril, introducing this work after just over two years at Character, starts with that gap: improvements in generation have outpaced the tools for judging what was generated.
Compared with studio production, Bril characterizes video generation as almost free. Producing usable footage remains harder. A character gains an extra limb; a door appears to open and close simultaneously; someone hovers instead of moving under plausible physics. Human review catches these failures, but it also becomes part of the production cost. Long-form generated video typically emerges from many short generations and substantial editing, rather than one successful request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Frame agreement does not establish a coherent story
The first evaluation tools come from image and text workflows. CLIPScore measures image–text compatibility, making it useful for asking whether a frame matches a prompt. LPIPS measures perceptual distance between images; applying it across frames can flag changes, although those changes may be legitimate motion rather than unwanted drift. These are useful signals, but neither establishes whether a sequence tells the intended story.
Video introduces requirements that only become meaningful across time:
| Requirement | Concrete question |
|---|---|
| Story and physics | Does the character walk downstairs, or hover? |
| Character continuity | Is this the same character across shots? |
| Pacing | Does moving between places take plausible time? |
| Audio synchronization | Does the door-slam sound coincide with the visible impact? |
A sequence can contain individually convincing frames while failing these checks. Video evaluation has to assess events and their relationships, not just the appearance of their component frames.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a repeatable judge before making it fast
A multimodal foundation model can inspect a video and answer questions about it, but simply using an LLM as a judge introduces another source of variation. Different people write different prompts, and the same model can produce very different judgments. Asking whether a clip is consistent or follows its prompt also leaves the broader quality question underspecified. Bril's first iteration was therefore a repeatable evaluation harness rather than an isolated model call.
The harness combines frame metrics with a consistently prompted LLM judge. Humans annotate the reports it produces, and their feedback goes back into the judge's prompt to align its assessments with the annotators' preferences. This creates a rerunnable process with a calibration loop: generate a report, inspect its judgments, correct them, and use those corrections in subsequent judging. The remaining problem is operational. Running the full process is slow and expensive, especially when users need to generate and revise many clips during creation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Catch drift before it becomes a finished video
Evaluation becomes more useful when it moves closer to the user and earlier in generation. Consider two starting frames intended to seed different shots. If the character's identity has already drifted between those images, the defect is visible before either video is generated. Correcting the starting frame avoids paying to animate an inconsistency that was present from the beginning.
The same principle applies at the next level of assembly. A three-, four-, or five-minute video may contain many short clips. If a six-second generation drifts, evaluate and regenerate that clip before combining it with the rest. This does not eliminate later checks: story coherence, pacing, and sound alignment need temporal context. It places each check at the earliest point where the necessary evidence exists, so a local failure can receive a local correction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Distill explanations into a small VLM
To make those checks practical near generation time, Bril distilled the committee of experts into a small vision-language model, or VLM. The desired output is more than a binary judgment that a video is low quality. It should explain the failure: an extra limb, a violation of physics, or audio that is out of sync. Those diagnoses give the creation process something specific to repair.
Bril reports approximately three seconds to score a 15-second video with the trained small VLM. The talk does not specify the checkpoint, hardware, frame sampling, or timing boundaries for that result. A larger model performed better in his tests, but was significantly slower; the additional quality did not justify the delay for this workflow. The selection criterion was useful judgment at creation-time latency, rather than the strongest evaluator regardless of response time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Train on comparisons, then inspect what they teach
Absolute quality scores introduce ambiguity before training even begins. Ask several people to rate a video's storytelling from one to ten, and the same clip might receive a four, five, six, or eight. Bril argues that asking which of two videos tells a better story is easier to agree on. The training data therefore used pairwise preferences—A versus B—rather than relying on a shared interpretation of an absolute scale. High-quality internet videos supplied positive examples; negatives came from corrupting good footage or generating poor footage.
The first version was confidently wrong. Bril reports that V1 awarded camera work 9.2 for a four-second shot in which the camera never moved. The example looked like a still image of the same character, yet the evaluator praised its cinematography. Other videos received positive physics judgments despite hovering ghosts and flying people.
The failures exposed what the dataset had actually taught. V1 learned surface coherence, artificial artifacts, and visual gloss instead of the intended quality axes. Pairwise training had not prevented shortcut learning: if the easiest difference between the preferred and rejected examples is their finish, a model can learn to reward that finish without learning storytelling or physics. The comparison format helps define a judgment; the data still determines which evidence the model uses to make it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repair the dataset without building an AI detector
The next revision paired real footage with AI footage. Bril had initially avoided that approach because it introduces an obvious confound: if human footage always means good and AI footage always means bad, the evaluator can become an AI detector rather than a video-quality judge. Recognizing provenance would satisfy the training labels without solving the intended task.
He applied two safeguards:
- Consistent encoding: Put both sides of the comparison through consistent encoding so artificial differences in the files do not provide an easy cue.
- Consistent annotation: Use the same method to annotate every evaluation axis on both videos, rather than judging real and generated footage under different standards.
Bril reports that the revised dataset produced a substantial improvement, but supplies no numerical validation result. The repair targets the source of the earlier failure: the relationship between the examples, the labels, and the evidence the evaluator can exploit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the generation agent a way to verify its work
With a usable evaluator, the creation system moved from a complex fixed pipeline to an agentic workflow. A pipeline can work well for a narrowly defined use case, but users introduce their own stories, characters, images, and voices. Those variations push the process beyond its original assumptions. Quality-validation tools let an agent inspect what it has produced, adapt to those differences, and repair problems as it goes.
The resulting design connects three decisions: compare alternatives, evaluate the axes that actually matter, and put those evaluations inside the generation loop. Storytelling, pacing, and physics need explicit supervision; they should not be expected to emerge from a generic quality score. The final criterion for the prepared talk is the intended story: the evaluator should help the system produce a video that communicates it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Audio alignment is narrower than understanding sound
The audio questions make the evaluator's boundaries more concrete. Bril describes a separate check for sound quality and intelligibility, followed by matching visual events to audio. The model can receive a video alone or the video together with its generation prompt. If the prompt describes a door slamming, it can guide the model toward the corresponding visual key frame.
The next operation is temporal correlation. In Bril's example, the door slams at frame six. That frame has a timestamp, and the evaluator looks for a spike in the audio at that time. Detecting a coincident spike does not establish that the sound is actually a door slam; it checks alignment without fully identifying the event acoustically.
Once visual-event and audio-peak timestamps are available, the alignment check can be expressed directly. This Python example uses an illustrative door-impact timestamp of 1.20 seconds, candidate peaks, and an application-selected tolerance of 0.08 seconds:
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Alignment:
nearest_peak_s: float | None
offset_s: float | None
within_tolerance: bool
def check_alignment(
event_s: float,
audio_peaks_s: list[float],
tolerance_s: float,
) -> Alignment:
if tolerance_s < 0:
raise ValueError("tolerance_s must be nonnegative")
if not audio_peaks_s:
return Alignment(None, None, False)
nearest = min(audio_peaks_s, key=lambda peak: abs(peak - event_s))
offset = nearest - event_s
return Alignment(nearest, offset, abs(offset) <= tolerance_s)
door_impact_s = 1.20
audio_peaks_s = [0.40, 1.24, 2.10]
result = check_alignment(door_impact_s, audio_peaks_s, 0.08)
print(result)
The signed offset distinguishes an early peak from a late one. The function intentionally returns an alignment result, not a claim about which sound occurred.
Lip synchronization remains unresolved in Bril's system at the time of the talk. Human talking heads at least offer identifiable lips for analysis. Animated characters are harder: their mouth movements may have no direct correspondence with human speech articulation. The team is still working through that problem rather than treating event-level audio alignment as a lip-sync solution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep human taste in the calibration loop
Human evaluation introduces its own question: whose taste should the judge learn? Bril's answer returns to JudgeJudy, the report-and-annotation harness. People can inspect its reports and annotate the judgments. The team periodically spends 10–15 minutes annotating videos across multiple axes. Rather than asking everyone to evaluate one video on ten dimensions, Bril randomizes the annotation tasks.
Those annotations calibrate the AI judges, whose resulting judgments then become training data for the next version of the model. This makes alignment an ongoing process rather than a one-time label collection exercise. It takes time, and it does not immediately dissolve disagreements over taste: something Bril considers excellent may leave another reviewer unconvinced. Human feedback continually adjusts the evaluator as those preferences become visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose distillation for the economics of the workflow
The small VLM came from the Qwen family. Bril had tried other models, but the team already had good experience post-training Qwen for other use cases. The available setup worked, and its performance was good enough. He does not identify the other candidates or the exact Qwen variant; the choice is presented as a practical fit rather than a comprehensive model ranking.
An audience member then asks whether the approach makes sense for a domain with hundreds or roughly a thousand videos. Bril separates serving capacity from the reason to distill. He describes deployment ranging from one model instance on one GPU to a hundred instances, but his primary motivation was speeding up creation. A smaller operation can still use the committee of experts—CPU and GPU metrics combined with frontier-model judges—without training its own distilled evaluator.
| Approach | Main investment | Operational trade-off |
|---|---|---|
| Expert committee | Integrate metrics and judges | Slower evaluation; potentially stronger judgments |
| Distilled small VLM | Curate data and train a model | Faster evaluation, plus model-serving costs |
The decision balances dataset curation and training effort against serving cost, latency, and evaluator quality. Bril speculates that a newly available model might judge better, but offers no measured comparison. Expensive judging may be acceptable for one or two videos; at thousands or tens of thousands of evaluations per day, its costs accumulate. There is no fixed video-count threshold in the talk: the relevant question is whether faster creation and repeated use repay the investment in the smaller evaluator.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A reusable harness, with tracing still requested
The final question concerns OpenTelemetry traces from the LLM judges. Bril confirms that the repository does not export those traces at the time of the talk. He describes the repository as a harness that can connect different agents and LLMs; internally, the team also runs a service version with an agentic harness and the metrics it needs.
Bril accepts the request and commits to adding OTel telemetry. The public README inspected on August 28, 2026 documents OpenTelemetry metrics export, which does not establish delivery of the requested judge traces. The closing integration question matters because evaluation is now part of creation itself: the harness must fit the surrounding agent system, and its judgments need to be observable there.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Multimodal evaluation harness combining automated metrics, AI judges, human annotations and calibration, with runnable examples.
Original paper on evaluating image–caption compatibility without reference captions.
Official implementation and examples for measuring perceptual distance between images.
Further reading
Speech mean-opinion-score prediction system with pretrained inference examples from the VoiceMOS Challenge 2022 work.
Official vision-language model repository with model variants, video-understanding examples and fine-tuning guidance.
Read the complete timestamped transcript
- 0:00
[upbeat music] [clears throat] So, hi, I'm, uh, Maor. I've been with Character for a bit over, uh, uh, two years, and we'll talk about, um, AI slop, right?
- 0:20
Um, um, I think that, you know, when we look at video generations as a whole, right? We have, like, two kind of, uh, parallel tracks. One is the, the video ge-generation, which became insanely good from, from models like Kling and Seedance and, uh, Veo and Sora.
- 0:40
We, we still re-re-remember Sora. But, but, but the part that got left behind is how we evaluate the quality of the video that was, um, uh, generated, right? So on, on the one hand, um, we, we, we still kinda squint at it and decide whether or not it, it's, i-it's good.
- 0:58
But on the other hand, we know that the generation has gotten a lot better. And when, when we, when we look at, at X or, uh, or whatever, um, uh, social you're, you're consuming your content on, there are a lot of guides on how to create am-amazing videos with, um, this model or, or that.
- 1:16
So the, the hard part wa-was never how to make video. The hard part was how do we generate, um, um, good enough video, and how do we judge h-if the video is, is good enough.
- 1:28
So, so now we've gone to a world where the, the generation of video is basically free, right? Free as... And especially when you compare it to how much studios, uh, would charge.
- 1:38
Um, and-- but, but the problem is the most-- the grand majority of videos that is generated is not that good, right? We, we have, like, uh, a l-lot of hallucinations, like a third limb, uh, opening and closing the, the door at the same time, hovering, physics, um, e-e-et cetera.
- 1:54
So unfortunately, in order to get hi-high-quality content, we need a human to judge. And, uh, I don't know, uh, when was the last time you've seen how someone is creating these, uh, l-long form, um, um, generated video.
- 2:09
It's us-usually a lot of shorter generations and a lot of editing.
- 2:13
The problem is because we're using a lot of the tools that we built for the text-- for the text era, for the image era, for videos, right? We're using things like CLIP Score, which is, is great to, to, to judge a single frame.
- 2:27
Um, uh, things like, um, um, uh, uh, LPIPS will help us kinda detect the drift between f-frames. But we don't have... Uh, um, I mean, but the problem is w-when you kinda combine all of these together, all these tools are good at watching the individual frames.
- 2:44
They're good at checking this, this, this, this. Does this, uh, uh, specific frame, does it match the, the, the prompt that generated it, right? Um, it will check consistency between frames, and it will check whether or not it matched the prompt that, that drove it.
- 2:59
But, but what it won't do, uh, it doesn't tell you if, um, uh, if, if it, if you told, if you told the story that you me-you meant to tell, right?
- 3:07
If you think about what is video, video is a storytelling medium. Vi-video is just another form on how we tell a story, right? For, for, um, a-a-any type of story.
- 3:17
So, so one of, one of the things we have to look at, does it tell the actual story? Does the physics make sense? Like for example, if we want a video of a character walking down stairs, does it actually walk or, or hover?
- 3:28
Um, um, does, does the character stay the same character across multiple shots? Um, does the pacing make sense? Like, you know, for example, people take time, uh, uh, going from one place to another.
- 3:41
We need to make sure that the pacing makes sense as well. And especially when we add audio, we wanna make sure that the audio is kinda synced with, with, with the imagery.
- 3:49
Like, for example, if someone is slamming a door, we want that, that, that, that, that s- that sound of the s- door being slammed to be exactly when the door is actually being slammed.
- 3:59
Now, the, the next it-, uh, iteration we all, uh, went to a while ago, we, we started using LLM, uh, as a judge for everything. And we have am-amazing foundational models that we just throw, uh, vi-videos at them.
- 4:11
The problem with them is that, A, they're slow, B, they're on- they're only as good as your prompts, and multiple people will prompt multiple ways, and the same model may respond in a very, very different way.
- 4:22
And sometimes the prompt we use, like, is it consistent? Um, uh, does this match the, um, uh, the prompt? But, but then the question we really care about, is it good?
- 4:31
And the answer varies. So [sighs] oops, sorry about that. So our first iteration is like, let's take all these things and build a repeatable benchmark on how we test video that w-we, we can rerun over and over and over again.
- 4:47
So, so that combines both metrics, as I said earlier, that, that knows how to view individual frames, but also, um, a consistent LLM as a judge, right? Where we also use human annotation to, to calibrate the, the LLM as a judge.
- 5:01
So for every report that we generate with that harness, uh, we're able to have humans annotate it and basically feed that feedback back in-into the, the, um, uh, uh, the LLM as a judge prompt to make sure that it's, it's aligned with what I think or what the annotator thought is good and, and, and we use it
- 5:20
to, to score the videos. The problem w-with, uh, uh, this approach, it's very slow, it's very expensive, and especially when we, we, we want to, to, to bring it, uh, for our users to be able to, uh, uh, generate a lot of video because creation is a very hard process.
- 5:35
And, um, uh, so the problem-- as I said, the problem is when, when... So this is a, a slow process, and we need to bring it as close to the users as possible and a-also earlier into the process.
- 5:46
The reason for that is if we take a look at all the metrics and there's, there are mistakes that we can find earlier than, uh, um, uh, uh, than later, then it's a lot cheaper to correct that, that particular mistake.
- 6:01
So for, um, uh, for example, right, on, um, o-on the left, we, we have, um, uh, a, a, um, a, uh, two... ... starting frames of different, different shots, right?
- 6:12
But it's easy to, to correct to, to v- view it at this point and see, did the character drift between frame one and, and frame two, because those frames will be used as starting frames to generate videos.
- 6:23
So if you can corre- catch the drift at this point and correct it, then, um, uh, then it's much cheaper to generate the video as a whole because we, we can correct it at a much, uh, uh, cheaper cost.
- 6:34
Um, and, and the same thing applies when we look at l- longer form video, right? When we see all these three, four, f- five-minute-long videos, they're usually a collection of a lot of shorter videos, and being able to catch a, a six-second generation that drifted and, and regenerated that, regenerate that, before we combine the whole video, uh,
- 6:54
will end up being a, a better result as a whole.
- 6:58
Um, [clears throat] and n- n- now, the, the, the other problem we're, we're trying to solve is some of these axes, right, only ex- exist across time, right? So for example, wh- when we look at the, the, the...
- 7:13
Right, we, we mentioned the s- story, right? So, um, does, does the story that we're trying to tell with that video, does it hold in that video? Does the video tell the exact story?
- 7:23
Does the pacing make sense, right? And we mentioned the, the, the, the sound. So as I said, right, the, the, the, you know, the underlying goal is to bring that evals closer to, to the, to the, the online generation because the, the sooner we are able to catch those mistakes, we were to, to...
- 7:41
The sooner we are able to catch the drift, right, then it's, um, uh, it's much easier, much, much cheaper to fix. Now,
- 7:51
s- so, so the, the, now, now the problem is that, as I said, this is a very slow process, so, um, uh, the, the solution is actually, actually to, to, to take all these committee of experts and distill it into one small model that is also, uh, very, very fast, but, um, uh, is, is able to give
- 8:09
us, um, a response that is not whether or not this video is slop or not, but why is it slop, right? Why, why is that video scored low versus the other?
- 8:19
Because, for example, it added an extra limb, because it didn't obey physics, because, um, um, the, the, the audio was, was, uh, out of sync. The, there... So, so the, the goal was, A, build it on top of a small VLM, and why is it a VLM?
- 8:33
VLM because we, uh, we, we needed the video, the, the, the model to be able to see the image. But also we need it to work fast, right, because we brought it closer to the g- uh, uh, um, generation, where in fact it takes about, uh, with the model we have trained, it takes about, um, three seconds
- 8:49
to score a three, uh, uh, a 15-second video. [clears throat]
- 8:55
Now, the, um... We, we also, um, tested a bigger model, and the results were better, but it was significantly slower. And, and, um, the, the decision was to go with the, the, the, the smaller model because the, um, the a- added value from the bigger model w- um, didn't justify the, the, the, the, the slowiness.
- 9:17
The other very interesting realization we came to is don't score compare. What does that mean? For example, if I'll ask a- any person in this room to look at a part- particular video and, and rank it from one to 10 on storytelling, right?
- 9:32
I'm pretty sure that, you know what will be a six for you, will be a five for you, will be a four for you and, and, and an eight for you, right?
- 9:38
But i- if I'll show you two videos, and I'll ask you which one of them is telling a better story, the grand majority will probably agree that B is telling a better story than A, right?
- 9:47
And if you do it enough times, then it, i- uh, it's easy to, um, uh, generalize the model at, um, uh, towards detecting what's better, uh, uh, ver- versus, uh, not.
- 10:01
So [clears throat] we trained on pairs, right? Um, uh, A versus B, as opposed to one through 10. Now, we manufactured badness. So l- l- luckily, the internet is full of very high quality videos, and it's very, very easy to get good videos, and it was very fun to create bad videos, A, by either corrupting good videos or by,
- 10:25
you know, just generating random slop. [clears throat] Um, now we shipped V1, and it was so wrong. It, it was wrong, but it was wrong in a very confident way. So, uh, uh, for e- for e- for example, the, the, the frame you see here is from, from a video that the model scored 9.2 on the camera work, and
- 10:47
the camera didn't move. For, for four seconds it was like a still image of the same c- character, but, but, but the model was, was very happy with, with, you know, the cinematography.
- 10:58
Um, so the physics in, i- in some other videos, uh, which I'm not showing be- because of, uh, of t- time limitations, it says that the physics look great, but it, it set it on, on ghosts hovering and, and people flying, et cetera.
- 11:15
So the... I mean, so the, so then the question is like, why was it wrong? The, the reason it was wrong is because how we generated that data, right?
- 11:25
It, it, um, it scored the vibe as opposed to the, the, the axes. So it, it learned how to, how to detect, um, coher- coherent videos, and it learned how to detect the, the, the, the, the, the artificial artifacts.
- 11:38
Basically, the gloss of the video as opposed to, uh, whether or not the video actually, um, told the, the, the, the, the... Sorry. [clears throat] The videos actually told the, the, the story.
- 11:51
And, um, and so, so, so the, the solution was, uh, to fix the dataset. And so the way we, we fixed the, the, the, the, the, the dataset, we actually...
- 12:04
I, uh, um, I started pairing real footage versus, um, AI footage. Now, the risk with that, and that, that's the reason why I voted, I voted, I avoided doing it at first, is because I didn't, didn't wanna create an AI detector.
- 12:18
Right? Because i- if you start creating pairs of good is, is, is, is human-generated video and bad is, uh, AI, uh, uh, video, then, then, then there's a very big chance of, of the model o- over fitting and becoming an AI detector as opposed to a s- um, uh, uh, video quality, uh, detector.
- 12:34
So the, um, there are two things I did in order to avoid that. A, I made sure that the encoding is, is consistent across both sides o- of, of the equation, so, um, uh, there, there, there's, there's no, um, um, artificial artifacts for, for video A versus video B.
- 12:53
Uh, a- and I, I used the exact same, uh, method o- of annotating both videos. So, so both the axes... So all the axes in those videos were annotated in the same way.
- 13:04
Uh, and surprise, it w- it turned out pretty awesome. Um, and so, so, so now what, what, what we're able to do, especially when w- you're, you're looking at, at, at, at videos.
- 13:18
A, we, we changed from a very complex pipeline, right, to an
- 13:24
agentic workflow. The reason behind this is, A, the, the pipelines work great if you have a very, very unique use case. But one- once you put, put, uh, uh, uh, put it in front of users, they'll have a very, very distinct story that they want to tell with their own characters, with, with their own i- images and
- 13:40
their own voice, and that's when it starts to drift. Um, by, by providing the agents with tools to validate the quality of, of the output i- it's creating, it's able to adapt to changes better, but it's also able to verify its own work and, and, and fix things as they go along.
- 13:57
So i- if you're gonna steal from, from these, uh, uh, uh, from this talk a few things. One, go relative, not absolute, right? As, as I explained earlier, the, the value of, of, of comparing video A versus, uh, uh, v- video B will always give you a better result going forward.
- 14:16
Uh, B, score the real axis that you care about. So if you, you care about s- um, uh, storytelling, if you care about pacing, if you care about, uh, physics, ha- score those axes and don't expect them to m- m- uh, miraculously appear.
- 14:31
And put eval inside the generation loop, right? Especially if your goal is, is, is, is to have a higher quality of generation. Get the, uh, the eval as close to the generation loo- uh, loop as possible.
- 14:47
E- event- eventually evaluate it as a story. Videos are stories. Videos are just another way for us to tell s- stories t- to others. And th- thank, thank you very much. [clapping]
- 15:05
All right. Any questions? Okay, down here. [laughs] Awesome. All right, I got two down here. Here you go.
- 15:16
Hi. Uh, how do you eval sound? Sound and, uh, video matching. I, I, I'm sorry. Can you repeat? How do you eval sound? Sound as effects- Oh, um- ...
- 15:25
and matching with the video? Oh, yeah. That's a fantastic question. So, um, um, so, so sound is actually a, a combination of a few things. One, I'm, I'm using, um, uh, uh, Atmos to, um, uh, to, to make sure that, that, that the, the, um, um, the, the sound quality i- is, is high enough and is, uh,
- 15:46
un- understandable. B, the model will, will, will, will learn to, learn to identify key frames, right? And, and, especially because when I feed something into the model, it's... I, it can be just the video or it can be the video plus the, the prompt that generated that video.
- 16:03
So for example, if the prompt will say, uh, uh, "The door slammed," right? It will look for a door being slammed and w- w- and, and will, will, will match the sound at th- that same frame.
- 16:18
Uh, uh, uh, did that answer your question? How does the model recognize sound? Uh, y- so it's, it, it's both by, by using, um, uh, um, uh, a- Atmos, but um, um...
- 16:30
And, and also to, to correlate the, the, the... So for example, when it's looking at the frames, right, it's, it, it's making sure that, that, um, uh, for example, the door being slammed at frame six, frame six has a specific timestamp.
- 16:43
So it's, it, it's looking for that, that, um, uh, uh, spike in the sound at that timestamp. It, it doesn't know that it is that sound, but it's looking for a specific, uh, spike of sound a- at that timestamp.
- 16:57
What about lip syncing? Lip syncing is an unsolved problem yet. [laughs]
- 17:04
One more question. We're trying, though. Can you repeat the question? Yeah. Go for it. Lip syncing. Yeah. So the, um, uh, uh, uh, the, the question was w- what about lip syncing?
- 17:14
For me-
- 17:15
Oh, that wasn't me, but, um-
- 17:16
No, no. Oh, I'm sorry
- 17:16
... I guess the lip syncing answer would be interesting, before I ask my question.
- 17:21
Uh, yeah. Um, um, a- as I said, it is an unsolved, um, uh, problem still. We're still, um, uh, work- we're working through it. Es- uh, especially for us, w- you know, some, some, some of the characters that we're trying to do are talking head are humans, right?
- 17:35
Which, you know, um, uh, we can look at the different techniques we're trying to identify the, the lips, but some of them are just talking, you know, um, uh, ta- talking animations that have no real correlation between, you know, the, the movement of, of, of the mouth and, and, and, and speech.
- 17:50
So unfortunately, I don't have a solution for that yet.
- 17:54
So I'm, uh, so I'm curious about, for example, if you wanted to further, um, enrich the dataset with, uh, human evaluation.
- 18:02
Yes.
- 18:02
Um, the question of, of taste in what is good, uh, because I think there is a big question mark about is that- Is that gonna remain the domain of humans?
- 18:13
But I've also seen people say that, well, most humans are really... They have terrible taste anyway in- [laughs] ... in videos and games and books.
- 18:20
Fair.
- 18:20
Um, so how would you construct and align sort of like any human, um, judges?
- 18:28
Yeah. So, um, so, so, so this is actually solved a- at first at the, the, at, at the Judge Judy part, where, uh, e- every report it'll generate, a human can go and annotate it, and we actually...
- 18:40
We, uh, uh, um, we do that. We, we periodically have sessions where everyone spends 10 to 15 minutes just, um, uh, annotating videos. And, and that usually happens on mu- multiple axes.
- 18:53
I, I won't ask everyone to annotate the same video on, on 10 different things. It'll, it'll, it'll be random. And, uh, I use, um, uh, the data to, to, to calibrate the, the, uh, AI judges and, and the results from that is actually being served, uh, as a data set for, for training for the next, uh, version
- 19:09
of, of that model. So it's a process that does take a little bit of time, and, and hopefully... And, and it does evolve over time, uh, but it's not immediate be- uh, you know, because also taste is very subjective and, and things that are great for me, you know, some, um, uh, that I think are fantastic, uh,
- 19:27
some people they come and say, "Ah, are you sure they're great?" Because, [laughs] you know.
- 19:31
So yeah, it's, it's, it's a process and, and, and I use the human, uh, uh, feedback to calibrate the models all the time.
- 19:39
Um, how did you land on the Qwen Small VLM? Did you try any others? Uh, I did. Um, so, so the intent I had was to, A, you know, find a small enough, uh, model.
- 19:51
Uh, the reason I went with Qwen is because we also had a very good experience, um, uh, with post, uh, uh, training Qwen on, on, on other use cases.
- 19:59
So it, um... I, I mean, yes, I could have... I, I did try a few others, but it just, you know, everything was just there and it was good enough.
- 20:09
So my question is about scale.
- 20:10
Yeah.
- 20:11
So obviously Character.ai produces thousands, millions, a bajillion videos.
- 20:15
Yeah.
- 20:15
At what scale does this become reasonable for my domain that is not Character.ai? So my domain has hundreds, maybe a thousand videos.
- 20:22
Mm-hmm. S- sure. Um, so if you're happy with, with the cohort o- of experts and, a- and, and it- and you don't need... Right. So I'll rephrase that. Um, uh, the, the scale is both for speed, right, as well as, as, as capacity because I, I can serve this model as one instance on one GPU, or I
- 20:42
can serve it as, you know, a, a, a, a 100 instances, right? So, so that's, that determines my, my scale. Um, the reason I chose to go towards a model is because I wanted to, to speed up the, the, the creation process, right?
- 20:55
Um, it would work, would've worked just as well if I didn't have this particular model. I would've used like the cohort of experts, right, from, uh, uh, uh, metrics that, um, um, uh, that are available both on CPU and GPU, as well as, um, uh, frontier models, right?
- 21:12
So it's, i- it, it was a balance out as, you know, A, how long did it take me to, to, to train this model and to curate the d- data set and, and get it to a working set, right?
- 21:22
Um, uh, and, and how much does it cost to serve it, uh, uh, versus how much it would've cost me to do this, A, slower. Now, potentially it is better, right?
- 21:31
I, I mean, like I assume that if you're gonna use the, um, uh, Fable, which came back today, right, it will probably give you a be- a better result.
- 21:38
But at what cost, right? If you do it for one or two, that's probably fine. If, if you do it for, um, uh, uh, thousands or tens of thousands per day, it adds up.
- 21:48
So it, it's, it, it's a matter of your, um, uh, your, your economics.
- 21:54
Cool. Right over here.
- 21:56
Yeah.
- 21:56
Um, on your right. There you go. Uh, last question.
- 22:01
It, it, it's very bright. I'm sorry. [laughs]
- 22:02
No worries, no worries. Um, my question is, I looked a bit at the repo. You guys don't export OTel traces of the LLMs judges yet.
- 22:11
Correct.
- 22:12
Is that something... Are you open to that-
- 22:14
Yes
- 22:14
... so you can connect to other, um, platforms?
- 22:17
Sure. So, so the, the, the, the, the repo itself, it, it, it's a harness, and you can connect any, any, um, uh, uh, agents or any LLMs, um, uh, uh, you want.
- 22:29
We actually have an internal version of this, uh, which is running it as a service, right, w- with a, with an agentic harness on top of it that has all the, the, the, the, the metrics w- uh, we care about.
- 22:40
But I, I, I, I do accept your feature request, and I'll be adding, uh, O- OTel, uh, telemetry t- to the, the harness. Awesome. Thank you very much. A, a warm welcome or, uh, round of applause for Maor.
- 22:53
Thank you. Thank you all. Thanks. [outro music]