Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta

A video can begin with the subject a viewer expected, then slip in an advertisement or political message for a small part of its runtime. Finding that insertion requires understanding where the content changes, down to clips and frames. In this recording, Aditya Gautam of Meta pairs that problem with another: finding copied or transformed videos and identifying their source.

The operating conditions make both tasks difficult. The data spans 100 million plus videos, includes viral and adversarial content, and carries multilingual text inside images and video. Uploads change as new AI tools appear, so the distribution moves from month to month. There is also no clear ground truth for the problems being solved; the system must work with judgments whose meaning depends on the content and the task.

Comparing modalities is the simpler starting point: represent images, video, audio, or text with embeddings and compare their cosine similarity. The harder task here is detecting a change within a video over time. A representation of the whole video does not by itself locate the small interval where an unexpected message appears. That calls for granular visual understanding and temporal analysis.

Originality adds a different comparison: the uploaded video against other content in the corpus. AI tools make duplication and transformation easier, which complicates attribution and credit. Repeated copies also produce user fatigue. Detecting similarity is therefore useful both for investigating a video's source and for understanding why viewers keep encountering versions of the same content.

The architecture divides content understanding, retrieval, and reasoning because each needs specialized work. That division has a condition: if one agent or one language model can solve the problem, there is no need to add a multi-agent system. Here, a reviewer receives the video or clip ID and acts as the central orchestrator, asking a perceiver to gather the visual evidence.

The perceiver fetches the video from the database and divides it into smaller parts using semantic embeddings and temporal changes. It does not sample at a fixed frame rate. Instead, it finds changes and compresses similar frames into one or two representative frames. The segmentation algorithm is outside the recording's scope, but its purpose is clear: reduce repeated visual work while retaining the places where the video changes.

Each clip becomes structured evidence: embeddings, tags, OCR, a natural-language description, and timestamps or frame ranges. The perceiver passes that information to the reviewer as a JSON object. This separates the visual extraction task from the temporal judgment: the reviewer can compare what happened in successive intervals using descriptions and metadata tied to their locations.

The concrete example is a sports video through the first six seconds, followed by political content from six to 6.5 seconds. The observable change is only half a second long. Once the perceiver represents those intervals separately, the reviewer can compare their topics, locate the transition, and consider whether it is a misalignment or a larger issue. The topic change becomes evidence for a judgment rather than an automatic verdict.

How does a fleeting insertion become something the reviewer can inspect? The diagram follows the sports example from its two time intervals into clip metadata and then a temporal comparison. The important relationship is between content and location: retaining both lets the reviewer identify the half-second change instead of treating the video as one undifferentiated subject.

The reviewer then asks the retriever for similar clips, supplying the video and metadata it has already deduplicated and post-processed. Retrieval extends the investigation beyond the current timeline: the system can compare the upload with other clips, entities, topics, and authors in the corpus.

Different signals need different search structures:

• Topics → inverted index. A topic points to the videos associated with it, giving retrieval a direct route from a named subject to matching content.
• Embeddings → vector database. Clip and video representations support similarity search across the corpus.
• Entities → graph database. Extracted entities provide another way to retrieve related content alongside topics and embeddings.

Using these stores together gives online inference several routes to relevant candidates and is intended to improve recall.

Much of this preparation happens offline. The perception machinery can run as a library in large-scale batch analysis; a Ray cluster is one example given. The resulting metadata is indexed, and periodic clustering assigns embedding and cluster IDs to clips and whole videos. Online retrieval then uses the query's clip ID and metadata to search those prepared stores, rather than rebuilding the corpus organization for every request.

Similarity produces candidates, which still need sorting. The retriever finds similar clips and authors, reranks them, and can use traditional classifier scores to identify spam or low-quality candidates. It returns the leading candidates to the reviewer. The reviewer combines them with the original content signals and can ask for more retrieval if the available evidence is insufficient. This supplies comparisons for the originality task; the recording does not specify a rule that conclusively selects the original creator from those matches.

The reviewer has two kinds of content evidence by this point: temporal signals from the video itself and comparisons with similar clips and authors. It also receives real-time interaction signals—reports, likes, dislikes, comments, and comment sentiment. Those signals capture responses that offline analysis and visual models can miss.

The reason to incorporate them is temporal as well as semantic. A change in sentiment or response may matter while the video is circulating, even though its pixels have not changed. Tools let the reviewer consider that evolving response alongside what the video contains. User interaction becomes an additional input to understanding the video.

All three agents use specialized small vision-language models, or VLMs. The motivation is the amount of visual work: processing billions of frames needs compression and economical serving. Model preparation therefore includes pretraining, fine-tuning, distillation, and quantization, with each model aimed at its particular task.

Pretraining addresses a mismatch in the visual data. Gautam contrasts the clean web data used by available foundational models with messy, in-house, user-generated content created for this workflow. Training on those image tokens and language adapts the vision component to the actual input distribution. That work is expensive, so the decision depends on whether it produces a useful improvement over adapting an existing encoder.

Instruction fine-tuning teaches the task and the form of the answer. The training data includes policies, guidelines, content signals, and desired outputs in a JSON schema, including misalignment or duplication scores and reviewer reasoning. In the model architecture, a vision encoder processes the clip, a projector bridges its representation to the language model, and the language model produces the instructed output.

The context gives each model its role, policy, available tools, and relevant metadata in a brief form. In-house labels define the issues and the output a human reviewer should receive. These two training stages solve different problems: visual pretraining improves understanding of the input, while instruction tuning teaches how to use the supplied context and express the result in a form the pipeline can process.

The next stage is a DPO improvement loop built around production examples. A subsample of inference results passes through an in-house LLM judge trained on human-labeled data, followed by a human review queue. The recording uses 95% as an illustrative performance threshold; it does not report an achieved result or define a benchmark metric for that number.

A poor result must be traced through the system before it becomes a training target. Human reviewers inspect agent traces, model calls, and tool interactions to find whether the failure came from reasoning, a wrong tool call, or retrieval that did not work. The correction may belong in the model or in the surrounding harness. A final wrong answer alone does not identify which component needs changing.

Once the needed improvement is understood, positive and negative examples support retraining. Daily production sampling keeps the loop supplied with fresh data and helps reveal drift. It also checks how the LLM judge is doing. The evaluator is part of the system that must keep learning, rather than a permanently reliable reference outside it.

Where does an incorrect production result turn into an improvement? The loop below places trace diagnosis between evaluation and retraining. That step matters because it separates a model's mistaken judgment from a tool or retrieval failure, allowing the correction to reach the component that caused it.

The model only needs to solve this internal domain problem. Gautam's useful test is blunt: “I really don't care if the model can solve a coding problem.” Frontier-sized VLMs carry inference cost and complexity that this workload cannot afford. Off-policy and on-policy knowledge distillation, followed by experiments with quantization, produce smaller candidates for deployment.

The decision comes from a table of model sizes and compression choices: which candidates preserve acceptable domain performance, and which save compute and inference cost? The recording does not provide the candidate sizes or measured savings. Its decision rule is to choose compression against task quality and production resources together, rather than assume that the smallest model is sufficient.

Evaluation begins with precision, recall, and F1 for task success, then opens up the pipeline:

• Retrieval quality and latency. Check how well retrieval finds relevant content and how long it takes, rather than attributing every final error to the reviewer.
• Reasoning quality and budget. Inspect whether the reviewer is overthinking. Reduce its budget when less work suffices; consider increasing planning or reasoning effort for more complex cases.
• Failures by component. Track edge cases and error rates at the nodes and hooks where tool calls, retrieval, or model behavior fail.
• System efficiency. Measure token cost and latency for each agent and for the combined system.

These measurements connect a final judgment to both its cause and its cost.

The judge needs its own evaluation against the human queue. User-generated content keeps drifting, so a judge that worked on an earlier distribution may need retraining on new labels. Otherwise, the improvement loop risks relying on an evaluator that has itself fallen behind the data.

The final optimizations reduce how much content needs expensive processing in the first place. They operate at different levels:

• Similar frames. Spatial-temporal reduction compresses repeated frames, reducing visual work within a video.
• Repeated content. Caching lets highly similar viral content reuse a decision and skip the multi-agent pipeline.
• Candidate videos. Metadata pruning narrows the set sent for review. Topics and creator history can inform the choice, including authenticity scores, video quality, and engagement.

The first reduces the input, the second avoids repeated inference, and the third decides which videos enter the system.

Those shortcuts make selection consequential. The sports example requires retaining a brief change; caching requires deciding when similarity is high enough to reuse a judgment; metadata pruning uses a creator's record to skip processing. The recording describes these mechanisms without giving their decision thresholds or error tradeoffs, so they are design choices to evaluate alongside the pipeline they bypass.

The closing advice keeps decomposition conditional and evaluation central. Divide the problem when specialized work requires it, adapt optimization to cost feasibility, and keep production monitoring and qualitative improvement running. On changing, adversarial video data, maintaining the system means continuing to inspect what it misses, what its tools do, and how its judgments compare with human review.