# Modality Misalignment and Originality Attribution in Short-Form Video — Aditya Gautam, Meta Aditya Gautam, Meta AI Engineer World's Fair 2026 [Talk page](https://ai.engineer/talks/jNE8No-wvok-modality-misalignment-originality-attribution) · [Video](https://www.youtube.com/watch?v=jNE8No-wvok) ## Modality Misalignment and Originality Attribution in Short-Form Video Aditya Gautam of Meta explains how specialized vision-language models divide perception, retrieval, and review to catch brief content changes and investigate copied videos—and how training, evaluation, and selective processing make that architecture practical at scale. ### A brief change can alter what a video is doing [0:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=12s) A video can begin with the subject a viewer expected, then slip in an advertisement or political message for a small part of its runtime. Finding that insertion requires understanding where the content changes, down to clips and frames. In this recording, Aditya Gautam of Meta pairs that problem with another: finding copied or transformed videos and identifying their source. ![Source frame: A brief change can alter what a video is doing](https://ai.engineer/api/talk-essay-media/f170a87052542c314412b685ae105f7cd200264cc4ce9001fd14539fb6b1709e) Source frame: A brief change can alter what a video is doing The operating conditions make both tasks difficult. The data spans 100 million plus videos, includes viral and adversarial content, and carries multilingual text inside images and video. Uploads change as new AI tools appear, so the distribution moves from month to month. There is also no clear ground truth for the problems being solved; the system must work with judgments whose meaning depends on the content and the task. Comparing modalities is the simpler starting point: represent images, video, audio, or text with embeddings and compare their cosine similarity. The harder task here is detecting a change within a video over time. A representation of the whole video does not by itself locate the small interval where an unexpected message appears. That calls for granular visual understanding and temporal analysis. Originality adds a different comparison: the uploaded video against other content in the corpus. AI tools make duplication and transformation easier, which complicates attribution and credit. Repeated copies also produce user fatigue. Detecting similarity is therefore useful both for investigating a video's source and for understanding why viewers keep encountering versions of the same content. ### Separate seeing the video from judging its timeline [3:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=229s) The architecture divides content understanding, retrieval, and reasoning because each needs specialized work. That division has a condition: if one agent or one language model can solve the problem, there is no need to add a multi-agent system. Here, a reviewer receives the video or clip ID and acts as the central orchestrator, asking a perceiver to gather the visual evidence. ![Source frame: Separate seeing the video from judging its timeline](https://ai.engineer/api/talk-essay-media/5650e912a74ba960257623cc6c9a4a06eafbdf7d79ebbd5393633695fb39191a) Source frame: Separate seeing the video from judging its timeline The perceiver fetches the video from the database and divides it into smaller parts using semantic embeddings and temporal changes. It does not sample at a fixed frame rate. Instead, it finds changes and compresses similar frames into one or two representative frames. The segmentation algorithm is outside the recording's scope, but its purpose is clear: reduce repeated visual work while retaining the places where the video changes. Each clip becomes structured evidence: embeddings, tags, OCR, a natural-language description, and timestamps or frame ranges. The perceiver passes that information to the reviewer as a JSON object. This separates the visual extraction task from the temporal judgment: the reviewer can compare what happened in successive intervals using descriptions and metadata tied to their locations. The concrete example is a sports video through the first six seconds, followed by political content from six to 6.5 seconds. The observable change is only half a second long. Once the perceiver represents those intervals separately, the reviewer can compare their topics, locate the transition, and consider whether it is a misalignment or a larger issue. The topic change becomes evidence for a judgment rather than an automatic verdict. How does a fleeting insertion become something the reviewer can inspect? The diagram follows the sports example from its two time intervals into clip metadata and then a temporal comparison. The important relationship is between content and location: retaining both lets the reviewer identify the half-second change instead of treating the video as one undifferentiated subject. Preserving the half-second change Time-linked clip evidence lets the reviewer compare the political interval with the preceding sports content. - First six seconds: sports — The initial interval establishes the video's preceding topic. - 6–6.5 seconds: political content — A brief interval changes the subject. - Perceiver: clip-level JSON — Embeddings, tags, OCR, descriptions, and time or frame ranges preserve what happened and where. - Reviewer: compare intervals — Locate the anomaly and assess whether it is misalignment or another issue. - First six seconds: sports → 6–6.5 seconds: political content (Topic changes at six seconds) - First six seconds: sports → Perceiver: clip-level JSON (Describe and locate the interval) - 6–6.5 seconds: political content → Perceiver: clip-level JSON (Describe and locate the changed interval) - Perceiver: clip-level JSON → Reviewer: compare intervals (Pass structured temporal signals) ### Search the corpus through topics, embeddings, and entities [6:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=390s) The reviewer then asks the retriever for similar clips, supplying the video and metadata it has already deduplicated and post-processed. Retrieval extends the investigation beyond the current timeline: the system can compare the upload with other clips, entities, topics, and authors in the corpus. ![Source frame: Search the corpus through topics, embeddings, and entities](https://ai.engineer/api/talk-essay-media/8161e09cf3f5a689228f1501af154d8adcaed6b7c088a71f2f4812e00edb9793) Source frame: Search the corpus through topics, embeddings, and entities Different signals need different search structures: - **Topics → inverted index.** A topic points to the videos associated with it, giving retrieval a direct route from a named subject to matching content. - **Embeddings → vector database.** Clip and video representations support similarity search across the corpus. - **Entities → graph database.** Extracted entities provide another way to retrieve related content alongside topics and embeddings. Using these stores together gives online inference several routes to relevant candidates and is intended to improve recall. Much of this preparation happens offline. The perception machinery can run as a library in large-scale batch analysis; a Ray cluster is one example given. The resulting metadata is indexed, and periodic clustering assigns embedding and cluster IDs to clips and whole videos. Online retrieval then uses the query's clip ID and metadata to search those prepared stores, rather than rebuilding the corpus organization for every request. Similarity produces candidates, which still need sorting. The retriever finds similar clips and authors, reranks them, and can use traditional classifier scores to identify spam or low-quality candidates. It returns the leading candidates to the reviewer. The reviewer combines them with the original content signals and can ask for more retrieval if the available evidence is insufficient. This supplies comparisons for the originality task; the recording does not specify a rule that conclusively selects the original creator from those matches. ### Add what viewers are experiencing now [9:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=590s) The reviewer has two kinds of content evidence by this point: temporal signals from the video itself and comparisons with similar clips and authors. It also receives real-time interaction signals—reports, likes, dislikes, comments, and comment sentiment. Those signals capture responses that offline analysis and visual models can miss. ![Source frame: Add what viewers are experiencing now](https://ai.engineer/api/talk-essay-media/75a674effc807f773ab5fbb4421ebaecfb97d34fde5ee4ae68f789c59561f8fe) Source frame: Add what viewers are experiencing now The reason to incorporate them is temporal as well as semantic. A change in sentiment or response may matter while the video is circulating, even though its pixels have not changed. Tools let the reviewer consider that evolving response alongside what the video contains. User interaction becomes an additional input to understanding the video. ### Train small VLMs for the data and the output [10:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=638s) All three agents use specialized small vision-language models, or VLMs. The motivation is the amount of visual work: processing billions of frames needs compression and economical serving. Model preparation therefore includes pretraining, fine-tuning, distillation, and quantization, with each model aimed at its particular task. ![Source frame: Train small VLMs for the data and the output](https://ai.engineer/api/talk-essay-media/5833f32de5020b8590015c48c9b9a7f0e39e1f69da81d38956a1ff96619f704a) Source frame: Train small VLMs for the data and the output Pretraining addresses a mismatch in the visual data. Gautam contrasts the clean web data used by available foundational models with messy, in-house, user-generated content created for this workflow. Training on those image tokens and language adapts the vision component to the actual input distribution. That work is expensive, so the decision depends on whether it produces a useful improvement over adapting an existing encoder. Instruction fine-tuning teaches the task and the form of the answer. The training data includes policies, guidelines, content signals, and desired outputs in a JSON schema, including misalignment or duplication scores and reviewer reasoning. In the model architecture, a vision encoder processes the clip, a projector bridges its representation to the language model, and the language model produces the instructed output. The context gives each model its role, policy, available tools, and relevant metadata in a brief form. In-house labels define the issues and the output a human reviewer should receive. These two training stages solve different problems: visual pretraining improves understanding of the input, while instruction tuning teaches how to use the supplied context and express the result in a form the pipeline can process. ### Trace production failures before retraining [13:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=820s) The next stage is a DPO improvement loop built around production examples. A subsample of inference results passes through an in-house LLM judge trained on human-labeled data, followed by a human review queue. The recording uses 95% as an illustrative performance threshold; it does not report an achieved result or define a benchmark metric for that number. ![Source frame: Trace production failures before retraining](https://ai.engineer/api/talk-essay-media/da9db04b7a5e21853e50b5ea1e8e7d370c16af8468a5aebb2a1c470e7a12ba5b) Source frame: Trace production failures before retraining A poor result must be traced through the system before it becomes a training target. Human reviewers inspect agent traces, model calls, and tool interactions to find whether the failure came from reasoning, a wrong tool call, or retrieval that did not work. The correction may belong in the model or in the surrounding harness. A final wrong answer alone does not identify which component needs changing. Once the needed improvement is understood, positive and negative examples support retraining. Daily production sampling keeps the loop supplied with fresh data and helps reveal drift. It also checks how the LLM judge is doing. The evaluator is part of the system that must keep learning, rather than a permanently reliable reference outside it. Where does an incorrect production result turn into an improvement? The loop below places trace diagnosis between evaluation and retraining. That step matters because it separates a model's mistaken judgment from a tool or retrieval failure, allowing the correction to reach the component that caused it. Production examples feed targeted improvement Daily samples support evaluation, trace diagnosis, and new training examples; human review also checks the judge. - Production inference samples — Subsample real results daily to track performance and drift. - LLM judge and human review — Evaluate system results and monitor the judge against human review. - Trace the failure — Inspect reasoning, tool calls, and retrieval to locate model or harness problems. - Correct and retrain — Use identified improvements and positive/negative examples to update the model. - Production inference samples → LLM judge and human review (Evaluate sampled results) - LLM judge and human review → Trace the failure (Investigate poor results) - Trace the failure → Correct and retrain (Identify the needed correction) - Correct and retrain → Production inference samples (Repeat with fresh production data) ### Choose model size against the whole system's cost [16:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=965s) The model only needs to solve this internal domain problem. Gautam's useful test is blunt: “I really don't care if the model can solve a coding problem.” Frontier-sized VLMs carry inference cost and complexity that this workload cannot afford. Off-policy and on-policy knowledge distillation, followed by experiments with quantization, produce smaller candidates for deployment. ![Source frame: Choose model size against the whole system's cost](https://ai.engineer/api/talk-essay-media/236a244d22cd3c1ce111ad58dada029879bd08e3bdf570741e659b64c0bd198b) Source frame: Choose model size against the whole system's cost The decision comes from a table of model sizes and compression choices: which candidates preserve acceptable domain performance, and which save compute and inference cost? The recording does not provide the candidate sizes or measured savings. Its decision rule is to choose compression against task quality and production resources together, rather than assume that the smallest model is sufficient. Evaluation begins with precision, recall, and F1 for task success, then opens up the pipeline: - **Retrieval quality and latency.** Check how well retrieval finds relevant content and how long it takes, rather than attributing every final error to the reviewer. - **Reasoning quality and budget.** Inspect whether the reviewer is overthinking. Reduce its budget when less work suffices; consider increasing planning or reasoning effort for more complex cases. - **Failures by component.** Track edge cases and error rates at the nodes and hooks where tool calls, retrieval, or model behavior fail. - **System efficiency.** Measure token cost and latency for each agent and for the combined system. These measurements connect a final judgment to both its cause and its cost. The judge needs its own evaluation against the human queue. User-generated content keeps drifting, so a judge that worked on an earlier distribution may need retraining on new labels. Otherwise, the improvement loop risks relying on an evaluator that has itself fallen behind the data. ### Save work before invoking the full pipeline [19:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=1170s) The final optimizations reduce how much content needs expensive processing in the first place. They operate at different levels: - **Similar frames.** Spatial-temporal reduction compresses repeated frames, reducing visual work within a video. - **Repeated content.** Caching lets highly similar viral content reuse a decision and skip the multi-agent pipeline. - **Candidate videos.** Metadata pruning narrows the set sent for review. Topics and creator history can inform the choice, including authenticity scores, video quality, and engagement. The first reduces the input, the second avoids repeated inference, and the third decides which videos enter the system. ![Source frame: Save work before invoking the full pipeline](https://ai.engineer/api/talk-essay-media/3b588ac2dcdb973def56092371d8a87dc332ae28be30ae73c8a961bf8205d2e2) Source frame: Save work before invoking the full pipeline Those shortcuts make selection consequential. The sports example requires retaining a brief change; caching requires deciding when similarity is high enough to reuse a judgment; metadata pruning uses a creator's record to skip processing. The recording describes these mechanisms without giving their decision thresholds or error tradeoffs, so they are design choices to evaluate alongside the pipeline they bypass. The closing advice keeps decomposition conditional and evaluation central. Divide the problem when specialized work requires it, adapt optimization to cost feasibility, and keep production monitoring and qualitative improvement running. On changing, adversarial video data, maintaining the system means continuing to inspect what it misses, what its tools do, and how its judgments compare with human review. ## Key takeaways - Time-linked clip evidence makes a brief topic change inspectable: the perceiver extracts it, and the reviewer compares it with surrounding content. - Offline indexing and clustering prepare the corpus for online retrieval; topics, embeddings, and entities supply different routes to similar clips and authors. - Production failures need trace diagnosis before correction: reasoning, tool calls, and retrieval can each cause a poor final result. - Choose specialized model sizes and reasoning budgets against domain quality, latency, and cost, while checking the LLM judge against fresh human labels. - Frame compression, cached decisions, and metadata pruning save work at different stages; ongoing evaluation and monitoring keep those optimizations useful as the data changes. ## Slides ### Modality Misalignment & Originality Attribution in Short-Form Video Aditya Gautam introduces two short-form video problems and a multi-agent approach to understanding content at platform scale. [0:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=20s) [Title slide introducing a multi-agent approach to short-form video understanding, presented by Aditya Gautam.](https://assets.ai.engineer/sites/aiecode2025/sha256/982b2a07738429f366715e3e30b91a577ee7403a65a0ff9670d0f337e0d5ef29.webp) Modality Misalignment & Originality Attribution in Short-Form Video AI ENGINEER WORLD'S FAIR 2026 · VISION & OCR TRACK A multi-agent approach to multimodal content understanding at platform scale. Aditya Gautam Interface text: AIE_WorldsFair_AdityaGautam Google Slides: docs.google.com/presentation/d/[illegible]/edit?slide=id.p1#slide=id.p1 Desk 1 Jun 30 12:05 ### Outline Gautam outlines the problems, multi-agent system, specialized SLMs, evaluation and optimization. Optimization includes selecting a smaller set of videos for the workflow. [0:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=24s) [Outline lists short-form video characteristics, problem definition, multi-agent systems, specialized SLM, evaluation, optimization, and takeaways.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3106a3d3a9aef286b9d5d6b0b91d8f371ec060c4fb4e8116bc6e987c25139e6.webp) Outline 1 SFV Characteristics 2 Problem Definition 3 Multi-Agent System 4 Specialized SLM 5 Evaluation 6 Optimization 7 Takeaways Interface text: AIE_WorldsFair_AdityaGautam sdocs.google.com/presentation/d/[illegible]/edit?slide=id.p2#slide=id.p2 Desk 1 Jun 30 12:06 ### Multi-Agent System Architecture Gautam splits the complex task across agents for perception, retrieval and adjudication. He says a multi-agent system is unnecessary when one agent can solve the problem. [3:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=226s) [Section title introduces specialized agents that perceive, retrieve, and adjudicate.](https://assets.ai.engineer/sites/aiecode2025/sha256/4390b3b07002d18bf0551b57085e4168ba1ee4a3b2bf68cbceeb05b20154d416.webp) SECTION 01 Multi-Agent System Architecture Decompose the task into specialized agents — perceive, retrieve, adjudicate. Interface text: AIE_WorldsFair_AdityaGautam Docs.google.com/presentation/d/[illegible]/edit?slide=id.p6#slide=id.p6 Desk 1 Jun 30 12:09 ### Three-agent system architecture The Reviewer asks the Perceiver for temporal metadata and the Retriever for similar clips and evidence, then reasons toward an originality and misalignment verdict. [4:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=253s) [Diagram showing the Reviewer coordinating the Perceiver and Retriever agents to produce a verdict.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a5a3884982a3b3cfe7a3285a3124cd3d53bbbcb27fdf38addd7373caec859a0.webp) THE SYSTEM Three agents: Perceiver indexes · Retriever finds the source · Reviewer judges TOOLS · CALLED ANYTIME Author info Model scores Comments/dislikes Golden Dataset Video + clip id Reviewer Agent • Orchestrates; reasons with CoT • Validate request -> Ask Preceiver • Temporal clip-data analysis • Get retrieval data i.e. similar clips, similar authors, timestamp etc. • Reason -> Iterate -> verdict 1 · metadata 2 · retrieve ReAct loop Perceiver Agent • Perceives the clip -> Temporal decomposition • Whole-video embedding + per sub-clip data • Per sub-clip: embeddings, OCR, objects, topics, tags, NLP description etc • Tools : Image and video libraries. INSIDE THE PERCEIVER Video extraction Preprocessing VLM Retriever Agent • Adaptive DB-caller + re-ranker • Top-N similar clips + evidence • Creation timestamps → earliest = the original STORES Vector Graph DB Key–value Inverted index VERDICT: originality · misalignment· CoT · confidence Interface text: AIE_WorldsFair_AdityaGautam ​​​​docs.google.com/presentation/d/[illegible]/edit?slide=id.p7#slide=id.p7 Desk 1 Jun 30 12:09 ### Perceiver: extract granular signals The Perceiver splits a video into sub-clips and uses a fine-tuned VLM to extract queryable signals. Distillation and quantization help make the specialized model cheaper to serve at scale. [7:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=447s) [A clip is decomposed and processed by a vision-language model into per-sub-clip signals.](https://assets.ai.engineer/sites/aiecode2025/sha256/2945419cae005fd2ad64e7bbed7414c1704acb8bd44efd2ee0a80dd7d7ede6fe.webp) AGENT 01 · THE EYE Perceiver — Extract granular signals Turns a raw clip into a queryable, per-sub-clip index — across every modality. One clip → Temporal decomposition → Fine-tuned VLM PER SUB-CLIP OUTPUTS • Embeddings • OCR text • Topics • NL descriptions • Objects / persons • Hashtags THIS AGENT'S VLM — BUILT ONCE, SERVED CHEAP Pre-training → Fine-tuning → Distillation → Quantization → Cost-effective VLM Specialized VLM, trained on our in-house SFV corpus. Audio and text (user comments) are processed the same way — every modality, per sub-clip. Interface text: AIE_WorldsFair_AdityaGautam https://docs.google.com/presentation/d/[illegible]/edit?slide=id.p8#slide=id.p8 Desk 1 Jun 30 12:13 ### Retriever: indexer and context builder The Retriever indexes Perceiver signals across four stores and clusters similar content daily. At query time, it searches the stores and re-ranks candidates using ML scores before sending the top N to the Reviewer. [8:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=501s) [Diagram of offline indexing and online retrieval across several stores, followed by ranking candidates.](https://assets.ai.engineer/sites/aiecode2025/sha256/bc92c5de9e3eb872b3d65f59e82332a79ec3a4428d84cae22d349fa56a8038a4.webp) AGENT 02 · THE MEMORY Retriever — Indexer and Context builder OFFLINE · INDEX EVERYTHING Perceiver results - Whole video and clip signals Vector DB embeddings Inverted index topics · hashtags Key–value ids · scores Knowledge graph objects · persons + daily K-means clustering to group similar content - Find hot clusters ONLINE · ADAPTIVE RETRIEVAL & RANKING Query + params video id · task Search the stores candidates + raw scores Lightweight FT LLM adaptive retrieve + re-rank Top-N candidates semantic · author · topic · hashtag Traditional ML scores ads · scam · spam Top-N blends candidates across retrieval categories — then hands them to the Reviewer. Interface text: AIE_WorldsFair_AdityaGautam https://docs.google.com/presentation/d/[illegible]/edit?slide=id.p9#slide=id.p9 Desk 1 Jun 30 12:14 ### Reviewer: the reasoning judge The Reviewer weighs clip signals, similar videos, classifier scores, feedback and creator metadata to reach a verdict. When unsure, it can call the Retriever and other tools for more evidence. [9:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=583s) [The Reviewer combines agent outputs, scores, feedback, and creator metadata to produce decisions and confidence.](https://assets.ai.engineer/sites/aiecode2025/sha256/55766aafa49de7fc90f6151b13817ab25b0b4f5276a175c00970d1a3ff7c1917.webp) AGENT 03 · THE ADJUDICATOR Reviewer — the reasoning judge WHAT IT SEES Perceiver : temporal signals Retriever: Similar clips Classifier scores: Originality etc Comments + User feedback Creator Metadata Reasoning + CoT The central brain. Weighs every signal, reasons step by step, loops back for more. TAC — reasoning must match the verdict OUTPUT • Modality-mismatch decision • Originality / dedup verdict • Confidence + reasoning trace ReAct loop: calls the Retriever and tools for more evidence when unsure. Tech: a high-reasoning LLM, fine-tuned on our labelled dataset. Interface text: AIE_WorldsFair_AdityaGauta[illegible] docs.google.com/presentation/d/[illegible]/edit?slide=id.p10#slide=id.p10 Desk 1 Jun 30 12:15 ### Specialized SLM Each agent gets a specialized small model built through pre-training, instruction SFT and DPO post-training, then compressed for serving. [10:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=638s) [Section slide showing pre-training, instruction fine-tuning, and post-training with DPO.](https://assets.ai.engineer/sites/aiecode2025/sha256/4dc703fa1b78743e052d52026932d04482777a3f42cbd458391c9eb9b8369873.webp) SECTION 02 Specialized SLM Pre-training → Instruction SFT → Post-training · DPO One small model per agent, built in three stages — then compressed to serve. Interface text: AIE_WorldsFair_AdityaGautam Desk 1 Jun 30 12:16 ### Pre-training the vision adapter Pre-training combines web-scale data with noisy in-house SFV data. Frame patches and video descriptions shift the vision adapter toward the target distribution before task-specific fine-tuning. [10:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=654s) [Web-scale and in-house video data feed pre-training to adapt the vision model to the target data.](https://assets.ai.engineer/sites/aiecode2025/sha256/bb4d673f7fc4be5b025c65dd58539bb053d998a23eda01f7a6a1389fc87d85b7.webp) STAGE 1 · PRE-TRAINING Teach the vision adapter our data Before any task, align the model to how SFV actually looks — on frame patches plus their descriptions. Web-scale data broad, polished — a starting point In-house SFV data human-curated, noisy, real UGC Pre-train on frame image patches + video descriptions & details Vision adapter, re-weighted • Weights shift toward our distribution • Learns the underlying SFV data • Foundation for task-specific SFT Goal: understand the underlying data first — both web-scale and in-house — so the encoder isn't surprised by real SFV. Interface text: AIE_WorldsFair_AdityaGautam Google Slides: docs.google.com/presentation/d/[illegible]/edit?slide=id.p12#slide=id.p12 Desk 1 Jun 30 12:16 ### Instruction fine-tuning The vision encoder, projector and language model train together on labelled SFV tasks. The projector bridges vision and language; training uses a fixed corpus, while live signals arrive at inference. [11:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=709s) [Pipeline from video clips through a vision encoder, projector and language model to a structured output, with notes on offline training.](https://assets.ai.engineer/sites/aiecode2025/sha256/968cfe1283f902dde43b70a7cad0ba8953a5638688d128306885ad5c3311d520.webp) STAGE 2 · INSTRUCTION SFT Fine-tune on the task — encoder, projector & LLM together Videos -> Clips frames + text Vision encoder patches → tokens ViT · SIGLIP Projector align to LLM space MLP Language model fuse + reason DECODER LLM Output verdict/metadata MISMATCH/OCR All three trained together on labelled SFV tasks WHAT SFT TEACHES • Specific instructions set, policies and in-house data • How to use tools — Retriever, author info, web & DB • Emit the structured output — CoT · verdict · tags TRAINED OFFLINE • Fixed labelled SFV corpus — human + ML • Targets: chain-of-thought · verdict · tags • Live signals come later — only at inference Interface text: AIE_WorldsFair_AdityaGautam 
docs.google.com/presentation/d/[illegible]/edit?slide=id.p13#slide=id.p13 Desk 1 Jun 30 12:17 ### Engineered fine-tuning context Each training example combines role, policy, tools, grounding and metadata with a structured label. The same in-house labelled data trains the model and builds the LLM-as-a-judge. [12:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=757s) [A list of context components leads to an example structured label; a footer describes in-house labelled data.](https://assets.ai.engineer/sites/aiecode2025/sha256/4eb15ff4a79a68eb6bfa7b38587c04802aa391b95dcb876870577c6ed135f8fa.webp) STAGE 2 · THE FINE TUNING DATA Each example is engineered context, not just a prompt THE ENGINEERED CONTEXT ROLE modality-misalignment & unoriginality reviewer — right altitude POLICY what is a violation · what to allow · when to abstain TOOLS a minimal, unambiguous set — Retriever · Author info GROUNDING the SFV retrieved context [Perceiver & Retrieve agents] ADDITIONAL METADATA Common mistakes - brief rundown of global corpus OUTPUT SCHEMA a strict JSON contract (see label →) STRUCTURED LABEL { reasoning: “CoT trace…”, modality: “mismatch @ 0:14”, verdict: “unoriginal”, tags: [“dance”, “music”], source: “creator_8421”, confidence: high... } Strict schema → easy to parse, grade & compare. In-house human + ML-classified labelled data. The same set trains the model — and builds the LLM-as-a-judge. Interface text: AIE_WorldsFair_AdityaGautam https://docs.google.com/presentation/d/[illegible]/edit?slide=id.p14#slide=id.p14 Desk 1 Jun 30 12:18 ### DPO on production hard cases Production samples pass through an LLM judge and human review. Chosen and rejected pairs then train DPO to prefer verified verdicts over wrong or shortcut calls, feeding an updated model back into production. [13:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=820s) [A production feedback loop generates preference pairs for DPO, with chosen and rejected examples below.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ac897cf0d67df82427d4a03b7c87d993bb307160015e1aed0b9787bd7391e6d.webp) STAGE 3 · POST-TRAINING DPO on hard cases mined from production Production is sampled continuously; a human queue surfaces the hard, ambiguous and wrong calls — DPO learns the granular difference and updates the model. Production live verdicts at scale Sample + judge LLM-judge pre-filters Human review queue hard · ambiguous · disagreements Preference pairs chosen vs rejected DPO update ship the new model continuous loop — the model improves every cycle HOW DPO USES THE PAIRS CHOSEN human-verified verdict + faithful CoT REJECTED the model’s wrong / shortcut call DPO Prefer chosen over rejected Bradley–Terry classification loss — offline, no reward model, no rollouts. The model learns the granular difference between a right and a nearly-right call. = our TAC signal. Interface text: AIE_WorldsFair_AdityaGautam Google Slides address: docs.google.com/presentation/d/[illegible]/edit?slide=id.p15#slide=id.p15 Desk 1 Jun 30 12:19 ### Compressing a teacher into a specialist To make serving scalable, Gautam proposes distilling a ~500B–1T teacher into a 3–8B specialist and training with simulated 4-bit precision, focused on modality misalignment and unoriginality. [16:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=963s) [A large teacher model is distilled into a smaller student, with notes on distillation and quantization-aware training.](https://assets.ai.engineer/sites/aiecode2025/sha256/fba49f1f4d9d27ec9c5ed9547bfd458be5be14886c28671ac44e14b920886854.webp) THEN: COMPRESS TO SERVE A frontier-scale teacher → a lean 3–8B specialist Efficient serving: Knowledge distillation -> then quantization-aware training : Specialized small VLM ~500B-1T Teacher VLM distill 3–8B Distilled Quantized student DISTILLATION · OFF-POLICY + ON-POLICY • Off-policy: imitate teacher on fixed data → exposure bias • On-policy: student generates, teacher corrects its own errors QUANTIZATION-AWARE TRAINING • Train with simulated 4-bit in the loop — not post-hoc PTQ • The model adapts to low precision as it learns • Accuracy holds where post-training quantization would drop it Scope is the lever: We ONLY care about modality misalignment & unoriginality — no world knowledge, coding or chat. Interface text: AIE_WorldsFair_AdityaGautam ​​docs.google.com/presentation/d/[illegible]/edit?slide=id.p16#slide=id.p16 Desk 1 Jun 30 12:21 ### Six evaluation dimensions Gautam evaluates task success alongside retrieval, reasoning, robustness, system efficiency and judge-human agreement, examining both the final outcome and where errors occur in the system. [17:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=1045s) [Six cards cover task success, retrieval, reasoning, robustness, system efficiency and LLM-as-judge.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ea1b096361919cb73e60ded20c0598d96691f8deb8a3a0b900230ccca5f376b.webp) EVALUATION Six dimensions, not one score 1 Task success Precision · Recall · F1 2 Retrieval Recall & latency, per source 3 Reasoning CoT & planning quality (TAC) 4 Robustness Error rate and source 5 System efficiency Token cost · Tool use · Latency 6 LLM-as-judge Correlation with humans labelling pipeline No single number is enough. Task success leads, but reasoning, retrieval, robustness, cost and judge-human agreement are all first-class. Interface text: AIE_WorldsFair_AdityaGautam https://docs.google.com/presentation/d/[illegible]/edit?slide=id.p18#slide=id.p18 Desk 1 Jun 30 12:23 ### STORM spatiotemporal token reduction A Mamba encoder adds inter-frame motion before pooling and sampling reduce the tokens sent to the LLM. The slide reports that ~25% of tokens reach the LLM, with processing up to ~65% faster. [19:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=1169s) [A diagram pools many redundant tokens into fewer enriched tokens, alongside notes on motion encoding and efficiency.](https://assets.ai.engineer/sites/aiecode2025/sha256/dee4b9041b7230151f0e86dd25d9e409e9069314e3be26d6636cd5783c84cc1b.webp) OPTIMIZATION · PERCEIVER STORM — spatiotemporal token reduction Consecutive frames are highly redundant. A Mamba temporal encoder enriches tokens with motion, so the rest can be pruned before the LLM. MANY REDUNDANT TOKENS pool FEW ENRICHED TOKENS HOW & WINS • Mamba temporal encoder injects inter-frame motion • Reduce via temporal + spatial pooling & test-time sampling • ~25% of tokens reach the LLM · up to ~65% faster • Wins: lower cost, latency & GPU memory Only informative tokens reach the LLM — perception runs on every upload, so this is a first-order lever. Interface text: AIE_WorldsFair_AdityaGautam Dept 1 [illegible] Jun 30 12:24 ### Semantic caching The Retriever compares each new clip with cached content. A similarity score above 0.9 lets the Reviewer skip reasoning and tag it unoriginal; lower scores go through multi-agent reasoning and are stored. [19:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=1184s) [A similarity-scoring workflow routes high-scoring clips past reasoning and lower-scoring clips through multi-agent review.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c1cdc5b473df81592dbba289404b957749198ea4241ac264f203627fed9f9b4.webp) OPTIMIZATION · RETRIEVER Semantic caching — instant dedup of viral re-uploads The Retriever computes the similarity score; a high score lets the Reviewer skip reasoning and save budget. New clip Embed + look up cache Retriever scores similarity HIGH score (sim > 0.9) Reviewer skips reasoning · tag unoriginal LOWER score full multi-agent reasoning + store WINS Adaptive budget the Retriever's score sets how much the Reviewer reasons Cost & latency only new / ambiguous cases reach the Reviewer Freshness unoriginal content caught fast Interface text: AIE_WorldsFair_AdityaGautam docs.google.com/presentation/d/[illegible]/edit?slide=id.p21#slide=id.p21 Desk 1 Jun 30 12:25 ### Metadata pruning Indexes and filtering shrink millions of candidates to hundreds. Creators with clean track records can be whitelisted; every new creator’s video is scanned until authenticity and originality confidence is high. [19:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=1199s) [A funnel reduces millions of candidates to hundreds using indexes, clustering, deduplication and whitelists.](https://assets.ai.engineer/sites/aiecode2025/sha256/45c6af5248391f7d740d2a84c076c5fd820a95dd5428d9c4cbbb40f5955991bd.webp) OPTIMIZATION · UPSTREAM Metadata pruning — shrink the search space The cheapest model call is the one you never make. Cut the candidate pool before the agents run. Millions of candidates Topic / object / creator indexes Clustering + dedup + whitelists Hundreds to the agents HOW • Indexes on topics, sub-topics, objects, persons • Daily offline clustering & dedup • Creator-similarity embeddings flag repeat offenders • Whitelist historically safe topics (nature, gardening, school) CREATOR TRUST · PERIODIC OFFLINE • Whitelist trusted creators with a clean track record • New creator → scan every video until authenticity & originality confidence is high Reduces the search space from millions to hundreds before any LLM runs. Interface text: AIE_WorldsFair_AdityaGautam Desk 1 Jun 30 12:25 ### Thank you Aditya Gautam closes by thanking the audience. [21:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=1268s) [Closing slide reads “Small models. Big opinions!” and includes the speaker’s name and professional contact link.](https://assets.ai.engineer/sites/aiecode2025/sha256/4066128ff7c1be80150bb6f15ad1c1db980d0bcab24b439041832cebc885abfa.webp) THANK YOU Small models. Big opinions! Aditya Gautam in/aditya-gautam-68233a30 Interface text: AIE_WorldsFair_AdityaGautam Google Slides: docs.google.com/presentation/d/[illegible]/edit?slide=id.p24#slide=id.p24 Desk 1 Jun 30 12:26 ### AI Engineer World's Fair [21:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=1281s) [AI Engineer World's Fair title card on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/44363ba8e9d3c30a4abeb54ec8b6cbd929b7331fcc59591dcbb3bad84d24f766.webp) AI Engineer World’s Fair HTTPS://AI.ENGINEER ## Full transcript [0:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=1s) [music] [0:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=12s) Hello everyone. Uh I am Adita and we [0:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=15s) will be talking about uh two main [0:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=17s) problems that happen on the short form [0:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=19s) surfaces. We are going to talk about how [0:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=21s) do we deal with these things at a scale. [0:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=23s) Uh [0:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=25s) so to get started first thing is we will [0:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=27s) understand what are the characteristics [0:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=28s) of the data that we are trying to deal [0:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=30s) with what are the two main problems that [0:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=32s) we are actually working on uh and what [0:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=35s) are the multi- aent systems to solve [0:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=37s) those problems at a scale what are the [0:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=40s) different specialized small scale WM [0:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=42s) that we can uh build uh to solve those [0:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=45s) individual agentic problem and how do we [0:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=48s) build those agents what are the [0:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=49s) different ways to actually optimize it [0:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=51s) like to make it like scalable at a very [0:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=53s) big scale and then we will be looking [0:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=56s) into the evaluation in a very holistic [0:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=58s) 360 manner not just like a precision [1:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=60s) recall what is out there how do we [1:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=62s) understand the entire multi-agenting [1:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=64s) pipeline both from LLM level tools MCP [1:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=67s) and whatn not everything out there and [1:09](https://www.youtube.com/watch?v=jNE8No-wvok&t=69s) then we will be getting into the [1:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=70s) optimization techniques which this which [1:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=72s) are very vision specific and metadata [1:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=75s) specific which will allow us and give us [1:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=77s) some intelligence to tell that we really [1:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=80s) don't need to do intell like this [1:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=81s) intelligent like workflow So for all the [1:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=83s) videos we can figure out what are the [1:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=85s) small set of videos which should be the [1:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=87s) candidate for doing these things and we [1:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=89s) will conclude with the takeaways. [1:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=92s) So the real life data is very messy. It [1:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=95s) is we are talking about at a scale of [1:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=97s) 100 million plus and a lot more viral [1:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=99s) content. There's a lot of adversal [1:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=101s) content people are trying to gain the [1:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=103s) system. There are a lot of uh [1:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=104s) multilingual text on the screens and in [1:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=106s) their videos and images. And the uh data [1:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=110s) is very dynamic. it keeps on changing [1:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=112s) from like month to month different AI [1:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=113s) tools are coming everything out there so [1:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=115s) it's it's really really dynamic that [1:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=117s) means there is a lot of drift issues and [1:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=118s) other thing that comes with a video data [2:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=121s) and then we know that there is no clear [2:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=123s) ground truth for the problem that we're [2:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=125s) trying to solve so these are like [2:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=126s) existing problem which happens in a real [2:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=128s) world video data set we will be the [2:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=131s) first problem that we will be talking [2:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=133s) about is the modality misalignment where [2:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=135s) the first is intramodality which is a [2:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=138s) very solved problem you can have a clip [2:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=140s) model you can have a modality on uh uh [2:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=143s) images, videos, uh audio, text and [2:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=145s) figure out what exactly is the cosign [2:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=147s) similarity on those embedding and figure [2:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=148s) it out. So this is a much simpler [2:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=150s) problem. What we will be talking about [2:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=153s) how we are doing intraodality u issues. [2:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=157s) So this is consider that like a video we [2:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=159s) have a big video and then uh suddenly [2:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=161s) you are seeing something some ad agenda [2:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=163s) some political things or something out [2:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=165s) there which shouldn't be there in the [2:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=167s) video as when you clicked on it. So how [2:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=169s) do we actually look into the small [2:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=171s) segments of the uh video understand this [2:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=174s) problem and figure out where there is an [2:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=176s) anomaly or kind of like adversarial [2:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=178s) behavior out there which shouldn't be [2:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=179s) there in the first place. So this is the [3:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=181s) first problem which requires a lot of [3:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=183s) granular understanding vision [3:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=185s) understanding and a video understanding [3:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=186s) to see what is happening at a very clip [3:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=188s) and a very frame level. The second [3:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=190s) problem is uh to understand the [3:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=192s) unoriginal content. In today's economy [3:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=195s) with the AI tools available, it's really [3:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=197s) easy to actually duplicate the content. [3:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=199s) When someone uh upload a video or [3:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=202s) something, we see that okay, it is being [3:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=204s) copied, it is being transformed and with [3:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=206s) tools, it is getting much easier to [3:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=208s) transform these videos. U so how do we [3:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=210s) actually detect this kind of unoriginal [3:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=212s) content? How do we figure out the source [3:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=214s) of the video? And um this has actually [3:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=217s) caused a problem in attribution credit u [3:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=220s) ecosystem imbalance and lot of like user [3:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=222s) fatigue where we are seeing a lot of [3:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=224s) repetitive videos which shouldn't be [3:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=225s) there in the first place. So uh coming [3:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=229s) to the multi- aent system uh the first [3:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=231s) thing is that uh why we are going to [3:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=233s) multi-agent not a single agent or single [3:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=235s) lenm because the problem is really [3:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=236s) complex. It requires a really [3:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=238s) specialized uh nodes and uh kind of like [4:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=241s) an understanding at each part of [4:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=243s) retrieval content understanding and then [4:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=245s) reasoning and if you can solve a problem [4:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=248s) with one agent one LLM you we don't need [4:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=250s) to actually do it on multi- aent [4:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=253s) systems. So this is where the [4:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=256s) centralized brain how we are actually [4:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=258s) thinking kind of like how the uh [4:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=259s) decomposition happens of this problem. [4:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=262s) The first is basically you have a video [4:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=264s) and kind of like a clip video ID and [4:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=266s) everything given to a reviewer agent. [4:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=268s) This is a centralized agent. This is [4:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=270s) basically the orchestrator. Consider [4:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=271s) that as an API gateway of uh doing a [4:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=274s) sing signal decomposition and finding [4:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=276s) what is happening around in the image. [4:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=278s) So what this guy do is basically this [4:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=280s) guy takes ask the perceiver agent and [4:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=283s) perceiver agent is the one which is [4:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=285s) basically considered that a very [4:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=287s) sophisticated VLM expert with lot of [4:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=289s) image and the video tools at uh [4:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=292s) disposal. Uh perceiver agent will take [4:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=295s) the ID from the retriever reviewer agent [4:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=298s) and it fetches the video from the [5:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=301s) database. It decomposes into smaller [5:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=303s) parts using different tools, different [5:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=305s) like semantic embeddings and different [5:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=306s) kind of like temporal change that [5:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=308s) happens which is a little bit beyond our [5:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=311s) scope but of this talk but it's uh a [5:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=314s) technique which is making sure that we [5:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=315s) are not doing at a fixed frame rate but [5:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=318s) we are finding where the temporal change [5:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=319s) happen and compressing those similar [5:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=322s) frames into one or two frames and then [5:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=325s) the perceiver agent get all the data [5:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=326s) from a clip level. The video embedding [5:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=328s) level get all the information about a [5:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=331s) clip embedding tags OCR whatever is [5:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=333s) present out there what is the natural [5:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=335s) language description and what are the [5:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=336s) time stamp from which frame to which [5:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=338s) frame what are those this metadata looks [5:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=340s) like it gives that data to the reviewer [5:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=343s) reviewer will look into all the temporal [5:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=345s) this JSON object provided by the uh [5:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=347s) perceiver agent in a raw form along with [5:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=349s) embeddings along with semantic ids tags [5:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=352s) OCR and everything and it does a [5:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=354s) temporal analysis it looks into okay you [5:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=357s) know for from first frame to frame or [6:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=360s) frame number 360 or like till 6 second [6:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=362s) it was a video talking about a sports [6:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=365s) and suddenly we are seeing from 6 second [6:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=368s) to 6.5 second or frame number this to [6:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=370s) this we are seeing that like this is [6:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=371s) changing into some political thing. So [6:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=374s) just by looking into the metadata the [6:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=377s) reviewer agent is able to understand [6:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=379s) comprehend and figure out what exactly [6:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=380s) is the anomaly coming in the this [6:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=382s) temporal space and once it do that it [6:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=385s) figures out whether this is actually a [6:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=387s) modality misalignment or there is some [6:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=388s) bigger issue out there on there. So [6:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=390s) review agent will look and talk to the [6:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=393s) retriever agent and it tells that hey [6:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=395s) this is the video I'm looking into. [6:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=396s) These are the some of the metadata that [6:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=398s) I have already uh kind of like ddup and [6:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=400s) kind of like post-processed. Now give me [6:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=402s) some understanding of this similar clips [6:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=405s) which are available in the corpus. So [6:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=407s) retriever agent will look into all the [6:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=409s) signals provided by perceiver agent. All [6:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=411s) the metadata of a clip level and a whole [6:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=413s) video level and index it into different [6:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=416s) databases adaptively. For example the [6:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=418s) topics has to be maybe like a inverted [7:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=420s) index where we have topics and a lot of [7:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=422s) videos. For the embedding it will be a [7:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=424s) vector databases which is available out [7:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=426s) there. And for different kind of [7:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=428s) entities which I've extracted we have a [7:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=430s) graph databases kind of like where you [7:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=432s) review your agent will figure out okay [7:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=434s) these are the multiple databases these [7:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=435s) are the entities and these are the [7:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=436s) metadata let me index it so that in the [7:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=439s) online inference time I can figure out [7:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=441s) for a particular clip what are the [7:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=442s) similar clips what are the similar [7:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=444s) entities and topics which are available [7:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=446s) to fetch and improve the recall. [7:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=449s) So going into individual agent we talked [7:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=451s) about perceiver it looked into the [7:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=453s) entire video do a temporal decomposition [7:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=455s) into small clips based on the semantic [7:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=457s) embedding and couple of algorithms and [7:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=460s) it fine-tuned the VLM to actually emit [7:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=462s) all the real uh concrete and a very um [7:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=466s) granular information about like each [7:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=468s) clips and the entire video. Uh because [7:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=471s) we are dealing at a very big scale that [7:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=473s) means we cannot just go with the [7:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=475s) standard VLM which are out there. There [7:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=476s) has to be a compression. There has to be [7:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=478s) a cost effective way to actually serve [8:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=480s) this these models to do it at like [8:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=483s) billions of frames. That's where we are [8:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=486s) getting into the pre-training [8:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=487s) fine-tuning and knowledge distillation [8:09](https://www.youtube.com/watch?v=jNE8No-wvok&t=489s) quantization to actually deploy and [8:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=491s) solve the very very specialized VLM [8:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=494s) based on this particular problem and we [8:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=496s) will be talking a little bit brief about [8:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=498s) that. U the retriever agent is the one [8:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=500s) which we talked at a very high level and [8:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=502s) then what it does in an offline [8:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=504s) processing is that all the signal which [8:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=506s) is decomposed by the perceiver agent in [8:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=508s) an offline fashion consider that rather [8:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=510s) an agent you are actually taking that [8:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=511s) library and doing an offline analysis on [8:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=513s) a big scale let's say ray cluster or [8:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=516s) something like that and then once you [8:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=518s) have all those metadata it is basically [8:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=520s) indexing it into different databases [8:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=523s) doing an uh periodic offline clustering [8:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=525s) and finding what are the similar content [8:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=527s) what should be the embedding ID what [8:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=528s) should be cluster ID for each clips and [8:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=530s) entire video. So that is a data which is [8:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=532s) basically used by online inference to [8:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=535s) find similar videos and in online manner [8:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=537s) it given a query a clip ID and every all [9:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=540s) the metadata details it's figured out [9:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=542s) what is out there in the corpus which [9:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=544s) this guy has is similar to what like [9:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=546s) what are the different things which has [9:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=548s) a high similarity. So what it does it [9:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=550s) looks into databases is it find similar [9:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=552s) clips and like similar authors and [9:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=554s) different other metadata information [9:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=556s) rerank those candidates find uh and use [9:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=559s) some of the tools about like for example [9:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=561s) the traditional model span score like a [9:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=563s) classifier score to see what are the [9:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=565s) different candidates which might be spam [9:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=567s) which might not be high quality and [9:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=568s) those things out there and once it has [9:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=570s) those top end candidates is give back it [9:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=573s) to the reviewer and reviewer is where it [9:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=575s) actually process this is my main content [9:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=577s) signals and clip embeddings These are a [9:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=580s) similar video which is given by [9:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=581s) retriever. Let me think over it and [9:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=583s) whether I need to retraate and get more [9:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=587s) data from retriever. So uh that's where [9:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=590s) reviewer have all the temporal signals [9:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=591s) from a single clip. It has all the [9:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=594s) information of similar clips [9:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=595s) understanding and what is similar [9:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=596s) authors and other thing. It also has a [9:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=599s) realtime information about how the users [10:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=601s) are actually interacting with this [10:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=602s) video. What are the different kind of [10:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=605s) like a reports or like likes, dislike, [10:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=607s) the comments? what is the u sentiment of [10:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=611s) those comments out there right so a lot [10:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=613s) of these signals which are actually [10:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=614s) missed by the offline uh signals and the [10:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=616s) VLMs and other kind of agents are [10:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=618s) actually also incorporated to see if [10:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=620s) there is a change happen in the [10:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=622s) sentiment what is the response I'm [10:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=624s) getting in the real time so there are [10:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=625s) different tools available for this to [10:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=628s) actually understand the video not just [10:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=630s) from the content and from the semantic [10:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=632s) perspective but understanding from the [10:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=634s) user interaction perspective those [10:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=636s) signals are really really important [10:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=638s) So once we have this judge and [10:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=640s) everything we build this uh agentic [10:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=643s) framework and each of these agentic [10:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=645s) frameworks uh all all these three agents [10:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=648s) are actually powered by specialized uh [10:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=650s) VLMs and small scale VLMs in a way. So [10:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=654s) we will be talking about uh pre-training [10:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=657s) first. Now uh in most of the cases you [11:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=660s) see that okay you have a pre-training [11:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=661s) you fine-tune your vision transform a [11:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=663s) little bit here and there and basically [11:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=664s) tune it for your specific purpose. [11:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=667s) Uh the thing is that these VLMs what we [11:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=670s) have outside and available the [11:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=672s) foundational model the front end models [11:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=674s) they are trained on a very clean very [11:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=676s) nice data set very web data which is [11:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=678s) very well tuned cleaned and everything [11:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=680s) but uh data inhouse for a specific [11:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=683s) purpose is not having the same data [11:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=685s) characteristics. It's it is messy. It is [11:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=687s) generated by users. It is for your [11:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=689s) specific workflow. That means you need [11:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=692s) to pre-train on those image tokens and [11:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=694s) language to fine-tune your vision [11:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=696s) transformer from scratch and see whether [11:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=698s) there is a delta in fine-tuning it and [11:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=700s) tuning it from the scratch. That's where [11:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=702s) the pre-training is helpful. It's little [11:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=704s) expensive but um it if if it can get a [11:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=707s) delta that that actually works really [11:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=709s) well. The second is instruction [11:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=711s) fine-tuning where we have the two [11:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=714s) problems. We have a certain policies and [11:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=716s) certain guidelines. We know the content [11:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=718s) out there, signal out there and it [12:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=720s) understand and tune it on that [12:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=723s) particular data set with it with a [12:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=724s) particular output which is schema like a [12:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=727s) JSON schema to understand what is the [12:09](https://www.youtube.com/watch?v=jNE8No-wvok&t=729s) modality uh misalignment or duplication [12:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=731s) of scores and other chain of thought [12:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=734s) reasoning provided by reviewer agent. So [12:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=736s) here we have a video clip we have a [12:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=738s) vision encoder which we are already [12:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=740s) pre-trained a little bit like now we are [12:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=741s) doing a little bit more training on [12:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=743s) that. There's a projector which is [12:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=744s) sitting between the vision transformer [12:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=746s) and the L language models and it is [12:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=748s) actually basically a bridge between them [12:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=750s) and then we have our output depending on [12:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=752s) what is the instruction finetuning data [12:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=754s) we have on this side. So this is the [12:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=757s) critical part of actually making your uh [12:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=761s) model performance like go up for your [12:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=764s) domain specific. So uh the context is [12:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=767s) basically you have a role, you have a [12:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=768s) policy, what are the tools available to [12:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=770s) ground that into some of the metadata or [12:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=772s) and what are the other things available [12:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=774s) out there give it entire things into uh [12:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=777s) in in a very brief manner into the [12:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=779s) context and let it figure out like what [13:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=781s) the structured label you have for these [13:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=783s) labels are actually in-house label. [13:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=785s) These are the one where we have created [13:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=787s) like what exactly modality things are, [13:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=790s) what are the different uh issues we are [13:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=792s) seeing, what are the different chain of [13:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=793s) thought reasoning there should be there [13:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=795s) in the model and how does the output [13:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=797s) looks like for a human reviewer. So this [13:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=799s) is like a very high quality data data [13:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=801s) set that we are fine-tuning it on for [13:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=803s) different agents. Right? So once we have [13:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=807s) done with the pre-training just to [13:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=808s) understand the vision aspect of or the [13:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=810s) other modality aspects of the videos [13:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=813s) then we go into the finetuning to make [13:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=815s) it understand and provide the context [13:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=817s) and the output in the manner that we [13:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=819s) would want to actually process the data [13:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=820s) set on. So u the next phase is the DPO [13:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=824s) phase where uh which is basically this [13:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=827s) technique is used a lot in the uh post- [13:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=829s) training to actually fine-tune our [13:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=831s) models into a specific uh like realm [13:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=834s) domain policy understanding or [13:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=836s) something. But this can also be used a [13:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=839s) lot in the prediction for actually [14:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=840s) understanding what are the samples which [14:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=842s) are actually getting uh not so good by [14:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=845s) the by your multi-ent systems and by [14:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=847s) your u llms. So what uh can be done is [14:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=852s) that like uh you have this uh production [14:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=855s) data set which is coming out you have a [14:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=857s) lot of inference happening you take a [14:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=859s) subsample of these uh data which is from [14:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=862s) the production you pass it through LLM [14:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=864s) as a judge which is trained inhouse on [14:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=866s) the human label data set and then you [14:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=869s) have the human review Q to see what is [14:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=871s) the performance coming up on on the [14:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=873s) actual system. Once you have this thing [14:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=876s) if your performance is coming amazing [14:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=878s) and it is above like your whatever is [14:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=880s) the prediction threshold you have 95% [14:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=882s) for each of the problem it's great but [14:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=885s) if it is not that means there is a way [14:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=887s) there has to be a way to loan on these [14:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=889s) samples where the model did not do well [14:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=892s) that's where you have a human in the [14:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=893s) queue it understand those all the traces [14:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=896s) which is there from all the agents the [14:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=898s) LLM call MCP and it figured out where [15:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=900s) the problem happened is it like a chain [15:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=902s) of thought reasoning or is it like some [15:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=904s) of the wrong tool tools are called the [15:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=905s) retrieval did not work. So that entire [15:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=908s) validations and basically understanding [15:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=910s) of each and every hook and node both at [15:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=913s) a model intelligence level plus at a [15:16](https://www.youtube.com/watch?v=jNE8No-wvok&t=916s) harness level is what you figure out and [15:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=919s) say that okay these are the improvement [15:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=920s) that we I need to do on these samples [15:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=923s) which was incorrectly um sampled by our [15:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=926s) um system. So once you have this thing [15:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=929s) you have a positive sample negative [15:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=931s) sample what needs to be updated and [15:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=933s) that's where you retrain your model to [15:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=936s) see what uh how can we improve it better [15:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=938s) and this is the continuous improvement [15:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=940s) where we are looking into these samples [15:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=942s) reiterating re um improving the models [15:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=945s) and having a new data set out there from [15:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=947s) the production u like samples and trying [15:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=950s) to understand whether the drift happen [15:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=952s) or what exactly the model is doing. So [15:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=954s) this human in the loop is like always a [15:56](https://www.youtube.com/watch?v=jNE8No-wvok&t=956s) continuous thing where you have a daily [15:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=958s) sampling from the production and trying [16:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=960s) to understand like how the model and the [16:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=962s) L&M as a judge are doing. [16:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=965s) So uh because of the cost and ability to [16:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=970s) solve at this scale we cannot go with [16:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=972s) the standard like a like a VLM models [16:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=975s) from frontier sizes because it's not [16:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=977s) scalable. It requires a lot of [16:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=979s) inference. It's a lot of complexity and [16:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=980s) we are solving a very specific problem [16:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=983s) like I really don't care if the model [16:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=985s) can solve a coding problem. I only care [16:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=987s) about my domain specific problem. That [16:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=989s) is all I care. This is not exposed to [16:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=991s) the customer. This is an internal thing. [16:33](https://www.youtube.com/watch?v=jNE8No-wvok&t=993s) So I would actually do a off policy and [16:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=995s) on policy knowledge distillation and do [16:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=997s) a little bit of quantizations depending [16:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=999s) on like some experimentation [16:40](https://www.youtube.com/watch?v=jNE8No-wvok&t=1000s) quantization to understand whether a [16:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=1002s) forbit work braining float what actually [16:45](https://www.youtube.com/watch?v=jNE8No-wvok&t=1005s) work really well. Then I would have a [16:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=1006s) table of different sizes of [16:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=1008s) distillations and quantization and [16:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=1010s) understand and see where exactly is my [16:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=1013s) performance like up to the mark and [16:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=1015s) where I'm gaining a lot of compute and [16:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=1018s) the cost resources um savings from on [17:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=1020s) inference production by doing these [17:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=1022s) optimization. So this is really critical [17:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=1025s) because as a problem what we have solved [17:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=1028s) it's all good but in production it has [17:11](https://www.youtube.com/watch?v=jNE8No-wvok&t=1031s) to be scalable. It has to be cost [17:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=1033s) effective. It cannot just be like [17:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=1035s) something which is um out of the um [17:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=1040s) which is just out in the market because [17:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=1041s) we are solving a very specific problem [17:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=1043s) out here. Coming back to the evaluation [17:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=1046s) basically first is the task of success. [17:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=1048s) Of course these are the binary things [17:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=1050s) the modality happen it does not happen [17:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=1052s) like there is alignment or disalignment. [17:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=1054s) So precision, recall, F1 are obvious [17:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=1056s) metrics to understand what the task is [17:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=1059s) success is. But we want to look at a [17:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=1061s) system in a very holistic manner. Not [17:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=1063s) just like the end goal but what is [17:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=1064s) happening at each nodes. How well like [17:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=1067s) the retrieval system is working. What is [17:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=1068s) the latency and recall of the system and [17:51](https://www.youtube.com/watch?v=jNE8No-wvok&t=1071s) how well we are able to reason it? What [17:53](https://www.youtube.com/watch?v=jNE8No-wvok&t=1073s) is the chain of thought reasoning coming [17:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=1074s) from these models at each agentic level [17:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=1077s) or wherever the its reasonings are [17:59](https://www.youtube.com/watch?v=jNE8No-wvok&t=1079s) applicable which is basically a reviewer [18:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=1081s) agent in our case. And what is the [18:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=1084s) quality of that? Is it like [18:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=1085s) overthinking? Can we reduce a budget [18:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=1086s) somewhere like to make sure that like [18:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=1088s) the models is actually doing a good job [18:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=1090s) with less uh budget or can we increase [18:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=1093s) it the budget of the planning or the [18:15](https://www.youtube.com/watch?v=jNE8No-wvok&t=1095s) reasoning things to have it a higher [18:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=1097s) budget and make sure that the complex [18:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=1100s) problem that we are solving is maybe [18:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=1102s) having a much higher performance and [18:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=1104s) accuracy. So having an adaptive adaptive [18:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=1106s) reasoning budgeting is also uh quite [18:29](https://www.youtube.com/watch?v=jNE8No-wvok&t=1109s) important. And then we have the [18:31](https://www.youtube.com/watch?v=jNE8No-wvok&t=1111s) robustness. what are the edge cases we [18:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=1112s) are seeing? Uh what are the error rates [18:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=1114s) we are having? What are the different [18:36](https://www.youtube.com/watch?v=jNE8No-wvok&t=1116s) nodes and hooks where they are [18:38](https://www.youtube.com/watch?v=jNE8No-wvok&t=1118s) happening? Is it like a tool call is not [18:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=1119s) working really well? It's a retrieval [18:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=1121s) part or LLM is not doing a good job [18:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=1123s) where or those kind of thing. And then [18:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=1126s) we have a system efficiency where we are [18:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=1127s) not looking uh just at the performance [18:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=1130s) of the output and everything but we are [18:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=1132s) looking at each and every aspects of the [18:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=1134s) token cost which LLM are calling can you [18:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=1137s) optimize LLM little bit more to actually [18:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=1138s) save the cost. What is the efficiency [19:00](https://www.youtube.com/watch?v=jNE8No-wvok&t=1140s) which is we are saying and the latency [19:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=1142s) of each agents and the entire system [19:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=1145s) combined together. Then we have a LLM as [19:07](https://www.youtube.com/watch?v=jNE8No-wvok&t=1147s) a judge where we see that like in any of [19:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=1150s) the prediction system there is always a [19:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=1152s) data drift happening especially when you [19:14](https://www.youtube.com/watch?v=jNE8No-wvok&t=1154s) have the user generated content. So how [19:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=1157s) our LLM as a judge which is used for [19:19](https://www.youtube.com/watch?v=jNE8No-wvok&t=1159s) evaluation and everything is doing with [19:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=1160s) respect to the human queue. Is there a [19:22](https://www.youtube.com/watch?v=jNE8No-wvok&t=1162s) drift happening? Do we need to retrain [19:24](https://www.youtube.com/watch?v=jNE8No-wvok&t=1164s) our LLM as a judge on new data set which [19:26](https://www.youtube.com/watch?v=jNE8No-wvok&t=1166s) is coming from labeling team or how do [19:28](https://www.youtube.com/watch?v=jNE8No-wvok&t=1168s) we actually do on those parts in [19:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=1170s) optimization? We have the spatial [19:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=1172s) temporal reduction optimization which [19:34](https://www.youtube.com/watch?v=jNE8No-wvok&t=1174s) actually look into the frame which are [19:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=1175s) similar and just compress them into the [19:37](https://www.youtube.com/watch?v=jNE8No-wvok&t=1177s) one aspect. This reduces the total [19:41](https://www.youtube.com/watch?v=jNE8No-wvok&t=1181s) processing of the videos by a huge [19:43](https://www.youtube.com/watch?v=jNE8No-wvok&t=1183s) extent. The second one is basically you [19:46](https://www.youtube.com/watch?v=jNE8No-wvok&t=1186s) have a caching you have a viral content [19:48](https://www.youtube.com/watch?v=jNE8No-wvok&t=1188s) which is basically coming up and you do [19:49](https://www.youtube.com/watch?v=jNE8No-wvok&t=1189s) not want to have the same content out [19:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=1192s) there which is doing going through the [19:54](https://www.youtube.com/watch?v=jNE8No-wvok&t=1194s) entire pipeline and that's where you [19:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=1195s) have a high similarity score and just [19:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=1197s) skip the multi- aent system and just [19:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=1198s) make a call on that. The third one which [20:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=1201s) is really important is the metadata [20:02](https://www.youtube.com/watch?v=jNE8No-wvok&t=1202s) pruning. This is where we actually [20:04](https://www.youtube.com/watch?v=jNE8No-wvok&t=1204s) shrink the space from lot of candidates [20:06](https://www.youtube.com/watch?v=jNE8No-wvok&t=1206s) based on some of the metadata. For [20:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=1208s) example, some topics and some creator [20:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=1210s) which already have a really good record. [20:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=1212s) That means we really don't need to [20:13](https://www.youtube.com/watch?v=jNE8No-wvok&t=1213s) process these all these images all all [20:17](https://www.youtube.com/watch?v=jNE8No-wvok&t=1217s) the videos from the creator which has a [20:18](https://www.youtube.com/watch?v=jNE8No-wvok&t=1218s) really good high authenticity score [20:20](https://www.youtube.com/watch?v=jNE8No-wvok&t=1220s) which are really doing really well on [20:21](https://www.youtube.com/watch?v=jNE8No-wvok&t=1221s) this where the video quality is really [20:23](https://www.youtube.com/watch?v=jNE8No-wvok&t=1223s) high engagement is good those kind of [20:25](https://www.youtube.com/watch?v=jNE8No-wvok&t=1225s) things. A metadata is something which [20:27](https://www.youtube.com/watch?v=jNE8No-wvok&t=1227s) can be used to filter out the video [20:30](https://www.youtube.com/watch?v=jNE8No-wvok&t=1230s) which shouldn't even go to the systems. [20:32](https://www.youtube.com/watch?v=jNE8No-wvok&t=1232s) So some of these flags u would be [20:35](https://www.youtube.com/watch?v=jNE8No-wvok&t=1235s) helpful. Um so the last takeaway that u [20:39](https://www.youtube.com/watch?v=jNE8No-wvok&t=1239s) u we basically we can take out of this [20:42](https://www.youtube.com/watch?v=jNE8No-wvok&t=1242s) room is that like decomposition t is a [20:44](https://www.youtube.com/watch?v=jNE8No-wvok&t=1244s) key. Uh decompose uh a problem as and [20:47](https://www.youtube.com/watch?v=jNE8No-wvok&t=1247s) when necessary. Adaptive optimization is [20:50](https://www.youtube.com/watch?v=jNE8No-wvok&t=1250s) really important for uh ROI and cost [20:52](https://www.youtube.com/watch?v=jNE8No-wvok&t=1252s) feasibility. Good evaluation is [20:55](https://www.youtube.com/watch?v=jNE8No-wvok&t=1255s) paramount. Everything in and out depends [20:57](https://www.youtube.com/watch?v=jNE8No-wvok&t=1257s) on this. This is the foundation of your [20:58](https://www.youtube.com/watch?v=jNE8No-wvok&t=1258s) entire system and entire VLMs and [21:01](https://www.youtube.com/watch?v=jNE8No-wvok&t=1261s) production monitoring and qualitative [21:03](https://www.youtube.com/watch?v=jNE8No-wvok&t=1263s) improvement are really essential to make [21:05](https://www.youtube.com/watch?v=jNE8No-wvok&t=1265s) sure this is sustainable in long term. [21:08](https://www.youtube.com/watch?v=jNE8No-wvok&t=1268s) With that, I'll end it and thank you [21:10](https://www.youtube.com/watch?v=jNE8No-wvok&t=1270s) very much for listening. Thank [21:12](https://www.youtube.com/watch?v=jNE8No-wvok&t=1272s) [applause]