Physical AI's Next Bottleneck Is Finding the Right Video — Rafael Levi, Bright Data

A system that can see, understand and act needs examples of actions and their consequences. Rafael Levi, who works on data collection at Bright Data, opens with that requirement: improving the AI still leaves the question of what to feed it. His shorthand is memorable: “AI without data is just a box.” The presentation focuses on finding video that can help train robots and world models.

The opening history traces a progression from robots learning through images and real-world actions in 2022, to AI controlling multiple robots in 2023, to wider access through open source in 2024. As more labs gain access to models, collecting suitable experiences becomes a practical constraint. Levi also distinguishes the excitement around humanoids from their autonomy: many of the robots people encounter are still remotely controlled.

The scale comparison explains why he turns to video discovery. Levi puts language-model training material at trillions of words, image training material at billions of labeled images, and existing robot-action video at only about a million videos. These are his broad estimates rather than a like-for-like dataset census, but the distinction matters: a large supply of text or pictures does not automatically supply sequences showing how objects move when someone acts on them.

Opening a door seems like an easy action to collect. Pay someone to record it, then add the video to a training set. The complication is that the instruction changes the circumstances: demonstrating how to open a door for a camera may produce a different movement from walking into the house during an ordinary day. Camera awareness can change behavior too. Levi calls this instructed data and considers it biased; the recording does not establish how much that difference affects robot performance.

Other collection methods trade different constraints:

• Simulation: Virtual environments are cheap to use, but Levi questions whether their physics is faithful enough for the robot behaviors he wants to train. The attraction is inexpensive experience; the concern is whether that experience carries over to the physical world.
• Teleoperation: A person can control a real robot and record its actions, but collecting more hours requires more operator time. His example of recording eight hours a day makes the scaling problem concrete: a large dataset still depends on many people spending many days operating machines.
• Existing datasets: Reusing a prepared dataset avoids fresh collection, but the available supply is limited in the account he gives.

The alternative is to look for actions people have already recorded. Web video contains people handling objects, opening doors and encountering the consequences of motion and gravity. That makes it a potential source of physical experience without commissioning every demonstration. It also creates a discovery problem: useful interactions sit among enormous amounts of unrelated footage.

Levi cites a Meta training result as motivation: about a million hours of real-world video followed by 62 hours of robot data, after which the system could control a real robot. The result is presented without the model identity, task or evaluation conditions, so the useful lesson is narrower than a universal recipe. A large video stage can supply experience that a smaller robot-specific stage then connects to control; the cited example still includes real robot data.

How can a video of a person pouring water help a robot? The explanation starts with change between frames. Compare two images, measure the movement between them, then repeat across the sequence. The changing positions provide information about motion; Levi describes extracting angles and distances from that progression. A still image captures a state, while the sequence exposes how the state changes during an action.

That is a high-level account of extracting motion, not a complete method for turning human movement into robot commands. Levi explicitly acknowledges that downloaded video does not get a training system “100% there”: further processing is necessary. Keep that distinction in mind through the product demonstration. Finding a relevant clip supplies material for the training pipeline; it does not finish the pipeline.

Collecting video first and deciding what is useful afterward can be expensive. Levi reports that NVIDIA discards about 96% of video collected for Cosmos training, and that Stable Video Diffusion discards 74%. He uses a million-hour download as an illustration of the first percentage: only 4% would remain useful. The rates are cited claims rather than independently established measurements here, but they identify the cost at issue. Footage rejected after collection has already consumed bandwidth, storage and processing.

Bright Data’s proposed order is “search first collect second.” Describe the action you need, search an existing video index for that action, then receive the relevant snippets. The intended unit of collection becomes the action clip rather than the entire source video. Queries can describe a person washing dishes or folding a T-shirt, even when those actions are not the subject of the video’s title.

Where does selection happen, and what work remains afterward? The flow below follows the dishwashing query developed in the next demonstration. Searching and trimming happen before the customer collects the training material. Motion and distance processing still follow retrieval. The proposed savings come from collecting a smaller, more relevant slice of video; the index is the infrastructure that makes that earlier selection possible.

The demonstration makes the query concrete: a person washing dishes by hand in a sink, removing grease from a plate with a sponge and soap, rinsing, and placing dishes in a drying rack, with close-up hand interactions. Each added detail narrows what useful footage should contain. A video broadly about kitchens is insufficient; the desired material shows the manipulation and the steps around it.

The search then returns clips of people washing dishes. The observable change is from a written description of the desired activity to candidate video snippets containing it. Those snippets may come from videos about cleaning a kitchen or another broader subject. That is the point of indexing actions: an incidental moment can become a search result even when the source video was named for something else.

Levi says the demonstration was recorded with about 100 million indexed videos and that the index had since reached 1.1 billion. The playback is sped up, so it does not establish search latency, and the examples do not quantify retrieval accuracy. A second query asks for a human folding different types of clothes and returns folding footage. Together, the examples illustrate the intended interface: describe a visible action in detail, then collect clips that can feed subsequent training work.

The same workflow is available through an API, so repeated searches need not depend on a person using the demonstration interface. Levi describes returning a video snippet and its URL, along with a link to the original video. The result also carries information about where and how the query matched:

• Timestamp: Locates the relevant moment in the source video.
• Match score: Describes how close the result is to the query.
• Frame count: Reports how many frames concern the requested content.

Brand discovery uses the same gap between a video’s title and its visible contents. Imagine the example Levi gives: a person applies makeup, and a particular brand is visible on the table, while the recording’s title concerns something else. Searching the brand name in titles could miss that appearance. Searching indexed video content is meant to find the moment when the product is present or used. Here the useful result is a brand appearance rather than a robot demonstration.

For training, the intended benefit is less irrelevant footage entering the collection and preparation process. Levi connects noisy data to model problems and describes the product as removing that noise. The demonstrated capability supports a more limited conclusion: action retrieval can select material before bulk collection. Match scores describe query relevance; the talk does not establish that retrieval eliminates all unsuitable training data or downstream model errors.

Driving footage extends the idea from household manipulation to road events. Dashcam videos can contain someone running a red light, stopping at one, turning left or turning right. These are distinct behaviors within a larger recording, and action search could collect examples of each. Levi proposes them as useful material for self-driving systems and other world models; the recording does not demonstrate a driving model trained on the returned clips.

The ending returns to physics and everyday movement: sitting, walking and moving through the world. Levi’s frustration is with commissioning huge numbers of recordings while relevant actions already exist online. His practical objection is discovery. A title search asks what the uploader called the video; a training-data search needs to ask what happened inside it.

The proposed product brings discovery across public video providers, including YouTube and Vimeo, into one place. Its usefulness can extend beyond training and brand appearances. The final example is a gamer looking for footage of someone beating a particular level, even when no video is titled for that specific event. The same mechanism applies: find the relevant action within a longer recording, collect that portion, and process it for the task at hand.