# When Agents Meet Physical Data: The Other Physics of Agent Harnesses

Dmitry Petrov

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/bUJgirn4_yc-when-agents-meet-physical-data-other-physics) · [Video](https://www.youtube.com/watch?v=bUJgirn4_yc)

## When Agents Meet Physical Data: The Other Physics of Agent Harnesses

Video, sensor data and robot telemetry need more than generated code: they need schemas, recoverable execution, cheap verification and memory of work already done.

Basic familiarity with Python, database tables and object storage will help you follow the schema and execution examples.

### What does an agent see in a bucket of videos?

[0:02](https://www.youtube.com/watch?v=bUJgirn4_yc&t=2s)

How should a coding agent work with video recordings, sensor data and robot telemetry—especially when one project combines all three? A directory listing reveals filenames, but not the objects, events or relationships inside those binaries. Generating a script is only the beginning of understanding and processing them.

Even structured analytics needs specialized context. Petrov points to [Anthropic’s self-service analytics work](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude): its internal analytics-question evaluations reported accuracy no higher than 21% without skills. That figure concerns those internal questions, not data projects in general. [OpenAI’s in-house data agent](https://openai.com/index/inside-our-in-house-data-agent/) uses six layers of context. Both systems start with the advantages of structured data: tables, warehouses and existing execution engines. Physical-data projects must first make sense of messy files.

![Slide contrasts a structured warehouse with schemas, indexes and a query engine against petabytes of files in S3/GCS, including video, sensor data and robot telemetry. A black hole illustration appears above.](https://ai.engineer/api/talk-essay-media/342b2bd9d3d4cf77e77f6a544721d77b6a4e9e7a674a643592848d7ceccc14a7)

Physical data contrasts structured warehouses with petabytes of files in S3/GCS.

Dmitry Petrov built Data Version Control, which he describes as Git for data, and now works on [DataChain](https://github.com/datachain-ai/datachain). His architecture starts with the LLM as the brain, then supplies the missing capabilities around it: sight to understand data, legs to execute processing, hands to verify results, and memory to retain important datasets and reuse earlier work. **The harness must make the data understandable, executable, testable and memorable.**

### A small file inventory can conceal millions of objects

[2:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=152s)

A few thousand files may look manageable until processing exposes their internal hierarchy. Videos contain clips; clips contain frames; frames contain detected objects; objects carry confidence scores, types, classes and labels. Relationships between those levels are part of the data too. Counting files therefore gives a poor estimate of the analytical workload.

Petrov compares this density to a neutron star: roughly city-sized on the surface, yet more massive than the Sun. His illustrative data equivalent is 2,000 video files yielding millions of internal objects. The storage inventory looks small while the representation needed for analysis expands dramatically.

### Turn metadata into rows without splitting the programming model

[3:56](https://www.youtube.com/watch?v=bUJgirn4_yc&t=236s)

The first response is often to write metadata as JSON beside the images or videos in S3. As the extracted object count grows, this creates millions of small metadata files, with latency, efficiency and consistency problems. Moving the metadata into a centralized database improves the storage model, but can leave researchers managing two systems, two languages and two stacks.

| Approach | Metadata representation | Main consequence |
| --- | --- | --- |
| JSON sidecars | Files beside the binaries | Many small reads and consistency work |
| Separate database stack | Centralized metadata records | Researchers manage a second programming environment |
| Pydantic plus Python-to-SQL translation | Typed Python schemas backed by rows | Schemas and processing stay in Python |

Pydantic supplies the common language for schemas and code. Python expressions can then be translated into SQL, keeping the database’s query capabilities without scattering separate SQL islands through the codebase.

![Slide titled “One language. One schema: Pydantic.” shows Python models for File, BBox and Detection beside boxes labeled typed rows, schema, versions and lineage.](https://ai.engineer/api/talk-essay-media/795179798451545df3d1b559a7ffcc0e8a4d5f652a36611d81e89d21d93a7c98)

Pydantic schemas organize file and detection data into typed, versioned rows.

This solves a particular boundary problem: extracting structure from physical binaries so ordinary analytical tools can operate on it. A business dataset that already lives in warehouse tables starts on the other side of that boundary.

### Scope a dashcam analysis before spending compute

[5:52](https://www.youtube.com/watch?v=bUJgirn4_yc&t=352s)

The demonstration uses open-source DataChain as a data harness for Claude Code. Setup proceeds in order:

1. Install DataChain with `pip`.
2. Install its skill for the selected coding agent.
3. Launch the agent and describe the task.

Petrov says the integration also supports three other coding agents. He skips permission prompts to accelerate the demonstration; that is a demo choice, not a requirement of the architecture.

The task is motion analysis on an open dashcam-recording dataset. Video makes a useful example because it is common—and sometimes the dominant modality—in physical-AI projects, while still exposing the difficulty of turning a binary recording into useful observations.

Before processing, the harness asks questions that determine both cost and output shape:

- **Detection model:** It proposes **YOLO**, and Petrov selects the smallest model size.
- **Velocity estimation:** He selects the simplest available method; an alternative would take more compute time.
- **Granularity:** He chooses per-frame output rather than per-track output.
- **Scope:** He keeps the January subset of 91 clips and declines the offer to expand it.

These choices establish what the resulting dataset will mean before the expensive work begins.

### Pay for extraction, then query the result

[8:39](https://www.youtube.com/watch?v=bUJgirn4_yc&t=519s)

Petrov reports that analyzing roughly 90 videos took 24 minutes. With processing complete, he asks how many videos contain people. The demonstration returns people detected in 82 of 91 clips. These are results from the selected dashcam task; the recording does not establish a reproducible hardware configuration for the timing.

The answer comes from the local database, not another pass over the videos or a download-and-parse loop through JSON sidecars. The generated Python loads the dataset, filters for the `person` label, counts the relevant clips and compares that count with the total. The expensive perception step has already produced the information needed for the analytical question.

Petrov reports that roughly 90 videos generated about 100,000 records. This is the hidden density made visible. He projects that thousands of videos can reach millions of records, and that deeper analysis of objects can multiply the count by ten, twenty or even hundreds. Those projections describe how the representation can expand, rather than additional measured runs.

### The data model connects detections to their source

[11:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=692s)

Inspecting the generated code reveals the central artifact: a Pydantic model derived from the task requirements. A detection carries its video file, frame ID, timestamp, class ID, confidence and bounding box. The class identifies a person, car or another object; the bounding box is itself a nested structure.

The file is also structured data. It carries a path, size and available storage metadata such as checksums, version information and an ETag. Those fields should remain distinct: an [S3 ETag](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html) is not universally a whole-object MD5 checksum, particularly for multipart uploads. A compact Python representation of the described fields is:

```python
from pydantic import BaseModel


class FileRecord(BaseModel):
    path: str
    size: int
    checksum: str | None = None
    version_id: str | None = None
    etag: str | None = None


class BoundingBox(BaseModel):
    x1: float
    y1: float
    x2: float
    y2: float


class Detection(BaseModel):
    file: FileRecord
    frame_id: int
    timestamp: float
    class_id: int
    label: str
    confidence: float
    bbox: BoundingBox
```

The nested Python object becomes queryable database data rather than another JSON file in S3.

Once these fields are rows, familiar analytical operations apply: filter labels, group observations, count matching clips and relate results back to source files. **The binary remains in storage; its extracted meaning becomes structured data.** That separation is what makes repeated questions inexpensive relative to repeated perception.

### Connect typed Python to distributed execution

[13:39](https://www.youtube.com/watch?v=bUJgirn4_yc&t=819s)

Schemas reveal what is inside the files and how large the problem is. Processing terabytes still requires an **execution engine**. It must use resources efficiently, account for potentially expensive LLM calls and preserve completed work when a job fails. A structured-data warehouse already provides an execution engine; an unstructured-data system needs to supply that capability around its files.

Teams may assemble orchestration and distributed compute with Ray or Spark. Petrov particularly likes Dask’s model of connecting computation to structured data. His corresponding design for physical data joins Python functions, Pydantic input and output schemas, metadata in a warehouse, and files in object storage. Together, these give the engine the information needed to distribute work across machines or threads according to the files being processed and the schemas being produced.

At the processing boundary, a file from S3 becomes a collection of detection objects destined for the warehouse. The user writes an ordinary typed Python function that yields those objects. Petrov’s point about avoiding extra function annotations is that no special execution decorator is needed: the specified input and output types still matter. The engine connects storage, execution and persistence around that function.

Resource allocation belongs outside the detection logic. Petrov illustrates the intended simplicity by asking to run on 40 machines. The saved configuration visible in the demonstration instead shows `parallel=4` and `workers=30`; it is a configuration example, not evidence of a completed 40-machine run.

| Function shape | Output per input file | Use |
| --- | --- | --- |
| Mapper | One object | A file-level result |
| Generator | Multiple objects | Detections within a file |

The pipeline applies the function and saves its output as a named dataset—a database table underneath. In the visible code, detection generation feeds the persisted `dashcam-jan` dataset.

![Code editor shows detection objects being yielded and a pipeline that reads video storage, sets parallel=4 and workers=30, loads a model, generates detections and saves dashcam-jan. The status line confirms the file was written.](https://ai.engineer/api/talk-essay-media/d667bfcc64f465dae3151a2642abc2d46c93cad7a5938dd0c810e4a79c1581ba)

The saved Python pipeline connects detection generation to a persisted dataset.

### A failed run should not erase completed work

[18:01](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1081s)

Processing heavy files can take a long time, especially when understanding their contents requires LLM calls. If a bug or API failure interrupts a run over hundreds of thousands of files, restarting everything discards both time and money. Repeating that failure compounds the waste.

**Checkpointing and incremental processing preserve different kinds of work.** Checkpoints let a repaired run recover completed results. Incremental processing lets a later run handle newly arrived files without recomputing existing outputs. Petrov describes both as essential. Current DataChain examples make the operational boundary explicit: recovery retains committed batches, an uncommitted batch may repeat, and the incremental example enables `delta=True`. A rerun therefore needs the appropriate recovery or delta configuration; merely invoking a script again is not a guarantee of zero repeated computation.

### Give the agent a cheap way to check its answers

[19:10](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1150s)

Coding agents use tests both for quality control and to reason about the environment. These are the harness’s hands: a way to touch the system and check what happened. Data work needs that feedback too, but a test that repeatedly processes raw videos can be prohibitively slow. Petrov draws on Anthropic’s distinction between software tasks with many acceptable implementations and data questions that usually have one correct answer.

The first improvement is to stop treating every question as a new Python computation over raw binaries. Build metadata datasets around the raw data so the relevant questions can be answered quickly. Dimensional and multidimensional modeling, star schemas and the one-big-table approach offer established ways to organize that layer. The choice of layout serves the same purpose: make the next analytical operation cheap enough to support iteration.

The harness changes the agent’s procedure before it answers:

1. Check whether existing datasets contain the metadata needed for a single SQL-like query.
2. If the metadata is missing, build the necessary layer.
3. Make that layer broad enough to answer related questions, rather than only the current wording.
4. Retain it for the user and teammates to reuse.

This turns a question into an opportunity to improve the data representation. Future checks can operate on the accumulated metadata instead of repeatedly paying for extraction.

### Remember why a dataset exists, not just its name

[21:50](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1310s)

Once useful slices exist, they can answer questions without rerunning expensive compute. But that saving disappears if another person or agent cannot discover them. Petrov describes teams paying twice, three times or four times for the same analysis. Coding agents such as Copilot, Codex and PI already use source-code context and indexes; a data harness needs comparable memory of the data work already performed.

A dataset name alone is insufficient. Memory should preserve why the dataset was built, the session context that motivated it and a useful description, often enriched by an LLM. It should also retain the producing source code, whose importance Petrov connects to OpenAI’s data-agent work. The knowledge base must expose this information to both agents and people. In the demonstration, it takes the straightforward form of Markdown files.

The record for the newly created dataset includes:

- **Description and rationale:** What the dataset contains and why the conversation led to its creation.
- **Storage dependencies:** The source storage location and directory.
- **Preview and schema:** Sample data and its structure.
- **Statistics and source code:** A sense of the dataset’s contents and the program that produced it.

The code is especially valuable because it explains the transformation, not merely the resulting column names.

These records connect source data in the bucket, producing code in the knowledge base and results in the warehouse into **data lineage**. Sharing that knowledge makes the existing dataset—and the resources already spent creating it—discoverable to teammates. Petrov’s intended behavior is that a later question about the same directory uses a suitable existing result instead of launching the extraction again. The harness’s memory therefore changes the next execution decision, not just the documentation available afterward.

### Build the stack around the cost of physical data

[25:52](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1552s)

The complete stack begins with the mass of unstructured files in object storage. A compute engine performs expensive extraction, including LLM calls where needed. A **Dataset DB** organizes the resulting metadata into datasets and useful slices. A **knowledge base** makes those results and their context available for reuse. Each layer makes the work beneath it more accessible without requiring that work to be repeated for every question.

![Four-tier pyramid labeled Raw files · S3, Compute engine, Dataset DB and Knowledge base, with build-once and read-forever arrows and the footer “Build it once. Read it forever.”](https://ai.engineer/api/talk-essay-media/a916358459f0f1f9a9137a62cdf0c9c402ddf9bfb4ef9e0971b3569cfbcba217)

The stack connects raw files, compute, a dataset database and a knowledge base.

Copilot, Codex and Claude Code can generate the processing code, but their default habits do not necessarily account for the cost and density of physical data. Petrov’s closing prescription is therefore not simply to choose a stronger model: teams are already using frontier models. The missing capability is a harness that understands these constraints and supplies the surrounding data system. DataChain implements some of those principles as an open-source project. Giving an agent that infrastructure is what Petrov means by adding mass to it: the ability to build on data work already done.

## Resources

- [How Anthropic enables self-service data analytics with Claude](https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude) — Anthropic's internal analytics architecture, including curated skills, semantic definitions, evaluation and maintenance.
- [Inside OpenAI’s in-house data agent](https://openai.com/index/inside-our-in-house-data-agent/) — Six context layers combining table usage, human annotations, code enrichment, institutional knowledge, memory and runtime inspection.
- [DataChain](https://github.com/datachain-ai/datachain) — Open-source Python library for typed datasets over object storage, with installation instructions and processing examples.
- [DataChain documentation](https://docs.datachain.ai/) — Current guides to DataChain's processing engine, dataset database and agent integrations.
- [DataChain as a data harness](https://docs.datachain.ai/harness/) — Maps coding-agent context and memory onto persistent datasets and a shared data knowledge base.
- [Amazon S3 checksums and ETags](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity-upload.html) — Explains object integrity checks and when an ETag does or does not represent a whole-object MD5 digest.

## Slides

### When Agents Meet Physical Data

Dmitry Petrov explores why coding agents struggle with physical data, including video, sensor data, and robot telemetry, and how to address those problems.

[0:05](https://www.youtube.com/watch?v=bUJgirn4_yc&t=5s) [Title slide for Dmitry Petrov’s talk, “When Agents Meet Physical Data: The Other Physics of Agent Harnesses.”](https://assets.ai.engineer/sites/aiecode2025/sha256/d442eea3573390b3b12ca22fb3d8275cc2d0fe9ad2adbc84bd514858b5807498.webp)

AI ENGINEER WORLD'S FAIR 2026

When Agents Meet Physical Data
The Other Physics of Agent Harnesses

Dmitry Petrov
Co-Founder, DataChain

[redacted]
[redacted]

### Layers of data context

OpenAI’s data agent builds context in six layers, from table usage and human annotations through institutional knowledge and memory to runtime context. The speaker situates this in structured warehouse data.

[0:59](https://www.youtube.com/watch?v=bUJgirn4_yc&t=59s) [OpenAI data agent context layers: table usage, human annotations, Codex enrichment, institutional knowledge, memory and runtime context.](https://assets.ai.engineer/sites/aiecode2025/sha256/21563a0eb1a379b13e32f3b5baccad575711b8361369fdacf0e3c1294e833e27.webp)

Layers of data context (OpenAI)

Data agent’s layers of context

6 RUNTIME CONTEXT
5 MEMORY
4 INSTITUTIONAL KNOWLEDGE
3 CODEX ENRICHMENT
2 HUMAN ANNOTATIONS
1 TABLE USAGE

OpenAI
In-house data agent ·
Jan 2026

### Physical AI and the extreme end of data

The speaker contrasts structured warehouses, with schemas and a query engine, with his world of petabytes of unstructured video, sensor data and robot telemetry in S3/GCS.

[1:09](https://www.youtube.com/watch?v=bUJgirn4_yc&t=69s) [A structured warehouse with schemas and a query engine is contrasted with petabytes of video, sensor data and robot telemetry in object storage.](https://assets.ai.engineer/sites/aiecode2025/sha256/304fd87fa2a0e7b71d022e6789d9466b162bfc3b517c4407e0484f8ec14517c9.webp)

THE EXTREME END OF DATA

Physical AI:
cross into a black hole - the laws change

THE FORGIVING VERSION
Structured warehouse
Schemas, indexes, a query engine

MY WORLD
Petabytes of files in S3/GCS
Video · sensor data · robot telemetry

### See: give the agent eyes

The first of four parts asks how an agent can see and understand complicated binary files in object storage. See is highlighted before Run, Verify, and Remember.

[2:30](https://www.youtube.com/watch?v=bUJgirn4_yc&t=150s) [Part one introduces seeing data, with See highlighted among See, Run, Verify and Remember.](https://assets.ai.engineer/sites/aiecode2025/sha256/b1fc9e4efe4be45276086b70f6d5f9d9c60e56b735abe7fb3abadf23adcacac8.webp)

PART 1 OF 4 · THE EYES

SEE
Give the agent eyes.

SEE
RUN
VERIFY
REMEMBER

### 2,000 files is a neutron star

Video files can contain nested clips, frames and objects. The slide compares 2,000 files and 100M objects to a neutron star: a small surface concealing too much for an agent’s context window.

[2:45](https://www.youtube.com/watch?v=bUJgirn4_yc&t=165s) [A neutron star analogy compares 2,000 files with 100 million objects and says the objects cannot fit in a context window.](https://assets.ai.engineer/sites/aiecode2025/sha256/fdffda20521adc177854d3785f2a1bf07dfd3109db925d2a3160de428e442751.webp)

PART 1 · SEE — PROBLEM

2,000 files is a neutron star.

RADIUS = FILES
2,000
size of a city

MASS = OBJECTS
100M
outweighs the sun

100M objects never fit a context window.
The agent is blind.

### JSON sprawl and two languages

Putting metadata in millions of JSON files creates latency; moving it to a database adds SQL alongside Python and a second stack to maintain.

[3:53](https://www.youtube.com/watch?v=bUJgirn4_yc&t=233s) [Slide comparing a pile of JSON files with a database that requires Python and SQL.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b1fdca316fa889b0feef8d9c560068f65ee485098edc17f8afbe2192124ef0a.webp)

LIVE · PART 1 — THE TRAP

JSON sprawl, then two languages.

OPTION A · A PILE OF JSON
Millions of tiny files
Slow · no versioning
Still not a dataset

OPTION B · A DATABASE
Python + islands of SQL
Two languages to maintain
Worse for agents

Two escapes from messy files — both break.

### DataChain dashcam dataset demo

The demo installs DataChain skills for Claude Code, then prompts the agent to detect objects and their velocity in dashcam clips and save the results as dashcam-jan.

[6:10](https://www.youtube.com/watch?v=bUJgirn4_yc&t=370s) [Terminal showing a command to install the DataChain skill for Claude locally.](https://assets.ai.engineer/sites/aiecode2025/sha256/75ce2c02ffe3df9162d802db85d12a1079fe21564e61e3c7c6b70c8b6105b52c.webp)

demo/ $ datachain skill install --local --target claude

Interface text:
datachain skill install --local --target claude

[6:42](https://www.youtube.com/watch?v=bUJgirn4_yc&t=402s) [A coding assistant receives a request to detect objects and their velocity in dashcam clips.](https://assets.ai.engineer/sites/aiecode2025/sha256/3744c258cdc770e755257e793f31bb4b888baacb13ade9c3ef1119a757b4570c.webp)

demo/ $ datachain skill install --local --target claude
Installed skills (local, target=claude):
  core → /Users/dmitry/src/demo/.claude/skills/core
  knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge
  jobs → /Users/dmitry/src/demo/.claude/skills/jobs

demo/ $
demo/ $ claude --dangerously-skip-permissions

Claude Code v2.1.191
Opus 4.8 (1M context) · Claude Team
~/src/demo

Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/
Detected objects with velocity. Save is as dashcam-jan dataset.

Pollinating… (2s · thinking)

Interface text:
Build fleet detections from dashcam clips
demo/  Opus 4.8 (1M context)
high · /effort
bypass permissions on (shift+tab to cycle) · ← for agents

### Dataset methodology loaded

Claude loads the DataChain knowledge skill and its dataset-creation methodology for a dashcam request to detect objects and their velocity.

[7:00](https://www.youtube.com/watch?v=bUJgirn4_yc&t=420s) [The assistant reports that it loaded the methodology for detecting objects and velocity in dashcam video.](https://assets.ai.engineer/sites/aiecode2025/sha256/df085ad480fc6572d3462ec1b080e65ce4e2ef65112f9a6f9caccabcee105639.webp)

knowledge → /Users/dmitry/src/demo/.claude/skills/knowledge
jobs → /Users/dmitry/src/demo/.claude/skills/jobs

demo/ $
demo/ $ claude --dangerously-skip-permissions

Build fleet detections from the dashcam clips in s3://dc-readme/fleet-cameras/2026-01/front/
Detected objects with velocity. Save is as dashcam-jan dataset.

I'll start by loading the datachain-knowledge skill since this involves creating a dataset from an S3 bucket.

Skill(/datachain-knowledge)
  Successfully loaded skill

This is Mode B (Dataset Creation). Let me start with the required precondition reads.

Read 2 files (ctrl+o to expand)

Methodology loaded. This is a Sense (L3) build — object detection + velocity on dashcam video.

Interface text:
Build fleet detections from dashcam clips
Claude Code v2.1.191
Opus 4.8 (1M context) · Claude Team
~/src/demo

### Choose a detection model

The harness uses YOLO with ByteTrack and asks for a model size. The demo chooses yolov8n, estimated at ~8 min for 91 clips, trading small-object accuracy for speed.

[7:27](https://www.youtube.com/watch?v=bUJgirn4_yc&t=447s) [A model selection screen compares three YOLO sizes, with the smallest selected.](https://assets.ai.engineer/sites/aiecode2025/sha256/81d602348e7a79391846c9c2c923b44b1b91d3cecc98fcc5838e2cf3b1867eb9.webp)

Which detection model? (method is ultralytics YOLO + ByteTrack tracker — the one sensible CPU option here at 28.6 ms/frame; this picks the weight class within it. Confidence threshold defaults to 0.25, overridable.)

1. yolov8n (~8 min) ✓
Nano. Fastest, ~28.6 ms/frame measured. ~8 min for 91 clips. Lower accuracy on small/distant objects. Good default for a first fleet pass. $0 (local CPU).
2. yolov8s (~12 min)
Small. ~3x inference cost, better recall on small objects. ~12 min for 91 clips. $0 (local CPU).
3. yolov8m (~20 min)
Medium. ~7x inference cost, best accuracy of the three on CPU. ~20 min for 91 clips. $0 (local CPU).
4. Type something.
5. Chat about this

Interface text:
Build fleet detections from dashcam clips
☒ Model  □ Velocity  □ Granularity  □ Scope  ✓ Submit
Enter to select · Tab/Arrow keys to navigate · Esc to cancel

### Define velocity

The presenter chooses image-plane velocity in px/s: bounding-box center displacement over time. Camera motion remains, so a parked car can register movement. Ego compensation adds compute cost.

[7:43](https://www.youtube.com/watch?v=bUJgirn4_yc&t=463s) [A velocity selection screen compares image-plane motion, ego-compensated motion, and both.](https://assets.ai.engineer/sites/aiecode2025/sha256/8367739f879da99edd6ceb480839b0f71e207e118bc320283134d54087919780.webp)

How should velocity be defined? (The dashcam itself is moving, so true real-world m/s is NOT recoverable without camera calibration + depth. The honest computable signal is image-plane motion of each tracked object's bbox center.)

1. Image-plane (px/s)
Raw apparent motion: bbox-center displacement between sampled frames / Δt → vx, vy, speed (px/s), direction (deg). Stores the full vector; downstream can normalize. Camera ego-motion NOT removed (a parked car reads non-zero as you drive past). Recommended for v1: simple, honest, reusable substrate.

2. Ego-compensated
Subtract global background flow (per-frame homography/optical flow) so static objects read ~[illegible] and the velocity reflects true relative object motion. More meaningful 'is it actually moving', but ~+50% wall and extra failure modes. Still stores raw + compensated.

3. Both
Store raw image-plane AND ego-compensated velocity per detection. Richest substrate, max reuse. ~+50% wall over the raw-only estimate.

Interface text:
Build fleet detections from dashcam clips
4. Type something.
5. Chat about this
Enter to select · Tab/Arrow keys to navigate · Esc to cancel

[7:46](https://www.youtube.com/watch?v=bUJgirn4_yc&t=466s) [A velocity menu compares image-plane motion, ego-compensated motion, and both.](https://assets.ai.engineer/sites/aiecode2025/sha256/f30490defb31b08a25a2a053913e4d143119e077a9b8fc5eb60b292569857515.webp)

How should velocity be defined? (The dashcam itself is moving, so true real-world m/s is NOT recoverable without camera calibration + depth. The honest computable signal is image-plane motion of each tracked object's bbox center.)

1. Image-plane (px/s)
Raw apparent motion: bbox-center displacement between sampled frames / Δt → vx, vy, speed (px/s), direction (deg). Stores the full vector; downstream can normalize. Camera ego-motion NOT removed (a parked car reads non-zero as you drive past). Recommended for v1: simple, honest, reusable substrate.

2. Ego-compensated
Subtract global background flow (per-frame homography/optical flow) so static objects read ~[illegible] and the velocity reflects true relative object motion. More meaningful 'is it actually moving', but ~+50% wall and extra failure modes. Still stores raw + compensated.

3. Both
Store raw image-plane AND ego-compensated velocity per detection. Richest substrate, max reuse. ~+50% wall over the raw-only estimate.

Interface text:
Build fleet detections from dashcam clips
← ☒ Model  □ Velocity  □ Granularity  □ Scope  ✓ Submit  →
4. Type something.
5. Chat about this
Enter to select · Tab/Arrow keys to navigate · Esc to cancel

### Row granularity

The presenter chooses one row per detection-frame, retaining per-frame velocity so later queries can filter or aggregate by track or clip.

[8:05](https://www.youtube.com/watch?v=bUJgirn4_yc&t=485s) [A menu compares rows per detection frame, track, and clip.](https://assets.ai.engineer/sites/aiecode2025/sha256/f4b0918cf04a123e3fdb759ea4704b551f1e54764a07f4f6e087d00d9d56c88b.webp)

Row granularity — one row per what?

1. Per detection-frame
One row per tracked object per sampled frame, carrying its instantaneous velocity (model's emit grain). Every downstream question — objects over X px/s, count per clip, per-track spee[illegible] profile — is one filter/group_by. Recommended.

2. Per track
One row per object track over its lifetime, with aggregate velocity stats (mean/max speed, duration). Compact, but loses the per-frame time series; recovering it needs a re-run.

3. Per clip
One row per video with object counts/summary. Too coarse — the velocity/label target gets buried in nested lists; avoid unless you only ever want clip-level totals.

4. Type something.
5. Chat about this

Interface text:
Build fleet detections from dashcam clips
Model · Velocity · Granularity · Scope · Submit
Enter to select · Tab/Arrow keys to navigate · Esc to cancel

### Build scope

The demo stays with 91 January front-camera clips. Adding 1,000 March clips would broaden coverage, but the menu estimates about 90 minutes versus 8 minutes for January alone.

[8:16](https://www.youtube.com/watch?v=bUJgirn4_yc&t=496s) [A scope menu offers 91 January clips, an expanded selection including March, or a 10-clip sample.](https://assets.ai.engineer/sites/aiecode2025/sha256/2f6acaec2674f236ab69101662a9b8307da33fb715d2db1e20b4a66dd5029da1.webp)

Scope of the build?

1. Jan front (91 clips)
s3://dc-readme/fleet-cameras/2026-01/front/ only — matches 'dashcam-jan'. ~8 min (yolov8n). Substrate covers Jan front only; the 2026-03/front sibling would need a separate build. Recommended.

2. + March front (1091 clips)
Also process fleet-cameras/2026-03/front (1000 clips). ~90 min (yolov8n). Broader reusable substrate across both months, but far beyond the 'dashcam-jan' ask — would save under a different name.

3. Sample (10 clips)
10 Jan clips only, ~1 min. One-shot sanity check; not full coverage, limited reuse.

4. Type something.

5. Chat about this

Interface text:
Build fleet detections from dashcam clips
Model · Velocity · Granularity · Scope · Submit
Enter to select · Tab/Arrow keys to navigate · Esc to cancel

### Counting clips with person detections

A filter and distinct count over the saved dashcam detections finds at least one person detection in 82 of 91 January clips, without rerunning inference.

[8:36](https://www.youtube.com/watch?v=bUJgirn4_yc&t=516s) [A recap reports 121,528 detections across 91 clips; a question asks how many videos contain people.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebce1c229f1f9238021d813b6083c02db67ad0924dce9b45ce87fae66d3fd09e.webp)

Because this is saved as reusable substrate, fleet questions now resolve as plain queries with no re-inference — e.g. fastest movers, per-clip object counts, or per-track speed profiles are each one filter/group_by over the 121k rows. Want me to build any of those as a Task on top?

One note: the bucket-level doc dc-knowledge/buckets/s3/dc-readme.md is still missing (its scan JSON predates this session and doesn't cover the fleet-cameras/ tree). I left it untouched since it's unrelated to this build — say the word if you'd like me to re-scan and enrich the bucket overview.

Worked for 24m 0s

recap: Built the dashcam-jan dataset: 121,528 YOLOv8n+ByteTrack object detections with image-plane velocity across all 91 January front-camera clips, saved with knowledge base updated. Next action is yours: optionally build a Task query (fastest movers, per-clip counts) on top. (disable recaps in /config)

how many videos have people in it

Interface text:
Build fleet detections from dashcam clips
demo/  Opus 4.8 (1M context) 12% ctx
bypass permissions on (shift+tab to cycle)

[9:11](https://www.youtube.com/watch?v=bUJgirn4_yc&t=551s) [A terminal reports person detections in 82 of 91 clips.](https://assets.ai.engineer/sites/aiecode2025/sha256/497fbd52ad8cf71bf9a82f7cefbef7a60d94bff0a6808e85de29402d69f8ebb3.webp)

recap: Built the dashcam-jan dataset: 121,528 YOLOv8n+ByteTrack object detections with image-plane velocity across all 91 January front-camera clips, saved with knowledge base updated. Next action is yours: optionally build a Task query (fastest movers, per-clip counts) on top. (disable recaps in /config)

how many videos have people in it

Reusing the dashcam-jan Sense layer — this is exactly the win it was built for. No re-inference; just a filter + distinct count over the saved rows.

Bash(cd /Users/dmitry/src/demo
    python3 -c "…")
  clips with ≥1 person detection: 82 / 91

82 of the 91 clips have at least one person detection.

Interface text:
Build fleet detections from dashcam clips
demo/  Opus 4.8 (1M context) 12% ctx
bypass permissions on (shift+tab to cycle) · ← for agents

### Detection count and caveats

A query over saved detections finds people in 82 of 91 clips without re-running detection. The count includes single-frame or distant detections at confidence ≥ 0.25.

[9:22](https://www.youtube.com/watch?v=bUJgirn4_yc&t=562s) [The terminal explains the 82-of-91 result and its confidence threshold.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e34f58c227ac18ef4ea86ff55168e28dcffa9dc477c5813b5216139e276d667.webp)

just a filter + distinct count over the saved rows.

Bash(cd /Users/dmitry/src/demo
    python3 -c "…")
  clips with ≥1 person detection: 82 / 91

82 of the 91 clips have at least one person detection.

This was a ~$0.001 query over the saved Sense layer instead of a ~13-min re-detection pass — the payoff of building dashcam-jan as reusable substrate.

A caveat on the number: it counts any clip with ≥1 person detection at confidence ≥ 0.25, so it includes single-frame, low-confidence, or distant blips. If you want a stricter definition — e.g. a person tracked across several frames, or confidence ≥ 0.5 — say so and I'll tighten the filter (still instant). I can also save it as a Task dataset (per-clip person presence/counts) if you'll ask more questions like this.

Save it as a Task dataset with per-clip person counts

Interface text:
Build fleet detections from dashcam clips
Cooked for 13s
demo/  Opus 4.8 (1M context) 12% ctx

### Code for counting clips with people

The Python code filters saved detections for people and counts distinct source clips: 82 of 91 contain at least one detection. The counts run as SQL over saved rows, without repeating inference.

[10:12](https://www.youtube.com/watch?v=bUJgirn4_yc&t=612s) [Python code filters person detections and counts distinct source clips.](https://assets.ai.engineer/sites/aiecode2025/sha256/257686dce9b8c4f341fb8ca08b56e101458e9eaa7a019c8efe12f66759b24eef.webp)

what code did you use to answer this

Here's the exact code I ran:

import datachain as dc

d = dc.read_dataset("dashcam-jan")
total = d.distinct("det.source.path").count()
with_people = d.filter(dc.C("det.label") == "person").distinct("det.source.path").count()
print(f"clips with ≥1 person detection: {with_people} / {total}")

What each line does:
- dc.read_dataset("dashcam-jan") — reads the saved Sense layer (no re-inference).
- d.distinct("det.source.path").count() → 91 total clips (det.source.path is the per-row pointer to the source .mov).
- .filter(dc.C("det.label") == "person") → keep only person detections, then .distinct("det.source.path").count() → 82 clips that contain at least one.

Both run as SQL over the warehouse-stored rows (Data Memory), not Python loops — that's why it's near-instant.

Note this was a one-off count, so I didn't .save() it (Task-layer, persist-by-exception). If you

Interface text:
Build fleet detections from dashcam clips

### Per-clip people counts

DataChain counts 82 of 91 clips with at least one person detection. The assistant then proposes grouping detections by source clip and saving the per-clip counts for reuse.

[10:36](https://www.youtube.com/watch?v=bUJgirn4_yc&t=636s) [The code explanation states that the count was not saved and introduces a reusable per-clip dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc9fab1aef47d32168302f1f2677331fa8f5afe9a01c85e1000de0b4e1550eeb.webp)

Here's the exact code I ran:

import datachain as dc

d = dc.read_dataset("dashcam-jan")
total = d.distinct("det.source.path").count()
with_people = d.filter(dc.C("det.label") == "person").distinct("det.source.path").count()
print(f"clips with ≥1 person detection: {with_people} / {total}")

What each line does:
- dc.read_dataset("dashcam-jan") — reads the saved Sense layer (no re-inference).
- d.distinct("det.source.path").count() → 91 total clips (det.source.path is the per-row pointer to the source .mov).
- .filter(dc.C("det.label") == "person") → keep only person detections, then .distinct("det.source.path").count() → 82 clips that contain at least one.

Both run as SQL over the warehouse-stored rows (Data Memory), not Python loops — that's why it's near-instant.

Note this was a one-off count, so I didn't .save() it (Task-layer, persist-by-exception). If you want it as a reusable per-clip dataset — e.g. person presence + counts per clip — the saved version would be:

Interface text:
Build fleet detections from dashcam clips

[10:41](https://www.youtube.com/watch?v=bUJgirn4_yc&t=641s) [Code groups person detections by source clip and defines a saved dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/188bda945d13a61774b7ca005f3face8f1255ef78f43bc4441606f0967c78095.webp)

    .group_by(
        n_person_detections=dc.func.count(),
        n_person_tracks=dc.func.count(dc.C("det.track_id")),  # approx; distinct tracks per clip
        partition_by="det.source.path",
    )
    .save(
        "dashcam_jan_people",
        attrs=["cast:task", "scope:onetime", "source:dashcam_jan_people"],
        description="Per-clip person-detection counts over dashcam-jan (conf≥0.25).",
    )
)

Want me to run that saved version?

how many records do we have here

Interface text:
Build fleet detections from dashcam clips
Brewed for 8s
demo/ Opus 4.8 (1M context) 12% ctx
bypass permissions on (shift+tab to cycle)

## Full transcript

[0:02](https://www.youtube.com/watch?v=bUJgirn4_yc&t=2s) How coding agent work with physical data? Video recordings, sensor data, robot telemetry, and sometimes all of those combined in a single multimodal project. If you try this, you probably have seen how badly it fails, and today we'll discuss the reasons and all the physics laws, uh, behind the problems and how to fix it.

[0:30](https://www.youtube.com/watch?v=bUJgirn4_yc&t=30s) This year, two frontier labs published very interesting results and surprising results that agents in general are not good at data. Uh, Anthropic published that, uh, accuracy, uh, for data projects on their agents is only twenty-one percent until you add specific data harnesses to them

[0:55](https://www.youtube.com/watch?v=bUJgirn4_yc&t=55s) and provide context. OpenAI published a whole layers of context, six layers of context in order to make the a... data agent work. And all of those are on structured data which lives in a warehouses with tables, execution engine, and all this like luxury.

[1:20](https://www.youtube.com/watch?v=bUJgirn4_yc&t=80s) In my life, I don't have this luxury unfortunately, because I live in a very extreme side of the data universe, messy, unstructured data.

[1:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=92s) Uh, I've worked with data for about ten years. Uh, I built, uh, Data Version Control project, Git for data, and now work on DataChain.

[1:45](https://www.youtube.com/watch?v=bUJgirn4_yc&t=105s) In order to make agent work for unstructured data, for physical data, we need to build not only the brain, which we already have, right? It's LLM. Uh, but we need to make harness, data harness, uh, to-- for agents to understand this physical world.

[2:08](https://www.youtube.com/watch?v=bUJgirn4_yc&t=128s) It should see the data properly. It should be able to run, kind of giving him a leg. Uh, it should be able to touch data, verify the result, run tests, and also remember the crucial important data sets, im- important result, so this information could be reused, uh, in the future project.

[2:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=152s) And the first we start from the how you see data. Uh, what does it take to understand all these complicated binary file that you have in your object storages?

[2:47](https://www.youtube.com/watch?v=bUJgirn4_yc&t=167s) Uh, in reality, how it usually looks, uh, there are just a several files, sometimes several thousand of files, right? It doesn't seems like a big deal, right? But what is usually happens, those files are very complex inside.

[3:04](https://www.youtube.com/watch?v=bUJgirn4_yc&t=184s) That what makes unstructured multimodal data complicated. And from two thou-- uh, because video recordings might have clips inside. Clips might have frames. Frames, objects. Objects, confidence, type, class, label.

[3:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=203s) Uh, there's connection between those pieces together. It makes kind of like explosion. It's-- it, it looks like a neutron star.

[3:34](https://www.youtube.com/watch?v=bUJgirn4_yc&t=214s) It small size on the surface, right? It's just a size of, uh, one city, but the mass of this object is tremendous. It's more than the mass of our sun.

[3:45](https://www.youtube.com/watch?v=bUJgirn4_yc&t=225s) Uh, and one... two thousand objects, two thousand files of videos could easily generate you millions of object inside the videos.

[3:56](https://www.youtube.com/watch?v=bUJgirn4_yc&t=236s) And how people usually do deal with these problems? They usually go through like a two major steps. First step,

[4:04](https://www.youtube.com/watch?v=bUJgirn4_yc&t=244s) let's put this meta information to JSON files and put it on S3 next to the images, right? And they end up with a millions of JSONs, crazy latency, not efficiency, not consistency.

[4:18](https://www.youtube.com/watch?v=bUJgirn4_yc&t=258s) And the next idea, why don't we use database? Brilliant. And the most advanced team do exactly this. Let's put a centralized database when all the met- metadata is, uh, in there.

[4:33](https://www.youtube.com/watch?v=bUJgirn4_yc&t=273s) Great, but this way you end up with a two system, with a two programming languages and all the mess around, uh, two different stacks. And this stacks

[4:44](https://www.youtube.com/watch?v=bUJgirn4_yc&t=284s) useless for most of the researchers because they don't wanna deal with this complexity.

[4:51](https://www.youtube.com/watch?v=bUJgirn4_yc&t=291s) We found that the easiest way for researchers and developers to deal with the schema is Pydantic. Uh, so you use the same language for the data, for the schemas, as well as code.

[5:03](https://www.youtube.com/watch?v=bUJgirn4_yc&t=303s) Uh, there are no SQL island in your code base. So in, uh, that way, you kind of transition from the messy world of unstructured data to the structure. The only thing you need is just like transpilers to the SQL, uh, from, from Python, and I'll show you, uh, how it works.

[5:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=323s) But that's the way when you can create schemas, uh, and work with the schemas efficiently. I wanna emphasize that the problem we are solving here is very specific for unstructured data, physical data.

[5:36](https://www.youtube.com/watch?v=bUJgirn4_yc&t=336s) It doesn't exist in the structured data world, right? For example, in those, uh, OpenAI and, uh, Anthropic blog posts, because they work with the structured business logic, right? We work with the physical data, uh, binaries and such.

[5:52](https://www.youtube.com/watch?v=bUJgirn4_yc&t=352s) Let's see how it works. I will be showing you our open source project DataChain, uh, with, uh, data harness to coding agents So first you need to install the tool, right, using pip install.

[6:08](https://www.youtube.com/watch?v=bUJgirn4_yc&t=368s) So we have already done this. Then, uh, Skill needs to be installed. So you

[6:17](https://www.youtube.com/watch?v=bUJgirn4_yc&t=377s) choose the coding tool that you use. We will be using Claude Code, uh, but we support three more different, uh, coding agents. And then you run

[6:33](https://www.youtube.com/watch?v=bUJgirn4_yc&t=393s) your favorite coding agent. Uh, of course, we are skipping permissions, uh, just to make it faster, right?

[6:41](https://www.youtube.com/watch?v=bUJgirn4_yc&t=401s) And then you define the prompt and just solve your problem. So in this case, we will be analyzing, uh, motions, uh, in, uh, video cameras. Uh, that's, uh, open dataset from, um,

[6:58](https://www.youtube.com/watch?v=bUJgirn4_yc&t=418s) uh, dashcam recordings. Uh, video, it's a very usual use case for, uh, physical AI projects. Uh, majority of the project do, uh, do include, uh, video recordings, uh, as, uh, one of the modalities, sometimes the major modalities.

[7:16](https://www.youtube.com/watch?v=bUJgirn4_yc&t=436s) Uh, and that's one of the, I would say, more interesting and more challenging problems to solve, uh, in this, uh, physical AI world. Let's use, uh, this modality as an example.

[7:29](https://www.youtube.com/watch?v=bUJgirn4_yc&t=449s) This data harness asks, uh, some questions from users to understand the scope better. So in this case, it, uh, we need to choose the model. So it decided to use a YOLO model, and we need to choose the size.

[7:43](https://www.youtube.com/watch?v=bUJgirn4_yc&t=463s) So let's use the smallest one. Uh, we need to

[7:49](https://www.youtube.com/watch?v=bUJgirn4_yc&t=469s) choose the velocity. Uh, so there's a few ways how to track, like speed, uh, but let's use the simplest one. So the second one, uh, adds more, uh, or will require more time

[8:06](https://www.youtube.com/watch?v=bUJgirn4_yc&t=486s) for the compute and granularity. So per detection frame, per track. Uh, let's do just per frame in this case and the dataset itself. So we ask for January data, right?

[8:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=503s) Ninety-one clip, but it also ask for, uh, if we need to extend scope and, uh, analyze more images. So let's, let's stay here.

[8:39](https://www.youtube.com/watch?v=bUJgirn4_yc&t=519s) So it took us twenty-four minutes to analyze those, uh, ninety videos, and now we have all the information. We can ask, uh, questions about this. For example, [keyboard clicking]

[8:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=534s) how many, um, videos have people in it?

[9:15](https://www.youtube.com/watch?v=bUJgirn4_yc&t=555s) This information lives, uh, in the database, right? As we discussed, and it can very quickly query this database and return the result, right? So, okay, ninety-two, uh, oh, eighty-two out of ninety-one clips have some people, uh, detected, right?

[9:38](https://www.youtube.com/watch?v=bUJgirn4_yc&t=578s) And, uh, you don't need to kind of go through all the JSON files and download and analyze and parse, right? Information is here in the local database. It can easily, uh, quickly an-answer the question.

[9:55](https://www.youtube.com/watch?v=bUJgirn4_yc&t=595s) Uh, we can even see the, um, source code. [keyboard clicking]

[10:12](https://www.youtube.com/watch?v=bUJgirn4_yc&t=612s) Okay, just an easy, nice, uh, Python code, uh, to get, uh, get data, get data- dataset, right? Analyze, uh, apply some, uh, simple filters, uh, when label is equal to person, uh, count the number, right?

[10:34](https://www.youtube.com/watch?v=bUJgirn4_yc&t=634s) And then see the total, total number. And it goes like, uh, it runs against database, so super fast. So how big is, uh... How many

[10:47](https://www.youtube.com/watch?v=bUJgirn4_yc&t=647s) records do we have here? Uh, so that's about the size of our, uh, neutron star, right? It's not supposed to be big because we analyzed only like ninety videos and ninety videos generated one hundred thousands of records, right?

[11:06](https://www.youtube.com/watch?v=bUJgirn4_yc&t=666s) So that's how explosion is happening. Uh, if you analyze, uh, thousands of videos, right, uh, it goes to like a million, uh, million scale. And if you go to deeper to the objects, uh, it can easily do, uh, it can easily be multiplied by ten, twenty, even hundreds. [keyboard clicking]

[11:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=692s) Let's take a look at the source code.

[11:37](https://www.youtube.com/watch?v=bUJgirn4_yc&t=697s) This code was generated by the agents, uh, power agents using the harness. Uh,

[11:46](https://www.youtube.com/watch?v=bUJgirn4_yc&t=706s) the magic part here is the data model. So this data class, uh, that was generated based on our requirements and, uh, it's just a Pydantic, uh, usual Pydantic data model, uh, with a file, uh, right, the video file, uh- Frame ID, timestamp, uh, class ID.

[12:10](https://www.youtube.com/watch?v=bUJgirn4_yc&t=730s) That's how you know if it's a person or car or some object, confidence score, bounding box. There are a few more items here, and the object is nested. If you look at the-- If you take a look at the bounding box, right, there's, uh, several columns, but file is more interesting one.

[12:31](https://www.youtube.com/watch?v=bUJgirn4_yc&t=751s) Uh, file has, uh, path and the checksums, like a version of the file, ETag, size of the file. So everything that your cloud storage providers provides for you. Uh, and this information is just, uh, become a row in a database, uh, instead of a JSON file in S3.

[12:55](https://www.youtube.com/watch?v=bUJgirn4_yc&t=775s) Uh, that's why you can easily, uh, answer, uh, all the questions, uh, all the analytical questions, right? Like before, we ask how many people are there, uh, and it's just a matter of like a simple Python code that runs against the database, uh, to return like results super quickly.

[13:18](https://www.youtube.com/watch?v=bUJgirn4_yc&t=798s) Uh, that's how you can make sense of all the messy data you have on your storage, right, in a file. Uh, but you analyze, uh, you analyze this us- using rec- regular analytical, uh, tools and, uh, tool tricks.

[13:39](https://www.youtube.com/watch?v=bUJgirn4_yc&t=819s) With the schemas, you can see what is inside the object. You can see the scale of the problem. But how to do the actual heavy lifting wh- when we, when we deal with the terabytes of this messy data?

[13:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=834s) You do know that, uh, data harness have to have execution engine in order to deal with, uh, physical data. It has to work efficiently. Sometimes it spent a lot of resources a- and tokens, uh, if you use LLMs, and if it breaks somewhere in the middle, so you should be able

[14:18](https://www.youtube.com/watch?v=bUJgirn4_yc&t=858s) to recover and catch up with all the result that it's already processed, so you don't wanna waste your resources. In the SQL world, structured data world, execution engine is your data warehouse, right?

[14:31](https://www.youtube.com/watch?v=bUJgirn4_yc&t=871s) That's obvious and easy. In unstructured data world, you have to reinvent one.

[14:38](https://www.youtube.com/watch?v=bUJgirn4_yc&t=878s) And in some teams, they use different types of orchestration tools or distributed compute on Ray or Spark. I like model of Dask, which, uh, connects a compute with the data, structured data.

[14:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=894s) Uh, but unstructured data needs its kind of like own approach, uh, when you connect the Python and schemas with the execution engine.

[15:07](https://www.youtube.com/watch?v=bUJgirn4_yc&t=907s) To simplify it, you can connect those Python functions, Pydantic schemas, input sc-- input parameters, output parameters for the function, as well as the data warehouse with all the meta information and files on the storage all together in a, in a holistic experience.

[15:30](https://www.youtube.com/watch?v=bUJgirn4_yc&t=930s) And that's wh-- this way it will be more s- simple for developers to code, and it might be simple for the engine to distribute jobs to different machines or different threads based on the files they process, the schema they produce, uh, so far and so on.

[15:47](https://www.youtube.com/watch?v=bUJgirn4_yc&t=947s) It's kind of like a Dask approach, if you wish, but for unstructured data world.

[15:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=954s) To make the execution efficient, uh, you need to run this, uh, data processing, data crunching function very efficiently, right, in a parallel way or distributed way. And we use data models and the data storage models a lot.

[16:14](https://www.youtube.com/watch?v=bUJgirn4_yc&t=974s) So it kind of connects the file, which is a file in your S3 bucket, uh, to the result. Uh, that's a set of detection object which goes to the data warehouse, right?

[16:30](https://www.youtube.com/watch?v=bUJgirn4_yc&t=990s) So this function connects the storage and the data warehouse in a regular Python way. Uh, so that's a very typical Python code. There are no, like, extra assumptions about this code, right?

[16:44](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1004s) We don't even use like annotation for the function. It's just a function with the specified types. So the types are important, uh, and all of those are Pydantic. And result, you just generate, uh, generate the objects, and the engine connects all the storages and, uh, warehouse together, right?

[17:08](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1028s) That's how you define the parallel, uh, parallelization layer. You can say, [keyboard clicking]

[17:17](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1037s) uh, I just need to run it in 40 machines, right? That's how easy it's supposed to be for researchers to run like distributed compute, right? You just spec-specify how many resources you need, so you got it.

[17:29](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1049s) And the function itself, right? So we just use it, uh, run it through a generator, or it can be like mapper with one-to-one function. One function, uh, one file, it returns like one object to the database or generator one function, it returns like multiple objects in database.

[17:45](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1065s) And it is saved as a, like, dataset, right? Uh, a table under the hood, uh, in your, uh, database.

[18:01](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1081s) Processing of those heavy files usually takes a lot of time, and sometimes it's very expensive because you use LLM in order to understand what is inside, uh, of your binary files.

[18:15](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1095s) And the last thing you want is to lose this compute It's really sad when you process, uh, hundreds of thousands of files and it fails in the middle because of bugs or API call issue, and you don't wanna execute everything from scratch, right?

[18:33](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1113s) You are losing the whole half of this compute and sometimes doing this over and over again. Incremental update and data checkpoints, it's a must-have in this data world. If you fail, you fix the bug, run it again, and catch up all the result that you have already.

[18:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1134s) If you got more files in the bucket, uh, you run the script again, and it gets only the new files and, uh, update data based on the new compute only without recomputing the, the other stuff.

[19:10](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1150s) Running tests is a very fundamental part of coding agents. They do this all the time for quality control, for reasoning to better understand what is around in their universe.

[19:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1163s) It's kind of like a hands of your agents, hands of your, uh, uh, harness. In data, running test is super slow process. And how to do this because quality and accuracy of the questions is even more important in the data question, uh, data projects compared to software projects.

[19:46](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1186s) Uh, as Anthropic put it in the right way, in, uh, software projects, there's a lot of ways how to-- you can solve a particular problem. In data, there is usually only one way and only one current correct answer, uh, to solve the problem.

[20:06](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1206s) How to answer questions very fast on your

[20:13](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1213s) binary data mess. And first of all, you should stop running all this like complicated Python scripts on top of raw data. That's the most expensive, the slowest way of doing this.

[20:25](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1225s) Instead, you need to organize layer of meta information, all the like datasets, uh, or tables around this, uh, raw data, uh, that could answer these questions very quickly. And this is something that data industry knows for like a dozens of years.

[20:41](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1241s) It's a dimensional, multidimensional data modeling, uh, star schemas, one big table approach, and all these like fancy theories around, around this. And we need to use this more in order to make our unstructured massive data processing, uh, to be more efficient.

[21:01](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1261s) And we actually incorporated these techniques inside the agents, and when you ask a question, instead of answering the question right away, agent ask itself, "Do I have a proper datasets, proper metadata to answer this question quickly in a single like SQL-ish query?"

[21:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1283s) And if not, it tries to build this layer. It tries to make sure this layer is general enough to answer not your particular question, but a set of questions related to the one that you ask.

[21:38](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1298s) And that's where you are building these layers and layers of information that's, uh, could be reused by you and your teammates.

[21:50](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1310s) Now you spent a lot of time, a lot of resources to compute those useful slices of the data, and you can use those to answer some interesting questions, uh, efficiently without reruning, re-running expensive compute.

[22:08](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1328s) And guess what? People are doing the same job over and over and over again. You're paying double, triple, quadruple price to solving the same problem. So if you discover this neutron star or black hole, it's better to share this information with people, uh, so they won't be wasting their resources and their time, uh, to doing this

[22:32](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1352s) stuff. And coding agent as well as data agent doing this, uh, those tricks with memory a lot. Uh, your coding agent like Copilot, Codex, PI knows a lot about your source code, about with all the indexes, uh, and such.

[22:56](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1376s) Uh, on the data harness, you need to build and provide this context, uh, to your agents.

[23:06](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1386s) And that's not only the fact that, hey, there is a dataset, you need to provide a lot more information with this. Uh, why this dataset was built kind of like a context from the session, uh, description of the dataset which usually enriched by LLMs.

[23:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1403s) Source code, probably the most important part here is the source code. Uh, that's one of the conclusion in OpenAI data agent blog post. Uh, and this information needs to be explosed, uh, exposed in a way like knowledge base, some way that can easily be used by agents, by people, uh, so you are not wasting this time over

[23:46](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1426s) and over again. And the knowledge base is organized in a very traditional way, I would say. Uh, just a set of MD files, right? So, uh, you see this is how it looks like in, in my directory, but when I get more datasets.

[24:03](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1443s) So the dataset that we created, that one,

[24:08](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1448s) uh, this is the MD file. Uh, so description of the dataset, right? The session context, uh, so why this dataset was created based on the discussion. Dependency to the storage, to the directory that we pointed, uh, to in the beginning.

[24:27](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1467s) Uh, preview of the data, uh, very useful information to kind of have a sense of the data. This schema Uh, some stats on the data and the source code.

[24:42](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1482s) Uh, as we discussed, the most crucial part, uh, to understand, uh, what is data is about.

[24:52](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1492s) And all these pieces together, right, uh, the source data, right, in the bucket, the source code, uh, in the knowledge base, and the result in the,

[25:06](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1506s) uh, in the data warehouse create like a data lineage, right? So everything is connected. Agents knows everything. If you share the knowledge graph, then all your teammates already know about this, uh, about this data set, about the resources you spent in order to process.

[25:25](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1525s) So next time, if someone asks question about this directory, uh, the recompute won't happen. Uh, agents will be using, uh, your, your result, uh, the result which based on resources you already-- that you have already spent, right?

[25:42](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1542s) That's the magic of the, uh, data harnesses when agent knows everything about your data.

[25:52](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1552s) Let's put all the pieces together in a single stack.

[25:58](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1558s) In the bottom of the stack, there is some huge mass of your unstructured physical data in object storages. No one makes sense of this data, and you need to run expensive compute LLM calls to extract some meta information through the compute engine and organize this meta information in a data sets, in a data

[26:23](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1583s) set slices through the some, uh, Dataset DB. Knowledge base is a way how you share this information.

[26:34](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1594s) This is a world when your favorite coding agents, such as Copilot, Codex, Claude Code,

[26:42](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1602s) do not operate efficiently. Their intuition pushes them in the wrong direction because laws of physics changes.

[26:54](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1614s) And in order to make it, you don't use stronger models. Everyone use frontiers. Instead, instead, you are building data harness, data harness that understand the laws of this physical data, and that's a way how to make your favorite coding agent efficient with these problems.

[27:18](https://www.youtube.com/watch?v=bUJgirn4_yc&t=1638s) We implement some of those principles in DataChain project, which is open source. So please check it out and put some mass on your agent. Thank you.
