This video was edited with AI agent. But how?

The recording opens with two kinds of educational video: a conversational introduction to Attention Is All You Need, presented as a landmark NLP paper, and a brisk description of OCaml as functional, imperative, and object-oriented. The latter ends with the jab that OCaml is “what Haskell wishes it could be.” These snippets set the scene for the practical problem that follows: automating the editing of learning videos.

Muhtesem introduces the Video Composer Agent, describing it as the world’s first open-source video-editing agent—a priority claim the presentation does not substantiate. The project began with a need to automate video editing for Reskill, a personalized-learning platform. Working with FFmpeg exposed limitations that pushed the team toward a more intuitive, flexible composition interface.

The library choice turned on where rendering happened and how editing operations were expressed:

That Remotion comparison reflects the team’s historical selection experience; current Remotion documentation supports browser rendering without server infrastructure, stable from version 4.0.491. After trying Core, the team met its author and decided to build the agent together.

Diffusion Studio Core exposes complex compositions through JavaScript and TypeScript. That gives the language model a direct way to act: generate code against the composition library, then execute that code. The editing program becomes the agent’s action. Instead of merely describing a desired edit, the model can express the operations that produce it through the same interface a programmer would use.

Muhtesem argues that code is the most expressive way to specify computer actions and points to unnamed research favoring code-based tool calls over JSON. The architectural consequence is clear even without a universal performance claim: the agent needs a programmable editing environment in which its generated actions can run. The talk supplies no video-editing benchmark comparing those action formats.

The agent starts a browser session with Playwright and connects to the operator UI, a video-editing web application designed specifically for agents. This is the environment where generated editing code operates on the composition. The application renders video directly in the browser using the WebCodecs API, avoiding the separate rendering backend that motivated the library search.

The browser is also connected to the agent’s Python environment. Helper functions transfer files from Python into the browser and back through the Chromium DevTools Protocol. This separates the agent’s orchestration environment from its editing runtime while providing a path for media to cross between them.

The editing workflow uses three tools, beginning with the user’s request:

1. Generate and execute: VideoEditingTool turns the prompt into editing code and runs it in the browser.
2. Retrieve context when needed: DocsSearchTool uses retrieval-augmented generation to supply relevant documentation.
3. Inspect the composition: after each execution step, sampled frames go to VisualFeedbackTool.

Muhtesem reports that the composition is sampled at one frame per second for visual review. This is the review cadence, not the output frame rate or rendering speed.

A small TypeScript helper expresses that sampling schedule. For a four-second composition, it requests review frames at seconds 0, 1, 2, and 3; extracting those frames remains the composition runtime’s job.

function reviewSampleTimes(durationSeconds: number): number[] {
  if (!Number.isFinite(durationSeconds) || durationSeconds < 0) {
    throw new RangeError("Duration must be finite and nonnegative");
  }

  const times: number[] = [];
  for (let second = 0; second < durationSeconds; second += 1) {
    times.push(second);
  }
  return times;
}

const sampleTimes = reviewSampleTimes(4);
// [0, 1, 2, 3]

Sampling gives the reviewer a sequence of visible composition states. It also leaves gaps between observations: an artifact that appears and disappears between samples may escape this check.

Muhtesem compares the visual-feedback arrangement to the generator and discriminator in a GAN: one part produces an edit, while another judges the result. Here that is a feedback analogy, rather than a description of adversarial training. Visual approval gates the final render. Once VisualFeedbackTool gives the green light, the agent proceeds to render the composition; the workflow slide presents the alternative as further refinement.

The team also shipped an llms.txt file and specific prompt templates to help agents use the editing environment. The on-screen example shows a documentation welcome section and terminology definitions. Those materials complement documentation search by giving the agent a prepared starting point for understanding the library.

Muhtesem calls llms.txt the equivalent of robots.txt for agents. The useful distinction is their purpose: llms.txt supplies curated context and documentation links, while robots.txt communicates guidance about automated access. For this editor, the relevant combination is library context plus task-specific prompting.

The browser runtime allows two deployment arrangements:

• Bring your own browser: run the agent against a browser session you provide.
• Connect remotely: attach through a WebSocket connection, with a separate GPU-accelerated browser session for each agent.

Muhtesem describes a load balancer behind the remote arrangement. This is a proposed deployment path rather than a demonstrated production service; the accompanying project description marks the remote WebSocket endpoint as work in progress.

The first agent implementation is in Python, with a TypeScript implementation underway at the time of the presentation. Muhtesem closes with a TypeScript variation on the familiar programming-language joke: any application that can be written in TypeScript eventually will be. That roadmap would bring the agent implementation into the same language ecosystem as the composition interface. The project is a collaboration between Diffusion Studio and Reskill.