# This video was edited with AI agent. But how?

Muhtesem

AI Engineer Summit 2025

[Talk page](https://ai.engineer/talks/r0AG44qYKsI-this-video-was-edited-ai-agent-but) · [Video](https://www.youtube.com/watch?v=r0AG44qYKsI)

## This video was edited with an AI agent. But how?

A browser-based video editor gives an agent a programmable workspace: generate editing code, retrieve documentation, inspect sampled frames, and render after visual approval.

Basic familiarity with language-model tool calling and JavaScript or TypeScript will help; no video-codec expertise is required.

### From learning clips to programmable compositions

[0:30](https://www.youtube.com/watch?v=r0AG44qYKsI&t=30s)

The recording opens with two kinds of educational video: a conversational introduction to [Attention Is All You Need](https://arxiv.org/abs/1706.03762), presented as a landmark NLP paper, and a brisk description of OCaml as functional, imperative, and object-oriented. The latter ends with the jab that OCaml is “what Haskell wishes it could be.” These snippets set the scene for the practical problem that follows: automating the editing of learning videos.

Muhtesem introduces the [Video Composer Agent](https://github.com/diffusionstudio/agent), describing it as the world’s first open-source video-editing agent—a priority claim the presentation does not substantiate. The project began with a need to automate video editing for [Reskill](https://re-skill.io/), a personalized-learning platform. Working with FFmpeg exposed limitations that pushed the team toward a more intuitive, flexible composition interface.

The library choice turned on where rendering happened and how editing operations were expressed:

| Option | Experience described in the talk |
| --- | --- |
| FFmpeg | Command-based editing became limiting |
| Remotion | Appealing, but required server-side rendering in the evaluated setup |
| Diffusion Studio Core | Suitable API without a separate rendering backend |

That Remotion comparison reflects the team’s historical selection experience; current [Remotion documentation](https://www.remotion.dev/docs/client-side-rendering) supports browser rendering without server infrastructure, stable from version `4.0.491`. After trying Core, the team met its author and decided to build the agent together.

![Slide titled “How it started?” lists FFmpeg command limitations, Remotion dependencies and server-side rendering, and Core’s client-side rendering.](https://ai.engineer/api/talk-essay-media/b43a109e4cdd2b590d4de11837d7ff93e07969c17ad808b4cc57f084a806f9e4)

Why the project moved from FFmpeg and Remotion to Core.

### Make editing actions executable

[1:47](https://www.youtube.com/watch?v=r0AG44qYKsI&t=107s)

[Diffusion Studio Core](https://github.com/diffusionstudio/core) exposes complex compositions through JavaScript and TypeScript. That gives the language model a direct way to act: generate code against the composition library, then execute that code. **The editing program becomes the agent’s action.** Instead of merely describing a desired edit, the model can express the operations that produce it through the same interface a programmer would use.

Muhtesem argues that code is the most expressive way to specify computer actions and points to unnamed research favoring code-based tool calls over JSON. The architectural consequence is clear even without a universal performance claim: the agent needs a programmable editing environment in which its generated actions can run. The talk supplies no video-editing benchmark comparing those action formats.

### Give the agent a browser workspace

[2:25](https://www.youtube.com/watch?v=r0AG44qYKsI&t=145s)

The agent starts a browser session with **Playwright** and connects to the operator UI, a video-editing web application designed specifically for agents. This is the environment where generated editing code operates on the composition. The application renders video directly in the browser using the **WebCodecs API**, avoiding the separate rendering backend that motivated the library search.

The browser is also connected to the agent’s Python environment. Helper functions transfer files from Python into the browser and back through the **Chromium DevTools Protocol**. This separates the agent’s orchestration environment from its editing runtime while providing a path for media to cross between them.

### Generate, inspect, then render

[3:00](https://www.youtube.com/watch?v=r0AG44qYKsI&t=180s)

The editing workflow uses three tools, beginning with the user’s request:

1. **Generate and execute:** `VideoEditingTool` turns the prompt into editing code and runs it in the browser.
2. **Retrieve context when needed:** `DocsSearchTool` uses retrieval-augmented generation to supply relevant documentation.
3. **Inspect the composition:** after each execution step, sampled frames go to `VisualFeedbackTool`.

Muhtesem reports that the composition is sampled at one frame per second for visual review. This is the review cadence, not the output frame rate or rendering speed.

A small TypeScript helper expresses that sampling schedule. For a four-second composition, it requests review frames at seconds `0`, `1`, `2`, and `3`; extracting those frames remains the composition runtime’s job.

```typescript
function reviewSampleTimes(durationSeconds: number): number[] {
  if (!Number.isFinite(durationSeconds) || durationSeconds < 0) {
    throw new RangeError("Duration must be finite and nonnegative");
  }

  const times: number[] = [];
  for (let second = 0; second < durationSeconds; second += 1) {
    times.push(second);
  }
  return times;
}

const sampleTimes = reviewSampleTimes(4);
// [0, 1, 2, 3]
```

Sampling gives the reviewer a sequence of visible composition states. It also leaves gaps between observations: an artifact that appears and disappears between samples may escape this check.

Muhtesem compares the visual-feedback arrangement to the generator and discriminator in a GAN: one part produces an edit, while another judges the result. Here that is a feedback analogy, rather than a description of adversarial training. **Visual approval gates the final render.** Once `VisualFeedbackTool` gives the green light, the agent proceeds to render the composition; the workflow slide presents the alternative as further refinement.

![“How it works” slide lists VideoEditingTool, DocsSearchTool, composition sampling at one frame per second, and VisualFeedbackTool deciding whether to render or refine.](https://ai.engineer/api/talk-essay-media/fbb1b6b4d583a80a66a68a25657a1c37b7aa0976826970b95fc36a922354d655)

The editing workflow combines code generation, documentation retrieval, and visual feedback.

### Supply documentation and prompt templates

[3:53](https://www.youtube.com/watch?v=r0AG44qYKsI&t=233s)

The team also shipped an `llms.txt` file and specific prompt templates to help agents use the editing environment. The on-screen example shows a documentation welcome section and terminology definitions. Those materials complement documentation search by giving the agent a prepared starting point for understanding the library.

Muhtesem calls `llms.txt` the equivalent of `robots.txt` for agents. The useful distinction is their purpose: [`llms.txt`](https://llmstxt.org/) supplies curated context and documentation links, while `robots.txt` communicates guidance about automated access. For this editor, the relevant combination is library context plus task-specific prompting.

![Slide titled “The /llms.txt file” beside a documentation excerpt with a welcome section and terminology definitions.](https://ai.engineer/api/talk-essay-media/118be2fca3c85903847ad69253eb0d7c22eefc5d4078a68c0536f778c1118c3f)

The /llms.txt file presents Diffusion Studio documentation for agents.

### Separate the agent from the browser it controls

[4:12](https://www.youtube.com/watch?v=r0AG44qYKsI&t=252s)

The browser runtime allows two deployment arrangements:

- **Bring your own browser:** run the agent against a browser session you provide.
- **Connect remotely:** attach through a WebSocket connection, with a separate GPU-accelerated browser session for each agent.

Muhtesem describes a load balancer behind the remote arrangement. This is a proposed deployment path rather than a demonstrated production service; the accompanying project description marks the remote WebSocket endpoint as work in progress.

The first agent implementation is in Python, with a TypeScript implementation underway at the time of the presentation. Muhtesem closes with a TypeScript variation on the familiar programming-language joke: any application that can be written in TypeScript eventually will be. That roadmap would bring the agent implementation into the same language ecosystem as the composition interface. The project is a collaboration between Diffusion Studio and Reskill.

## Resources

- [Video Composer Agent](https://github.com/diffusionstudio/agent) — Python video-editing agent repository with setup instructions and documentation-search examples.
- [Diffusion Studio Core](https://github.com/diffusionstudio/core) — TypeScript browser video engine with composition, transitions, animation and rendering examples.
- [VisualFeedbackTool source](https://github.com/diffusionstudio/agent/blob/main/src/tools/visual_feedback.py) — Image-sample review tool that checks editing goals and produces structured feedback and a render decision.
- [Executable Code Actions Elicit Better LLM Agents](https://arxiv.org/abs/2402.01030) — CodeAct research comparing executable Python actions with text and JSON action formats on tool-use tasks.
- [Remotion client-side rendering](https://www.remotion.dev/docs/client-side-rendering) — Current browser-rendering APIs, examples and limitations for Remotion's web renderer.
- [The llms.txt proposal](https://llmstxt.org/) — The revised proposal for publishing concise website guidance and documentation links for agents.
- [Attention Is All You Need](https://arxiv.org/abs/1706.03762) — The original Transformer paper introduced in the opening video snippet.

## Slides

### Video editing agent setup

The code equips an agent with documentation search, video editing, and visual feedback tools, then prompts it to compose a background, animated title, and logo.

[0:03](https://www.youtube.com/watch?v=r0AG44qYKsI&t=3s) [Code configures an agent with documentation, video editing, and visual feedback tools, then requests a background, animated title, and logo.](https://assets.ai.engineer/sites/aiecode2025/sha256/629d65b2025912d0f719050212125dd584ce3c9fae9e0324107d081cb584d222.webp)

client = DiffusionClient()

agent = CodeAgent(
    tools=[
        DocsSearchTool(),
        VideoEditorTool(client=client),
        VisualFeedbackTool(client=client),
    ],
    model=Anthropic(),
    system_prompt=get_system_prompt(),
)

agent.run("""
- Set raycast.png as the background of the composition
- Add a text in the top-left corner with the phrase "Attention Is All You Need", and
animate it
- Place logo.png in the top-right corner
""")

### Using the same session

A follow-up prompt asks the agent to generate captions from transcript.json and use podcast.mp3 as the audio source for a waveform visualization.

[0:26](https://www.youtube.com/watch?v=r0AG44qYKsI&t=26s) [An agent prompt requests captions from a transcript and a waveform visualization using podcast audio.](https://assets.ai.engineer/sites/aiecode2025/sha256/144266a89a22faf51131af5543925d64f3db732fbec069797d8e0dd71a36ab0d.webp)

Using the same session...

agent.run("""
- Generate captions from transcript.json.
- Use podcast.mp3 as the audio source and create a waveform visualization
""")

### New session

A new agent prompt lays out horizontal and vertical videos side by side, 50px apart, with rounded corners, shadows, and a full-frame background.

[0:45](https://www.youtube.com/watch?v=r0AG44qYKsI&t=45s) [An agent prompt requests side-by-side horizontal and vertical videos with rounded corners, shadows, and a background.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d80e2b142c4c203c81471ac43f713c049601083738a432a94ee5c7b57e6b5d3.webp)

New session...

agent.run("""
- Create a side-by-side video composition with horizontal.mp4 (16:9, left) and vertical.mp4 (9:16, right), separated by 50px.
- Apply rounded corners and a subtle drop shadow to both videos
- Set background.png as the full-frame background.
""")

### This video was edited with an AI agent

Muhtesem introduces an open-source video editing agent, using this video as the starting point for explaining how it works.

[1:05](https://www.youtube.com/watch?v=r0AG44qYKsI&t=65s) [Title slide asks how the video was edited with an AI agent, with a presenter inset in the upper right.](https://assets.ai.engineer/sites/aiecode2025/sha256/61fbefc998de5c01b8c6cd65236d89ffb1e118c552af6803c1ee9d8061015f20.webp)

This video was edited with AI agent. But how?

AI Engineer
SUMMIT

### How it started

The team needed automatic video editing for Reskill.io. After encountering FFmpeg’s limitations and Remotion’s reliance on server-side rendering, they chose Core for its client-side rendering.

[1:11](https://www.youtube.com/watch?v=r0AG44qYKsI&t=71s) [Slide compares limitations of FFmpeg and Remotion with Core’s client-side rendering.](https://assets.ai.engineer/sites/aiecode2025/sha256/217fcce99bbe24bd4591ac6a7062842716bb87a91d5bee69abcc38d874fad1e3.webp)

How it started?

ffmpeg - realized limitations, llms hallucinate advanced commands

remotion - complex architecture with many dependencies, also unreliable server side rendering

core - no need for rendering backend the cool thing about the library is that runs in client side

### The Agent

The agent writes its actions as code, using Core’s JavaScript/TypeScript interface for complex compositions. The speaker cites research favoring code-based tool calling over JSON.

[1:46](https://www.youtube.com/watch?v=r0AG44qYKsI&t=106s) [Slide describes an LLM writing actions in code using Core’s JavaScript and TypeScript interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b253d1297fbfbf623755b12cdfec3a1deba84321553b48e87b201980a072cdf.webp)

The Agent

LLM engine writes its actions in code

Core can do complex compositions via a JS/TS-based programmatic interface

Multiple research papers have shown that having the tool calling LLMs in code is much better

### Operator web app

The agent opens Operator with Playwright. The video editor is designed for AI agents and renders videos in the browser using WebCodecs.

[2:23](https://www.youtube.com/watch?v=r0AG44qYKsI&t=143s) [Slide describes a Playwright browser session and an agent-oriented video editor that renders with WebCodecs.](https://assets.ai.engineer/sites/aiecode2025/sha256/07d67cb6b8214adf3194f3f7b216bcacc42ce350d6ac12195b5cd1fdfe2edff6.webp)

https://operator.diffusion.studio/

Agent starts a browser session using Playwright

Operator web app is a video editing UI designed specifically for AI agents

Renders videos directly in the browser using WebCodecs

### How it works

VideoEditingTool runs generated code in the browser; DocsSearchTool retrieves context as needed. After each step, VisualFeedbackTool checks samples, currently at 1 frame per second, before the system decides whether to render or refine.

[2:57](https://www.youtube.com/watch?v=r0AG44qYKsI&t=177s) [Slide outlines code generation, documentation retrieval, visual feedback, and a decision to render or refine.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f3904b76fca510f0654d96430015b39883426fb35fec34f92f5bc6b424e8953.webp)

How it works:

• A VideoEditingTool generates code based on user prompts and runs it in the browser
• If additional context is needed, DocsSearchTool uses RAG to pull relevant info
• After each execution step, the composition is sampled (currently 1 frame per second) and analyzed using VisualFeedbackTool
• The feedback system decides whether to proceed with rendering or refine further

### The llms.txt file

Diffusion Studio’s llms.txt introduces the core engine and its terminology for agents. The speaker suggests pairing it with template prompts for video editing.

[3:51](https://www.youtube.com/watch?v=r0AG44qYKsI&t=231s) [Slide shows a sample documentation file introducing the Core engine and its terminology.](https://assets.ai.engineer/sites/aiecode2025/sha256/6b264d34dc95ea3016b50bf7a30fb991666e9a7f0f2a3f32445ba0e96153c610.webp)

The /llms.txt file

# Diffusion Studio Documentation

---

# 00-welcome.md

Getting started with the core engine

## Welcome

`@diffusionstudio/core` is a 2D motion graphics and video rendering engine powered by WebCodecs. Developers commonly use it for video editing automations and to build editing [playgrounds/web apps](https://playground.diffusion.studio).

## Terminology

To better understand the documentation, here are some key terms you should know:

- Composition: The root object that contains all assets to be rendered in a single video.
- Track: A chronological sequence of clips of the same type (e.g., video clips).
- Clip: An object that contains specific rendering information, such as position and duration of visibility within the video.
- Framerate: The rate at which frames are displayed, measured in frames per second (FPS). If not specified otherwise, Diffusion Studio uses a default frame rate of 30 FPS.
- Encoder: A tool that compresses the size of each frame within the sequence, using common codecs such as H.264/AVC1 or H.265/HEVC.
- Renderer: An abstraction responsible for creating and managing the rendering context (WebGL or Canvas), drawing the composition state to the screen.
- Ticker: A tool that ensures the composition state is updated at the correct frame rate and rendered to the screen.

### Scalable architecture

The CPU-based agent connects over WebSocket through a load balancer and Express proxy to GPU-accelerated operator instances. Each agent can receive a separate remote browser session.

[4:09](https://www.youtube.com/watch?v=r0AG44qYKsI&t=249s) [Diagram connects a serverless agent through a load balancer and proxy to GPU-accelerated operator instances.](https://assets.ai.engineer/sites/aiecode2025/sha256/9cf876792c47f74f1b9e2225c9b1def5b50e3621ebdfb094ca3c786658ad7989.webp)

Yes, it’s scalable!

FastAPI
Async Agent

Serverless Function (CPU)

ws connection
LB
express
proxy
operator.diffusion.studio
...
proxy
operator.diffusion.studio

GPU Accelerated

### Agents in TypeScript?

The first agent is written in Python; a TypeScript implementation is underway.

[4:36](https://www.youtube.com/watch?v=r0AG44qYKsI&t=276s) [Slide titled “Agents in TS?” with a quotation predicting that applications will eventually be written in TypeScript.](https://assets.ai.engineer/sites/aiecode2025/sha256/3415f53defbd3c8f0039a7ce1b8514ec99f7d8ab75a2645538af4fd81ec2d2ba.webp)

Agents in TS?

“Any application that can be written in TypeScript, will eventually be written in TypeScript.”

### Thank you

Diffusion Studio and Re-Skill collaborated on the project. The closing slide links to the agent repository and Re-Skill's website.

[4:53](https://www.youtube.com/watch?v=r0AG44qYKsI&t=293s) [Closing slide thanking the audience and naming Diffusion Studio and Re-Skill, with project and website links.](https://assets.ai.engineer/sites/aiecode2025/sha256/9de4f34f6177d87d199ea6af52a140c1f4272992499469daaf9cdf1c16463b59.webp)

Thank you!

Diffusion Studio x Re-Skill

https://github.com/diffusionstudio/agent

https://re-skill.io/

## Full transcript

[0:00](https://www.youtube.com/watch?v=r0AG44qYKsI&t=0s) [on-hold music]

[0:30](https://www.youtube.com/watch?v=r0AG44qYKsI&t=30s) Today, we're tackling a paper that's basically legendary, at least in the world of natural language processing. NLP, for those in the know.

[0:38](https://www.youtube.com/watch?v=r0AG44qYKsI&t=38s) Right. Attention is all you need. It's not just a catchy title.

[0:41](https://www.youtube.com/watch?v=r0AG44qYKsI&t=41s) Definitely not.

[0:51](https://www.youtube.com/watch?v=r0AG44qYKsI&t=51s) OCaml, a general-purpose functional programming language that is also an imperative language and also an object-oriented language. It is what Haskell wishes it could be.

[1:04](https://www.youtube.com/watch?v=r0AG44qYKsI&t=64s) Hey everyone. My name is Muhtesem, and I'm excited to talk about the world's first open source video editing agent.

[1:13](https://www.youtube.com/watch?v=r0AG44qYKsI&t=73s) Backstory is that we needed some automatic tool to edit videos for Reskill.io, a platform for personalized learning. While doing so, we quickly realized limitations of FFmpeg and started looking for more intuitive and flexible alternatives.

[1:28](https://www.youtube.com/watch?v=r0AG44qYKsI&t=88s) Remotion was nice, but it did rely on server-side rendering. After trying out Core, we really liked the API as it did not require the separate rendering backend. We met with the author of the library and decided to collaborate and build this agent together.

[1:47](https://www.youtube.com/watch?v=r0AG44qYKsI&t=107s) The Core library from Diffusion Studio can do complex compositions via JavaScript, TypeScript-based programmatic interface, meaning we can use LLM to generate code to run this. And if we take a step further and let our LLM write its own action in code, it's a perfect match, simply because code is the best possible way to express actions performed by

[2:12](https://www.youtube.com/watch?v=r0AG44qYKsI&t=132s) a computer. Lastly, multiple research papers have shown that having LLM tool calling in code is much better than in JSON.

[2:25](https://www.youtube.com/watch?v=r0AG44qYKsI&t=145s) Now, let's take a look at current architecture. Agent starts a browser session using Playwright

[2:33](https://www.youtube.com/watch?v=r0AG44qYKsI&t=153s) and connects to operator UI. This web app is video editing UI designed specifically for AI agents. It renders video directly in browser using WebCodecs API.

[2:47](https://www.youtube.com/watch?v=r0AG44qYKsI&t=167s) It also has helper functions for transferring files from Python to browser and back via Chromium DevTools Protocol.

[3:00](https://www.youtube.com/watch?v=r0AG44qYKsI&t=180s) This is a typical flow of agent. We have three main tools, VideoEditingTool, DocsSearchTool, and VisualFeedbackTool. First, a VideoEditingTool generates code based on user prompt and runs it in browser.

[3:15](https://www.youtube.com/watch?v=r0AG44qYKsI&t=195s) If additional context is needed, DocsSearchTool uses RAG to pull the relevant information. After each execution step, a composit- the compositions are sampled currently at one frame per second, and they are, they are fed to VisualFeedbackTool.

[3:34](https://www.youtube.com/watch?v=r0AG44qYKsI&t=214s) VisualFeedbackTool can be thought as a generator and discriminator, like in famous GAN architecture.

[3:43](https://www.youtube.com/watch?v=r0AG44qYKsI&t=223s) After the VisualFeedbackTool gives green light, the agent proceeds to render the composition.

[3:53](https://www.youtube.com/watch?v=r0AG44qYKsI&t=233s) We also shipped llms.txt, which is essentially robots.txt, but for agents. You can see sample in the screen. llms.txt, in addition with specific template prompts, will take you far in your video editing journey.

[4:12](https://www.youtube.com/watch?v=r0AG44qYKsI&t=252s) While, while you can bring your own browser and run the agent, the current setup is also flexible enough to let the agent connect to a remote browser session via WebSocket, and each agent can get a separate browser session, which is GPU accelerated.

[4:31](https://www.youtube.com/watch?v=r0AG44qYKsI&t=271s) And of course, there's a load balance, load balancer behind this.

[4:38](https://www.youtube.com/watch?v=r0AG44qYKsI&t=278s) Of course, the first version of the agent is in Python, but TypeScript implementation is underway. As the fame, famous saying goes, "Any applications that can be written in TypeScript will be written in TypeScript."

[4:54](https://www.youtube.com/watch?v=r0AG44qYKsI&t=294s) Thank you very much. This was collaboration between Diffusion Studio and Reskill.
