This video was edited with AI agent. But how?

Talk transcript

1. 0:00

   [on-hold music]

2. 0:30

   Today, we're tackling a paper that's basically legendary, at least in the world of natural language processing. NLP, for those in the know.

3. 0:38

   Right. Attention is all you need. It's not just a catchy title.

4. 0:41

   Definitely not.

5. 0:51

   OCaml, a general-purpose functional programming language that is also an imperative language and also an object-oriented language. It is what Haskell wishes it could be.

6. 1:04

   Hey everyone. My name is Muhtesem, and I'm excited to talk about the world's first open source video editing agent.

7. 1:13

   Backstory is that we needed some automatic tool to edit videos for Reskill.io, a platform for personalized learning. While doing so, we quickly realized limitations of FFmpeg and started looking for more intuitive and flexible alternatives.

8. 1:28

   Remotion was nice, but it did unreliable server-side rendering. After trying out Core, we really liked the API as it did not require the separate rendering backend. We met with the author of the library and decided to collaborate and build this agent together.

9. 1:47

   The Core library from Diffusion Studio can do complex compositions via JavaScript, TypeScript-based programmatic interface, meaning we can use LLM to generate code to run this. And if we take a step further and let our LLM write its own action in code, it's a perfect match, simply because code is the best possible way to express actions performed by

10. 2:12

    a computer. Lastly, multiple research papers have shown that having LLM tool calling in code is much better than in JSON.

11. 2:25

    Now, let's take a look at current architecture. Agent starts a browser session using Playwright

12. 2:33

    and connects to operator UI. This web app is video editing UI designed specifically for AI agents. It renders video directly in browser using WebCodecs API.

13. 2:47

    It also has helper functions for transferring files from Python to browser and back via Chromium DevTool protocol.

14. 3:00

    This is a typical flow of agent. We have three main tools, VideoEditingTool, DocsSearchTool, and VisualFeedbackTool. First, a VideoEditingTool generates code based on user prompt and runs it in browser.

15. 3:15

    If additional context is needed, DocsSearchTool uses RAG to pull the relevant information. After each execution step, a composit- the compositions are sampled currently at one frame per second, and they are, they are fed to VisualFeedbackTool.

16. 3:34

    VisualFeedbackTool can be thought as a generator and discriminator, like in famous GAN architecture.

17. 3:43

    After the VisualFeedbackTool gives green light, the agent proceeds to render the composition.

18. 3:53

    We also shipped llms.txt, which is essentially robots.txt, but for agents. You can see sample in the screen. Llms.txt, in addition with specific template prompts, will take you far in your video editing journey.

19. 4:12

    While, while you can bring your own browser and run the agent, the current setup is also flexible enough to let the agent connect to a remote browser session via WebSocket, and each agent can get a separate browser session, which is GPU accelerated.

20. 4:31

    And of course, there's a load balance, load balancer behind this.

21. 4:38

    Of course, the first version of the agent is in Python, but TypeScript implementation is underway. As the fame, famous saying goes, "Any applications that can be written in TypeScript will be written in TypeScript."

22. 4:54

    Thank you very much. This was collaboration between Diffusion Studio and Reskill.