HTML Is All Agents Need

Talk transcript

1. 0:00

   [upbeat music] [singing] Get on in the car.

2. 0:22

   I done give the arm. Bust it open, huh. Let me see you jump. I fucked up his muscle when we hustle, bustin', what? The-

3. 0:36

   Since we're talking about generative media, I thought starting with a video might be good. That video was made completely by an agent in a single shot, all utilizing HTML.

4. 0:46

   My name is James Russo. I am the co-creator and tech lead of HyperFrames at HeyGen. Today, we're gonna be talking about why we think HTML is all your agents need to create great videos.

5. 0:58

   Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents.

6. 1:06

   All right. Some hands up there. Um, coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds. Um, however, we think that launching your product or your feature, getting it out into the world, is still quite hard, and we at HeyGen have been trying to close that gap.

7. 1:27

   If you're not familiar with HeyGen, our mission is to solve communication through video. We started by creating the best AI avatar on the market.

8. 1:35

   You can think of this as the A-roll, the footage you see here on the left side of the screen. Um, it's the narrator, the character, the main subject of the video.

9. 1:43

   However, it's kind of plain, and great videos have a lot more pieces to them. They have B-roll, which is the images and the, the other pieces of media and assets.

10. 1:53

    They have animations. They have captions. They have music. All of this is needed to create a great video. And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos.

11. 2:13

    So how do you give agents the ability to generate all of these layers and build up this composition needed to create a great video? Our bet is on HTML.

12. 2:23

    HTML, CSS, and JavaScript are the native languages of LLMs. Most of their training data, every webpage that gets scraped at the end of the day, is essentially just HTML, CSS, and JavaScript under the hood.

13. 2:34

    Uh, when you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language. Uh, we like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese.

14. 2:49

    Even if you give them a bunch of examples, um, and teach them a bunch of things, you will not get the best output from them because it's not their native tongue.

15. 2:57

    So why not let the LLMs and agents talk in their native tongue when creating videos?

16. 3:04

    We're not the only ones saying this. Over the last few months, Tariq, Andrej Karpathy have been talking a lot about how HTML is the new markdown. It's a great output, um, for LLMs to give you visual representation.

17. 3:16

    For what it's worth, I submitted this talk before both of these tweets came out. Uh, I can show you proof later if you're curious. But we all kinda came to the same conclusion here, which is that, uh, let LLMs talk in their native language, and you can get better output from them.

18. 3:34

    It wasn't necessarily an easy thing. We've been working on this for over a year now and tried a bunch of things along the way. Uh, there's a bunch of different, uh, frameworks and tools to help create great videos, and we like to think about them, uh, on this axis of quality of results and agent-friendly.

19. 3:52

    Um, for us, After Effects, Premiere Pro, these are kinda like the gold standards of creatives. They create great output, but they're not very agent-friendly, and even with the more recent connectors into it, it's more of a, a co-pilot or something that can help you facilitate things you already know how to do.

20. 4:10

    It doesn't give it creative output. We tried things like Lottie and Rive, which are coding, uh, languages in JSON or custom XML formats, um, and they can get you pretty good output.

21. 4:23

    But agent-friendly isn't necessarily true for those because they're not, um, their native tongue, and they aren't as editable or controllable, which is a big thing for us, is that controllability layer.

22. 4:36

    We played around with Remotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding. Um, however, we noticed that we had to teach them the framework.

23. 4:48

    We had to teach them the language and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it.

24. 4:57

    Um, then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do.

25. 5:07

    When we just gave them examples of what the output we wanted was, they naturally gravitated to HTML, CSS, and JavaScript and gave us great output. Um, and we decided, let's not fight the model, but find a way where we can let them talk in their native tongue.

26. 5:24

    So how we did this was starting with a very small model, Gemini 3 Flash, as our design partner. We knew that if the smaller models could author workable code in a framework, then the larger models and these coding agents could 100% do it as well.

27. 5:42

    We tried with a bunch of different wrappers around HTML, CSS, and JavaScript, adding in a lot of context and m-making our system prompts bigger, adding in skills. Um, but to our surprise, the thinnest wrapper ultimately won, which is essentially just HTML at the end of the day with a few data attributes as metadata to let the agent

28. 6:01

    know and to let us know, uh, about timing and things like that. Um, our-- Yeah, as I already mentioned, our thinking was very simple. If this smaller model could get it, as the models got better, it would continue to improve.

29. 6:13

    They would naturally understand it more as well as training data came into the picture. Um, and this is when we knew the format of what we were building was right.

30. 6:24

    Here's a little bit more detailed example of what a video looks like, uh, in HyperFrame. So here is on the left, just HTML, um, with a few attributes, and on the right, the webpage that it renders.

31. 6:37

    Um, the preview and the render are all done in the browser. The same pixels that the browser sees is ultimately what your video is going to see. Um, and anything-- any webpage that your LLM or agent knows how to write, it can now write into a video.

32. 6:54

    This gave us HyperFrames, our open source framework that turns your agent's HTML into video.

33. 7:03

    So now that we had the HTML and the language part of it, um, the next hard part was how do we actually turn this into a deterministic MP4 video that anyone can post or utilize?

34. 7:14

    And this was a lot harder because browsers are async on purpose. Um, they have a different set of requirements and concerns to video rendering. They need to work across a bunch of different networks and a bunch of different, uh, constraints, but that's not true for video.

35. 7:30

    So on the left, you can kinda see here what a, a browser renderer might do. It's okay to loading things asynchronously, like fonts. So as the page loads in, the font and style of the text might change.

36. 7:42

    Images and videos and other assets can load in asynchronously as well. Um, so you might not see them initially on the page, but they'll come in at some point.

37. 7:51

    Whereas for video, we need everything on the page a hundred percent of the time so that we can show you exactly what you expect in the, the video.

38. 8:01

    How we do this in HyperFrames is, um, essentially freezing the clock and seeking frame by frame. For those who aren't super familiar with video, uh, basically a video is a series of frames or images strung together to create motion across an entire video.

39. 8:18

    So we took this insight of ours and basically applied it to our rendering of HTML as well. We freeze the clock in the browser, and then we seek deterministically to every single moment in time or every single frame, uh, wait for everything to load on the page, ensure that it's loaded and ready to go, and then we

40. 8:38

    take a screenshot and move on to the next frame and do that over and over again, uh, until we get all of the necessary frames to encode that into a video.

41. 8:46

    So the same input that is previewed in the browser is also rendered into the video. Hence the name HyperFrames.

42. 8:56

    The power of this is that anything you can render in a browser essentially can now be in your video. Things like Three.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie, all of these are renderable in the browser and therefore all of them are renderable in HyperFrames.

43. 9:13

    This is a big part of how we create a lot of our videos, is finding inspiration on the internet, taking these as examples, tweaking them to our needs, and then putting them into our videos.

44. 9:25

    The next part outside of the actual framework and rendering is how do you get great or good output out of the agents? And a big part of this is the skills that we couple with our framework.

45. 9:37

    Our skills focus on taste and video aspects. Because the LLMs and agents already know how to write HTML and CSS and JavaScript, we don't have to teach them the language.

46. 9:47

    We just taste them how to create good videos. This is a big difference between other frameworks where if you look at certain skills, it's really just like, how do you write something in that framework?

47. 9:58

    Um, this allows us to focus on the important parts of what makes a great video, and a big part of this is constantly evaling and using agents to improve them.

48. 10:07

    Uh, this allows us to raise the floor of videos and ensure that the base output of a single shot prompt gets you pretty good results.

49. 10:18

    Here's an example I'm gonna show of a single shot utilizing our website-to-video skill. All you have to do is give it a website. It-- We tell it how to go to that website, retrieve all the assets and information it needs, uh, to create that design and theming and branding using Design MD or Frame MD in our case.

50. 10:38

    Um, and then from there, we just teach the LLM basic motion examples and things like that of great videos that you can then go ahead and get good output like this.

51. 10:54

    However, great output takes craft. Similar to AI coding, you can get decent output by just giving a single prompt and having something that works for your needs, but getting great output from agents requires craft, taste, the same principles of any, uh, software engineer before AI coding as well, breaking the problem up into individual pieces and working iteratively.

52. 11:18

    The same is true for getting great output for HyperFrames videos. Um, we wanna make sure our, our power users have full control over this, and the same way we created videos before HyperFrames is how we create them with HyperFrames.

53. 11:31

    We think about the narrative and the vision and the mission of this video. We storyboard it frame by frame and think about what each frame needs to do. We then move that into adding motion frame by frame utilizing HTML, CSS, and JavaScript, merging it into one cohesive video, um, and then utilizing our studio that we released with

54. 11:51

    HyperFrames for that last mile editing, ensuring that humans are always in the loop and have access to do anything that they would do in their normal video editor within the open source frameworks editor so that you can manually drag, tweak, um, et cetera.

55. 12:06

    And that gets you output that looks like this. Uh, we actually just released this today, keyframes in HyperFrames, uh, which is a big aspect of what makes great videos for motion designers, and it allows you to basically coordinate all of this different motion, uh, frame by frame or keyframe by keyframe, um, in our studio and make sure

56. 12:25

    that you can do anything that a professional motion designer might be able to do in After Effects.

57. 12:35

    So it's not just a demo. We've been... Uh, we released HyperFrames a couple of months ago. It's been running at scale, in our opinion. Over 1.3 million videos have been rendered by open source users of HyperFrames in the last 90 days.

58. 12:47

    267,000 creators have tried it. We have about 15,000 videos every single day being rendered utilizing the open source framework and 32,000 GitHub stars. Um, but this is only just the start for us.

59. 13:00

    As I mentioned already, it's open source, free forever. Anyone can go ahead and use HyperFrames right now. It works with any coding agent that you have, Claude Code, Codex, Cursor, any number of other ones up here, any ones I'm not aware of even.

60. 13:13

    If your agent knows how to write HTML, CSS, and JavaScript, it knows how to create a HyperFrames video. Now, with this, the same agent that is helping you create your product can also help you make your launch video.

61. 13:28

    The one honest thing that we're gonna say here is that the models still aren't good at creative work. We spend a lot of time evaling and trying to improve this in our skills and push it even further.

62. 13:38

    Um, but we think there's something at a higher level that needs to change here, which is why we've started to work on a code-to-video benchmark where we are trying to work with, uh, the LLM labs, any creators, uh, who are working on video agents to ensure that we can raise the floor of videos for everyone.

63. 13:56

    Uh, if anyone is interested in this space or working on this, we'd be happy to be collaborators, talk more about this. Feel free to find me or any member wearing a HyperFrames T-shirt in the crowd afterwards.

64. 14:05

    I'd love to chat to you guys more.

65. 14:09

    And then one more thing. Uh, some of you may have seen the AI Engineer World Fair showcase video. Uh, we collaborated heavily with the team to create this with them utilizing HyperFrames, um, as a little surprise of what it can do.

66. 14:22

    Here's a great example of the motion graphics and designing that you can do with HTML, CSS, and JavaScript.

67. 14:31

    So yeah, uh, agents have made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need in order to make great videos and launch your product into the world.

68. 14:42

    Here is a link to HyperFrames, the project, the open source repo. Feel free to check it out, star it, download the skills. You can reach me on X [REDACTED:username].

69. 14:50

    And thank you for attending. I'll be outside if anyone wants to chat. [audience applauding] [upbeat music]