AI Engineer World's Fair 2026
HTML Is All Agents Need
Read the talk
HTML Is All Agents Need
HyperFrames turns familiar web code into video by pairing a thin authoring format with controlled browser rendering, while leaving room for craft and human editing.
From a talk by James Russo
Before you start: Basic familiarity with HTML, CSS, and JavaScript will help with the animation example; no video-rendering experience is required.
The product is built. Now make the launch video.
The recording opens with a music video: a finished piece of generative media before any explanation of the tooling. James Russo says an agent made it completely in a single shot, using HTML. Introducing himself as the co-creator and tech lead of HyperFrames at HeyGen, he asks a practical question: who now finds making the launch post or launch video harder than building the product with a coding agent? Some hands go up. Software creation has become easier; communicating what that software does still takes work.
HeyGen approaches that gap through its mission of communication through video. Its avatar provides A-roll: the narrator, character, or main subject. But a speaking subject alone makes a fairly plain video. B-roll supplies images and other media; animation, captions, and music add the remaining layers. Giving an agent a useful video canvas means letting it compose all of these pieces, not merely generate the person speaking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the language the model already knows
HTML is the proposed canvas, with CSS for styling and JavaScript for behavior. Russo’s rationale is familiarity: models have substantial exposure to web code, so these languages give them an expressive starting point. He calls them the models’ native languages. A custom DSL or JSON structure adds a translation task before the agent can make anything visually interesting. His analogy is asking Shakespeare to write a poem in Japanese or Chinese: examples can teach the format without necessarily preserving the fluency of the original language.
This interest in HTML extends beyond video. Russo shows posts by Thariq and Andrej Karpathy discussing HTML as the new Markdown—a way for models to produce visual representations instead of only text. He says his talk submission preceded both posts. The useful design question is whether a familiar, expressive output format lets the model spend more of its effort on the result rather than on learning how to represent it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quality is only half the authoring problem
The team arrived at HTML after more than a year of experiments. Russo compares the alternatives along two axes: quality of output and friendliness to agents. His account distinguishes tools that help a knowledgeable human operate an editor from formats in which an agent can author a composition directly.
| Approach | What worked for the team | Where it became difficult |
|---|---|---|
| After Effects and Premiere Pro | High-quality creative output | Connectors mainly assisted existing editor expertise |
| Lottie and Rive | Promising visual results | Agent familiarity, editing, and control |
| Remotion | Strong examples of video through code | Framework instruction and examples consumed attention |
These are Russo’s observations from the team’s experiments. He groups Lottie and Rive with specialized JSON or XML representations, but Rive’s runtime format is actually binary .riv; its JSON core definitions are a separate implementation detail. The broader concern was how much representation-specific instruction the agent needed. With Remotion, Russo says teaching the framework and supplying examples ultimately constrained the creativity they were seeking.
The next change came with Gemini 3 and other newer models, around November 2025. Russo reports a step improvement in the team’s results: when given examples of the desired output, models naturally gravitated toward HTML, CSS, and JavaScript. That suggested a different engineering strategy—support the representation the models were already choosing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The thinnest workable HTML contract
The team used Gemini 3 Flash as a design partner. Flash followed the initial Gemini 3 introduction in December 2025. Their heuristic was to make the format workable for a smaller model, expecting more capable coding agents to handle it too. They tried wrappers around web code, more context, larger system prompts, and skills. The thinnest wrapper won: ordinary HTML with a few data-* attributes carrying metadata such as timing.
The demonstration places HTML on the left and its rendered webpage on the right. The browser handles both preview and final rendering, so the composition does not need to be translated into a separate visual language for export. The same HTML becomes the preview and the source of the video frames. HyperFrames packages that approach as an open-source framework for turning agent-authored HTML into video.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stop the browser clock before capturing the frame
Authoring the composition is only the first problem. Producing a deterministic MP4 is harder because browsers are deliberately asynchronous. A page can initially display fallback text, then change its typography when a font arrives. Images and videos can appear later still. Those behaviors are reasonable for an interactive page on a variable network; they are undesirable when a captured frame permanently records the incomplete state. Video needs every intended asset ready at capture time.
HyperFrames treats video as a sequence of still images and controls rendering accordingly:
- Freeze the browser’s clock.
- Seek the composition to the time of the next frame.
- Wait for the required assets and verify that the page is ready.
- Capture a screenshot.
- Advance to the next frame and repeat.
- Encode the collected frames into the video.
The renderer advances composition time explicitly instead of recording whatever the browser happens to display while wall-clock time passes.
A small browser example makes the seek operation concrete. This title moves across the screen over two seconds; setting currentTime selects its state without waiting for playback:
html
<!doctype html>
<meta charset="utf-8">
<style>
body { margin: 0; background: #101828; }
h1 { color: white; font: bold 64px sans-serif; }
</style>
<h1 id="title">Ready to launch</h1>
<script>
const title = document.querySelector('#title');
const motion = title.animate(
[
{ transform: 'translateX(0px)' },
{ transform: 'translateX(240px)' }
],
{ duration: 2000, fill: 'both', easing: 'linear' }
);
motion.pause();
window.seek = (seconds) => {
motion.currentTime = seconds * 1000;
};
window.seek(1);
</script>
At one second, the title is translated 120 pixels. This illustrates the animation side of the mechanism; asset readiness and screenshot capture belong to the renderer. Russo connects the HyperFrames name to using the same browser input for preview and video. That shared input reduces discrepancies, but reproducibility also requires controlled fonts, assets, randomness, and browser environment. His rendering companion article distinguishes controlled Linux capture from less robust screenshot heuristics on macOS and Windows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The browser becomes the visual toolbox
Using the browser opens up an existing ecosystem: Three.js, charts, SVGs, shaders, WebGL, WebGPU, and Lottie can all supply visual material. There is an additional requirement for animation, however. Content that owns its own clock needs a seekable adapter or prerendered frames so the renderer can request the right state at the right time. Browser compatibility supplies the pixels; controllable time makes those pixels usable in deterministic video.
That changes the creative workflow too. The team looks for inspiration on the internet, takes useful examples, adapts them to the composition, and incorporates them into videos. A browser effect can become a video element without first being rebuilt inside a conventional motion graphics editor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Teach motion, pacing, and taste
A rendering framework makes a video possible; it does not make the video good. HyperFrames couples the framework with skills focused on taste and video principles. Because the agent already knows web languages, the instructions can concentrate on motion, pacing, and craft instead of spending most of their space teaching framework syntax. The team repeatedly evaluates output and uses agents to improve those skills, aiming to raise the quality floor of a single prompt.
The website-to-video demonstration starts with only a website. The skill directs the agent to visit it, retrieve relevant assets and information, and capture its design, theme, and branding in Design.md, or Frame.md in HyperFrames’ workflow. Basic motion examples then guide how that material becomes an animated composition. The website supplies the visual identity; the skill supplies a process for turning it into a video.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From a usable first pass to a finished video
Single-prompt output can be sufficient, but polished work still requires decomposition and iteration. Russo compares this with AI-assisted software engineering: having something that works does not eliminate the need for judgment. HyperFrames preserves the production process the team used before the framework rather than replacing it with a single generation step.
Their sequence moves from intent to composition to editing:
- Define the narrative, vision, and mission of the video.
- Storyboard it frame by frame, deciding what each frame needs to accomplish.
- Add motion in HTML, CSS, and JavaScript.
- Merge the pieces into one cohesive video.
- Use HyperFrames Studio for the final manual adjustments, including dragging and tweaking elements.
The editor keeps humans involved after code generation, when a small visual adjustment may be easier to make directly than to describe in another prompt.
Russo then demonstrates keyframes, which he says were released that day. They let a creator coordinate different motions frame by frame or keyframe by keyframe in Studio. He positions those controls against the work a professional motion designer would do in After Effects: the goal is to retain detailed creative control alongside agent authoring.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use the coding agent already building the product
Russo says HyperFrames had launched a couple of months earlier and was already operating at what the team considered scale. He reports over 1.3 million videos rendered by open-source users in the preceding 90 days. He also reports 267,000 creators had tried it, about 15,000 videos were being rendered daily with the open-source framework, and the repository had 32,000 GitHub stars. These are usage figures reported at the talk, rather than measurements of rendering speed or creative quality.
The framework is open source under Apache-2.0, and Russo describes it as free forever. His promise concerns the project; it should not be read as a guarantee that every hosted service or generated media asset has no cost. He names Claude Code, Codex, and Cursor as compatible agents, with the broader requirement being the ability to write HTML, CSS, and JavaScript. That brings the workflow back to the opening problem: the agent helping build the product can also help make its launch video.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Creative quality remains an open problem
Russo’s closing qualification is direct: models still struggle with creative work. The team continues evaluating and improving skills, but it sees a need for progress beyond the framework itself. It has begun work on a code-to-video benchmark, seeking collaboration with model labs and people building video agents to raise the quality floor for everyone. This is work underway, not a presentation of benchmark results. Russo invites interested collaborators to speak with him or the HyperFrames team at the event.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A showcase, then the project handoff
The final demonstration is the AI Engineer World’s Fair showcase video, created with HyperFrames through substantial collaboration with the event team. It presents motion graphics and design built with HTML, CSS, and JavaScript, complementing the opening single-shot example with a collaboratively produced piece.
The closing returns to the gap between building and launching. Familiar web code gives the coding agent a route into video production; rendering infrastructure, skills, and editing make that route useful. Russo ends by pointing viewers to the project and open-source repository, inviting them to download the skills and try the workflow on the product they are already building.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source HTML-to-video framework with installation instructions, coding-agent skills and seekable animation workflows.
Technical specification for Rive's binary runtime format, including versioning and serialized objects.
Google's December 2025 announcement of the Flash model used in HyperFrames' authoring experiments.
Further reading
James Russo explains the experiments and browser-engineering work behind HyperFrames, including frame capture and preview parity.
The frame-by-frame rendering contract, readiness requirements and environment controls needed for reproducible output.
Read the complete timestamped transcript
- 0:00
[upbeat music] [singing] Get on in the car.
- 0:22
I done give the arm. Bust it open, huh. Let me see you jump. I fucked up his muscle when we hustle, bustin', what? The-
- 0:36
Since we're talking about generative media, I thought starting with a video might be good. That video was made completely by an agent in a single shot, all utilizing HTML.
- 0:46
My name is James Russo. I am the co-creator and tech lead of HyperFrames at HeyGen. Today, we're gonna be talking about why we think HTML is all your agents need to create great videos.
- 0:58
Show of hands here who finds creating that launch post or that launch video harder than actually building now with coding agents.
- 1:06
All right. Some hands up there. Um, coding agents have democratized building, made it incredibly easy for anyone to create anything that comes to their minds. Um, however, we think that launching your product or your feature, getting it out into the world, is still quite hard, and we at HeyGen have been trying to close that gap.
- 1:27
If you're not familiar with HeyGen, our mission is to solve communication through video. We started by creating the best AI avatar on the market.
- 1:35
You can think of this as the A-roll, the footage you see here on the left side of the screen. Um, it's the narrator, the character, the main subject of the video.
- 1:43
However, it's kind of plain, and great videos have a lot more pieces to them. They have B-roll, which is the images and the, the other pieces of media and assets.
- 1:53
They have animations. They have captions. They have music. All of this is needed to create a great video. And for us, it's important that we nail every layer, not just the avatar, to solve communication through video and give agents the right canvas to create great videos.
- 2:13
So how do you give agents the ability to generate all of these layers and build up this composition needed to create a great video? Our bet is on HTML.
- 2:23
HTML, CSS, and JavaScript are the native languages of LLMs. Most of their training data, every webpage that gets scraped at the end of the day, is essentially just HTML, CSS, and JavaScript under the hood.
- 2:34
Uh, when you try to teach a model a new DSL or even your own custom JSON structure, it's forcing it to speak another language. Uh, we like to think of this as trying to ask Shakespeare to write a poem in Japanese or Chinese.
- 2:49
Even if you give them a bunch of examples, um, and teach them a bunch of things, you will not get the best output from them because it's not their native tongue.
- 2:57
So why not let the LLMs and agents talk in their native tongue when creating videos?
- 3:04
We're not the only ones saying this. Over the last few months, Tariq, Andrej Karpathy have been talking a lot about how HTML is the new markdown. It's a great output, um, for LLMs to give you visual representation.
- 3:16
For what it's worth, I submitted this talk before both of these tweets came out. Uh, I can show you proof later if you're curious. But we all kinda came to the same conclusion here, which is that, uh, let LLMs talk in their native language, and you can get better output from them.
- 3:34
It wasn't necessarily an easy thing. We've been working on this for over a year now and tried a bunch of things along the way. Uh, there's a bunch of different, uh, frameworks and tools to help create great videos, and we like to think about them, uh, on this axis of quality of results and agent-friendly.
- 3:52
Um, for us, After Effects, Premiere Pro, these are kinda like the gold standards of creatives. They create great output, but they're not very agent-friendly, and even with the more recent connectors into it, it's more of a, a co-pilot or something that can help you facilitate things you already know how to do.
- 4:10
It doesn't give it creative output. We tried things like Lottie and Rive, which are coding, uh, languages in JSON or custom XML formats, um, and they can get you pretty good output.
- 4:23
But agent-friendly isn't necessarily true for those because they're not, um, their native tongue, and they aren't as editable or controllable, which is a big thing for us, is that controllability layer.
- 4:36
We played around with Remotion quite a bit and honestly thought it was a great example of what LLMs and agents could do with coding. Um, however, we noticed that we had to teach them the framework.
- 4:48
We had to teach them the language and give them a bunch of examples on how to write code properly, which ultimately took away a lot of the creativity of it.
- 4:57
Um, then we have HTML, which around November of last year when Gemini 3 and the latest models came out, we saw a step function improvement in what LLMs could do.
- 5:07
When we just gave them examples of what the output we wanted was, they naturally gravitated to HTML, CSS, and JavaScript and gave us great output. Um, and we decided, let's not fight the model, but find a way where we can let them talk in their native tongue.
- 5:24
So how we did this was starting with a very small model, Gemini 3 Flash, as our design partner. We knew that if the smaller models could author workable code in a framework, then the larger models and these coding agents could 100% do it as well.
- 5:42
We tried with a bunch of different wrappers around HTML, CSS, and JavaScript, adding in a lot of context and m-making our system prompts bigger, adding in skills. Um, but to our surprise, the thinnest wrapper ultimately won, which is essentially just HTML at the end of the day with a few data attributes as metadata to let the agent
- 6:01
know and to let us know, uh, about timing and things like that. Um, our-- Yeah, as I already mentioned, our thinking was very simple. If this smaller model could get it, as the models got better, it would continue to improve.
- 6:13
They would naturally understand it more as well as training data came into the picture. Um, and this is when we knew the format of what we were building was right.
- 6:24
Here's a little bit more detailed example of what a video looks like, uh, in HyperFrame. So here is on the left, just HTML, um, with a few attributes, and on the right, the webpage that it renders.
- 6:37
Um, the preview and the render are all done in the browser. The same pixels that the browser sees is ultimately what your video is going to see. Um, and anything-- any webpage that your LLM or agent knows how to write, it can now write into a video.
- 6:54
This gave us HyperFrames, our open source framework that turns your agent's HTML into video.
- 7:03
So now that we had the HTML and the language part of it, um, the next hard part was how do we actually turn this into a deterministic MP4 video that anyone can post or utilize?
- 7:14
And this was a lot harder because browsers are async on purpose. Um, they have a different set of requirements and concerns to video rendering. They need to work across a bunch of different networks and a bunch of different, uh, constraints, but that's not true for video.
- 7:30
So on the left, you can kinda see here what a, a browser renderer might do. It's okay to loading things asynchronously, like fonts. So as the page loads in, the font and style of the text might change.
- 7:42
Images and videos and other assets can load in asynchronously as well. Um, so you might not see them initially on the page, but they'll come in at some point.
- 7:51
Whereas for video, we need everything on the page a hundred percent of the time so that we can show you exactly what you expect in the, the video.
- 8:01
How we do this in HyperFrames is, um, essentially freezing the clock and seeking frame by frame. For those who aren't super familiar with video, uh, basically a video is a series of frames or images strung together to create motion across an entire video.
- 8:18
So we took this insight of ours and basically applied it to our rendering of HTML as well. We freeze the clock in the browser, and then we seek deterministically to every single moment in time or every single frame, uh, wait for everything to load on the page, ensure that it's loaded and ready to go, and then we
- 8:38
take a screenshot and move on to the next frame and do that over and over again, uh, until we get all of the necessary frames to encode that into a video.
- 8:46
So the same input that is previewed in the browser is also rendered into the video. Hence the name HyperFrames.
- 8:56
The power of this is that anything you can render in a browser essentially can now be in your video. Things like Three.js, charts, SVGs, shaders, WebGL, WebGPU, Lottie, all of these are renderable in the browser and therefore all of them are renderable in HyperFrames.
- 9:13
This is a big part of how we create a lot of our videos, is finding inspiration on the internet, taking these as examples, tweaking them to our needs, and then putting them into our videos.
- 9:25
The next part outside of the actual framework and rendering is how do you get great or good output out of the agents? And a big part of this is the skills that we couple with our framework.
- 9:37
Our skills focus on taste and video aspects. Because the LLMs and agents already know how to write HTML and CSS and JavaScript, we don't have to teach them the language.
- 9:47
We just taste them how to create good videos. This is a big difference between other frameworks where if you look at certain skills, it's really just like, how do you write something in that framework?
- 9:58
Um, this allows us to focus on the important parts of what makes a great video, and a big part of this is constantly evaling and using agents to improve them.
- 10:07
Uh, this allows us to raise the floor of videos and ensure that the base output of a single shot prompt gets you pretty good results.
- 10:18
Here's an example I'm gonna show of a single shot utilizing our website-to-video skill. All you have to do is give it a website. It-- We tell it how to go to that website, retrieve all the assets and information it needs, uh, to create that design and theming and branding using Design MD or Frame MD in our case.
- 10:38
Um, and then from there, we just teach the LLM basic motion examples and things like that of great videos that you can then go ahead and get good output like this.
- 10:54
However, great output takes craft. Similar to AI coding, you can get decent output by just giving a single prompt and having something that works for your needs, but getting great output from agents requires craft, taste, the same principles of any, uh, software engineer before AI coding as well, breaking the problem up into individual pieces and working iteratively.
- 11:18
The same is true for getting great output for HyperFrames videos. Um, we wanna make sure our, our power users have full control over this, and the same way we created videos before HyperFrames is how we create them with HyperFrames.
- 11:31
We think about the narrative and the vision and the mission of this video. We storyboard it frame by frame and think about what each frame needs to do. We then move that into adding motion frame by frame utilizing HTML, CSS, and JavaScript, merging it into one cohesive video, um, and then utilizing our studio that we released with
- 11:51
HyperFrames for that last mile editing, ensuring that humans are always in the loop and have access to do anything that they would do in their normal video editor within the open source frameworks editor so that you can manually drag, tweak, um, et cetera.
- 12:06
And that gets you output that looks like this. Uh, we actually just released this today, keyframes in HyperFrames, uh, which is a big aspect of what makes great videos for motion designers, and it allows you to basically coordinate all of this different motion, uh, frame by frame or keyframe by keyframe, um, in our studio and make sure
- 12:25
that you can do anything that a professional motion designer might be able to do in After Effects.
- 12:35
So it's not just a demo. We've been... Uh, we released HyperFrames a couple of months ago. It's been running at scale, in our opinion. Over 1.3 million videos have been rendered by open source users of HyperFrames in the last 90 days.
- 12:47
267,000 creators have tried it. We have about 15,000 videos every single day being rendered utilizing the open source framework and 32,000 GitHub stars. Um, but this is only just the start for us.
- 13:00
As I mentioned already, it's open source, free forever. Anyone can go ahead and use HyperFrames right now. It works with any coding agent that you have, Claude Code, Codex, Cursor, any number of other ones up here, any ones I'm not aware of even.
- 13:13
If your agent knows how to write HTML, CSS, and JavaScript, it knows how to create a HyperFrames video. Now, with this, the same agent that is helping you create your product can also help you make your launch video.
- 13:28
The one honest thing that we're gonna say here is that the models still aren't good at creative work. We spend a lot of time evaling and trying to improve this in our skills and push it even further.
- 13:38
Um, but we think there's something at a higher level that needs to change here, which is why we've started to work on a code-to-video benchmark where we are trying to work with, uh, the LLM labs, any creators, uh, who are working on video agents to ensure that we can raise the floor of videos for everyone.
- 13:56
Uh, if anyone is interested in this space or working on this, we'd be happy to be collaborators, talk more about this. Feel free to find me or any member wearing a HyperFrames T-shirt in the crowd afterwards.
- 14:05
I'd love to chat to you guys more.
- 14:09
And then one more thing. Uh, some of you may have seen the AI Engineer World Fair showcase video. Uh, we collaborated heavily with the team to create this with them utilizing HyperFrames, um, as a little surprise of what it can do.
- 14:22
Here's a great example of the motion graphics and designing that you can do with HTML, CSS, and JavaScript.
- 14:31
So yeah, uh, agents have made building incredibly easy. Launching is still quite hard. We think HTML is all your agents need in order to make great videos and launch your product into the world.
- 14:42
Here is a link to HyperFrames, the project, the open source repo. Feel free to check it out, star it, download the skills. You can reach me on X [REDACTED:username].
- 14:50
And thank you for attending. I'll be outside if anyone wants to chat. [audience applauding] [upbeat music]