SOTA Generative Media Panel — Dumitru Erhan, Shane Gu & Nicole Brichtova, Google DeepMind

Adding a sloth to a podcast is a good demo. Doing it automatically across a collection of recordings requires an API. That distinction opens swyx’s conversation with Google DeepMind’s Dumitru Erhan, Shane Gu and Nicole Brichtova: what changes when generative media becomes something developers can put inside a workflow?

Brichtova begins with two releases described at the session. Nano Banana 2 Lite is the fastest, cheapest image model in the Nano Banana family, offering generation and editing that she describes as better than the original Nano Banana and close to the larger models’ quality. The useful quantity is the roughly three-second latency she describes: a short wait lets someone try an idea, inspect the result and revise it while the idea is still fresh. Some outputs can also serve as production assets, so the iteration tool and the final-output tool need not be separate.

The second release is the Gemini OmniFlash APIs, making video generation and editing available to developers following their pre-announcement at I/O. Brichtova says they are priced the same as Veo 3.1 Fast at the time of the session. For swyx, the immediate value is automation: the animal additions shown in an earlier podcast become a repeatable operation rather than something he can only admire in a first-party demo. The panel then moves from those playful edits to the less conspicuous work that could make the models useful every day.

Omni’s two central capabilities change different parts of media production:

• Generation from references: Storyboard images can supply the visual plan, while an audio track supplies a reference voice for a character. Those inputs lead to a generated video. Other output modalities are a direction the team wants to pursue, rather than a capability established here.
• Editing an existing recording: Natural-language instructions can ask for an addition, a removal or a cleanup. A noisy beach-vacation video is the practical counterpart to the sloth demo: someone can request the desired change without first knowing which editing tool performs it.

Marketing campaigns and educational materials already appear among the uses Brichtova describes. Education offers a further possibility: adapting an explanation to a learner’s knowledge and preferred style. Making educational media is an observed use; a fully personalized learning experience is the larger ambition that this capability could help support.

Erhan gives a concrete example of a selective edit. His visiting parents wanted to use a gadget whose instructions were in English. He photographed the instruction sheet, including its diagrams, and asked Nano Banana to translate it into Romanian while keeping everything else the same. The input image supplied the layout and drawings; the instruction identified the text as the part to change; Gemini supplied the translation under the hood. He describes a result that looked identical with approximately correct translated text. The observable success was preserving the illustrated document while making its instructions readable to his parents.

That example makes text rendering a functional requirement. The diagram and its wording belong together: translating the words separately would leave the reader to reconstruct their relationship to the original page. Erhan extends the same idea to video translation and redubbing, where access improves when the model can alter language while retaining the surrounding media.

The next question is whether a video system should do everything in one pass. swyx introduces video agents as an alternative: let several operations cooperate rather than require a single generation to finish the whole task. Gu’s interest is in connecting symbolic reasoning with a video foundation model that carries information about space and time.

Language helps make that connection. Detailed captions describe more than a list of visible objects; they can describe what happens and how the scene comes about. Gu offers a hypothesis about spurious correlations—the risk that a model predicts from an associated feature rather than the factor that actually explains an outcome. Two routes could help:

• Diverse training data: Examples spanning interventions on the world’s causal relationships can separate features that merely occur together.
• Conditioning on explanatory information: A caption or prompt can supply information about the process behind the image or video, giving the model a more informative condition for generation.

This is a proposed explanation for language’s usefulness, not a guarantee that caption conditioning identifies causality.

A video model may also contain useful capabilities beyond producing finished clips. Gu names the evaluation work Video Models, Zero-Shot Learners and Reasoners, followed by the Nano Banana team’s Vision Banana work. The idea is to apply generative models to computer-vision tasks, visual quizzes and robotics scenes without first training them specifically for each task. His interpretation is that learning space-time structure gives video models physical intuitions that can complement text reasoning. He also acknowledges that the quiz-solving ability has substantial room to improve.

Gu expects eventual unification, but the immediate opportunity is cooperation among strong understanding and generation capabilities. Erhan adds the engineering reason to keep specialized models: Nano Banana 2 Lite and a model intended for something like a 4K, thirty-second video have different training and product demands. Putting both into one checkpoint could compromise a useful niche. His six-month and five-year comparisons are forecasts, not a release schedule.

Omni’s name points toward fully multimodal input and output, while specialized models remain useful along the way. Unification also depends on transfer: learning images and video plausibly helps both, and audio belongs with the events visible in video. Transfer between coding, 3D representations and video generation is less settled in this discussion. Combining tasks only earns its cost if learning them together helps the capabilities people actually need.

Captioning raises a deeper design question: what should sit between a request and the media that fulfills it? swyx finds English an inefficient way to specify everything across time and suggests code as another possibility, pointing to FFmpeg, Matplotlib and Manim as examples of tools that produce visual material through programs. Gu confirms that coding representations are being explored, without choosing them as the answer.

Gu connects this to the question of why reasoning should use natural-language chain of thought. Additional computation could instead happen through continuous tokens or another internal representation. His defense of language is about reusing pretraining: the model has learned extensively through text, so reasoning in text can draw directly on those learned capabilities. An unconstrained internal representation must find another way to acquire or use that structure; extracting it through reinforcement learning can be expensive. His earlier Large Language Models are Zero-Shot Reasoners provides a concrete research counterpart to this interest in eliciting capabilities already present in a pretrained model.

There is also a product reason to retain language: people use it to communicate their intentions. A text interface makes text a natural default even if the model’s full representation is richer. That does not require every aspect of the world to pass through a caption. When swyx describes language as a narrow, lossy pipe, Gu explicitly clarifies that the proposal is to use language and video together.

Returning to the earlier discussion of world models, the term needs a similarly specific meaning. Erhan jokes that everything now gets called one. Gu’s preferred definition is the model used in model-based reinforcement learning: the modeling component used to reason about the world. For him, video is a complementary foundation for the spatial and temporal information language alone cannot adequately carry. Using that foundation to reach human-level general intelligence is his research vision, rather than a capability demonstrated by the current products.

The researchers’ move from recognition to generation helps explain the remaining difficulty. Recognizing a picture as a cat compresses many possible appearances into one label. Going from “cat” to an image must choose among those appearances. Generation is therefore a more ambiguous mapping than simply reversing recognition. Better understanding can still improve generation, and Gu sees reinforcement learning as a bridge through which the ability to judge an output can help improve the process that creates it.

Gu draws a possible development path from language models: creative demonstrations became usable chatbots through instruction tuning; better instruction following and fewer hallucinations then made reasoning and reinforcement learning more useful. He hopes video follows a comparable path toward reliable space-time simulation that can work alongside text reasoning. Erhan identifies an unfinished step: frontier multimedia models are not yet commonly strong at both understanding and generation of video. Their conceptual relationship does not by itself establish that one model already does both well.

Audio makes the case for joint modeling unusually concrete. Erhan describes Veo 3 as generating audio and video together. The reason is causal: a person speaking produces both moving lips and sound. Treating those as consequences of one underlying event gives the model a reason to keep them synchronized.

The earlier approach he contrasts with it generates the pixels, generates audio and then adds a mechanism to move the lips to match. Synchronization becomes a repair applied to outputs that were created separately. Joint generation instead addresses that relationship inside the generative process. Erhan credits this choice with solving many of the synchronization problems of earlier approaches.

What relationship does joint generation preserve? The diagram shows the panel’s conceptual explanation: one speaking event produces two coordinated outputs. The connection between lips and sound comes from their shared cause; it need not be invented after the pixels are complete.

Joint generation does not settle how to describe the desired sound. Gu points to music, vocal tone and pitch as sensory information that is difficult to verbalize. Taste, smell and subtle differences in skin color raise a similar question: people can perceive distinctions they struggle to name. He recounts a professional wine taster borrowing language used to describe a romantic partner to communicate taste. His proposed connection to survival and evolutionary sensitivity is speculative; the practical issue is the gap between what someone notices and what they can specify.

Brichtova extends the problem to visual style. An expert may see a difference that an ordinary user cannot describe, and even existing terms such as prosody or disfluency require domain knowledge. References help by supplying the desired quality directly. A voice sample can carry information that a user’s description omits. A language model’s gaps can also propagate into generation when the description depends on its understanding of the domain.

Earlier in this discussion, Erhan had resisted turning these shortcomings into a fundamental ceiling. The audio problems he sees look addressable with more or better data; he does not think the team has exhausted that route. Brichtova agrees that some apparent representational limits may be areas the models have not yet learned well enough. The discussion leaves both possibilities open: language can omit important detail, while better training can improve how much detail the system understands and uses.

swyx makes the missing detail audible through a podcasting example. Voice in a large room, a car or a phone call has a different character. He identifies uniformly studio-quality sound as a tell of generated video and suggests studio recordings in training data as a cause. That is an observation and proposed explanation, rather than a demonstrated account of the model’s data. His stronger test is spatial: if a person moves farther away, their voice should become softer or more diffuse. A convincing audiovisual scene must change the sound when the depicted relationship changes.

Gu translates this into a conditioning problem. An underspecified caption permits many valid outputs: the same words could describe speech nearby or far away, in a reverberant room or a dry studio. Here “multimodal” means that several possible outcomes remain under the condition, not simply that the model handles several media types. Drawing on latent-variable modeling, he wants the representation to capture most of that variation so that the output, (X), becomes close to determined given the representation, (Z). Richer captions and richer non-language representations reduce how much the generator must guess.

Faces expose the same difficulty over time. Skin texture, nodding and changing microexpressions tell another person how someone is responding. Brichtova sees major progress but continued headroom in these details. A still image can look convincing while a sequence of expressions reveals that the system has not captured the small, coordinated changes humans read during a conversation.

The team ran an experiment that complicates the meaning of quality. They described real videos, used those captions to generate equivalent videos with Omni and asked humans to compare the originals with the generated versions. Erhan reports that people largely preferred the generated videos, by a large margin. He immediately narrows the result: the preference does not establish greater realism or usefulness, and he does not accept reinforcement learning as the confirmed explanation.

The visible changes explain why the comparison can reward the wrong property. The generated version can be sharper, more HDR-looking and more flattering in its skin tones. Those qualities make it attractive at a glance. Deeper inspection may reveal that it does not solve the original task any better. Asking whether someone likes a clip collapses appearance, realism and task success into a single answer, even though those qualities can disagree.

Expert sensitivity changes the judgment. Gu describes a manga artist who finds small errors in eye gaze disturbing. Brichtova connects this to expertise developed through years of craft: an artist or architect can notice faults an average viewer misses. The panel’s shorthand for the appeal of overly smooth, saturated content is the “Instagram filter.” Gu’s advice is to keep developing sensitivity and prompting for the differences that matter. Brichtova pairs that with a model requirement: instruction following should let people move away from the default through language or references.

Defaults still affect everyone who does not override them. Brichtova recalls Nano Banana infographics whose default style became too cluttered: the model behaved like an overeager student, trying to fit everything it knew into one image. With Omni, the team explicitly compared muted and saturated styles and chose color palettes during tuning. Those decisions often fall to the modeling team, raising a real product question about whether an art director or other creative expert should help set them.

Feedback catches changes the developers may overlook. A checkpoint optimization can blur the grass that a regular user cares about. Another tester noticed a tendency to add wedding rings to hands, which Erhan had not noticed himself. Gu suggests spurious correlations in preference-based optimization as a possible reward-hacking mechanism: a feature associated with preferred outputs could become a repeated decoration. The cause of the rings remains unknown in the discussion. The established failure is simpler: an unwanted pattern persisted until someone with a different viewing habit pointed it out.

How do you evaluate video? “Painfully.” The short answer earns its humor from the breadth of the task. Automated raters struggle with aesthetics, while free-form video editing permits an enormous range of requests. A suite can test known capabilities, but it cannot casually enumerate every possible addition, removal or audiovisual change.

The evaluation methods supply different kinds of signal:

• Objective asset checks: OCR can inspect rendered text in an infographic. A malformed letter is a concrete defect that may make the asset unusable, so this is more amenable to automatic rating than aesthetic taste.
• Human comparisons: The team evaluates thousands of items. When candidate models are close, decisions can still involve about 10 people in a room comparing videos side by side. That room is one decision setting, not the scale of the entire evaluation program.
• Live experiments: Larger-scale use provides signal about minute differences that smaller comparisons may miss.
• Expert and workflow feedback: Early-access users can reveal that a model which performs well on an evaluation slice nevertheless breaks a daily workflow.

Adding capabilities therefore adds evaluation work. The joke about needing an “add a sloth” evaluation identifies a genuine coverage problem: successful editing is defined by the particular change someone requested. A pleasant-looking result can fail that request, and a broad preference score may conceal the failure.

Gu wants human evaluation effort to become reusable through better video understanding. A generated reconstruction of a movie scene may look polished while containing semantically inconsistent events or details. Detecting that inconsistency is an intelligence task beyond judging aesthetics. His aim is to amortize human labels: use the work people do identifying failures to improve systems that can recognize those failures across future outputs.

The data question starts with familiar offers: gaming footage, real-world video, images and labeling services. Erhan avoids prescribing a detailed shopping list that would disclose the project’s direction. For Omni, he identifies high-quality material, including professionally shot footage, as useful. The harder opportunity Brichtova describes is data about what people are actually trying to accomplish.

Consider her marketing-campaign example. A user begins with a picture of a new product, turns it into a video ad and then produces assets for different advertising formats and platforms. The final collection reveals the deliverables. It does not reveal the sequence of requests, rejected candidates and revisions that made those deliverables acceptable. That trajectory lives inside the tools where the work happens, and neither the model provider nor a data vendor necessarily has access to it.

What disappears when training data contains only the finished campaign? The diagram separates the asset progression from the human choices along the way. A product picture can become a video and then several formatted assets, but the choice of one candidate over another carries additional information—perhaps the eye gaze was correct. Capturing that choice and its reason could teach a system what success means within the workflow.

This is why craft cannot be recovered from a pile of outputs alone. Brichtova emphasizes the time spent iterating and selecting. Gu describes tacit knowledge held by people: the starting point, inspiration and conversations behind a paper or a creative work. Public artifacts preserve the outcome more readily than the path. Brichtova connects this to generated prose that falls into similar patterns and descriptions; producing plausible language does not ensure the personal connection that makes a story matter.

The ending gives that missing process a possible route back into model development. Gu describes increased investment in forward-deployed engineers at Google Cloud and Google DeepMind, including creative generative-media work. swyx wants FDEs to contribute to evaluations as well as solutions. Gu gives the role a broader scope: helping customers use models, deriving insights from that use and returning those insights upstream to the modeling teams.

Gu uses an unusually broad personal definition of post-training: anything between pretraining and the final user experience. Within that definition, an FDE can improve both the customer’s harness for using the model and the provider’s understanding of what the model must learn. The practical connection is customer feedback reaching modeling, rather than remaining a local workaround in one integration.

Brichtova supplies two failures from conversations with users, and Erhan adds a third requirement from branded media:

• Rug-pattern replication: An interior-design user wants one pattern reproduced across 10 rug sizes, including custom dimensions. A model can produce an attractive rug while failing to replicate the pattern consistently across those sizes.
• Earring try-on: The earrings’ size must make sense relative to the wearer’s head. A convincing portrait is insufficient if the product’s scale is wrong.
• Brand language: Erhan’s IKEA example requires more than arbitrary blue and yellow. Brichtova sharpens the requirement by asking which shade of blue. Brand guidance may arrive as images or PDFs rather than a concise verbal rule, but those specific visual choices still determine whether an asset fits the brand.

These examples make the desired evaluation concrete. The instruction-sheet edit needed the diagrams retained and the language changed. Rug designs need a pattern retained across dimensions; try-on needs the relationship between product and person preserved; branded media needs specific colors and visual choices. Talking to the people doing those tasks exposes requirements the modeling team may never encounter in its own use. A beautiful result can still fail, and detailed customer feedback explains why.

swyx closes by calling for broader exploration of generative models beyond coding. The panel’s enthusiasm for coding remains intact; the appeal is to give images, video and audio comparable attention. The workflows discussed here provide concrete places to put that effort: translating illustrated instructions, editing recordings, preserving product scale and carrying a brand’s visual language through a campaign.