Why AI Agents Need Million-Token Context — Thomas Wolf & Olive Song, MiniMax

A coding agent’s job can extend well beyond writing code. It may need to understand images or videos, use tools and carry information through a long interaction. In this conversation with Hugging Face co-founder and Chief Science Officer Thomas Wolf, MiniMax’s Olive Song introduces MiniMax M3 as a model built to bring those capabilities together.

Song describes M3 as having around 400 billion total parameters and 20 billion activated parameters. Those are different measures: the full model contains substantially more parameters than the activated count suggests. Its distinguishing combination is coding and agentic capability, image and video understanding, and a one-million-token context window supported by MiniMax Sparse Attention, or MSA. The design targets applications in which these demands arrive together.

The earlier long-context example is straightforward: put a book into a model and ask for a review. That requires reading a large input, but the task can remain a single exchange. An agent introduces a different pressure. It interacts with an environment, receives tool responses and goes through multiple rounds. Each round adds information that may matter to a later decision. A short window can become insufficient even when the original user request was small.

MSA separates choosing relevant context from performing attention over it. An index branch selects what matters at a higher level. A sparse attention branch then performs its calculations on the selected blocks. The expensive computation follows a selection step, giving the architecture a way to support longer inputs without treating every block as equally necessary for every calculation. Song presents this as a simple design intended to scale both context length and future model size; the discussion does not specify the block-selection rule or quantify its accuracy–cost tradeoff.

Where does the work get reduced? The flow below makes the selection step visible between the long context and the attention calculation. Keeping a large amount of information available and calculating attention over selected parts are separate operations. The index branch determines which blocks the sparse branch uses.

Wolf places this design in the history of attention optimization. Quadratic attention costs encouraged work on linear attention; FlashAttention then renewed attention to efficient kernels. MSA returns to an architectural question: which attention calculations should happen at all? Efficient kernels and selective computation address different parts of the cost.

The scale change is striking against Wolf’s reference point of GPT-2 and its 1,024-token context. His question about trillion-token attention pushes the conversation into research ambitions. Song treats ultra-long context as something to explore, with architecture and hardware both requiring further work. A million-token model is the subject of the discussion; trillion-token attention remains a prospective direction.

Architecture and inference optimization still have room to improve, especially when demanding tasks also need efficient execution. Asked whether an agent invented sparse attention, Song gives a more human answer: an intern designed the architecture. Wolf’s reaction—“Still some work for interns here”—lands because the contribution sits at the center of the model rather than at its margins.

That contribution fits the research process Song describes. Shared infrastructure lets people experiment with released models. A researcher can devise an evaluation, find a weakness and propose an improvement. Interested colleagues join the project, work for weeks or months, and bring successful results into final model training. Access to the model and the means to evaluate it make proposing a project practical.

The time horizon depends on the problem. Architecture work can require extended investigation, experiments and even revised pre-training evaluations. The process leaves room for that deeper exploration rather than assuming every improvement will fit a short iteration.

The next architectural decision concerns when vision enters training. M3 uses what Song calls native multimodality: text and visual understanding train together from the first step. The motivation comes from problems MiniMax encountered when adding vision later. By then, the model had already converged toward text understanding, making the new capability harder to fit into the learned system.

The timing choices have distinct difficulties:

• After text pre-training: Adding adapters and training vision afterward harmed text performance in MiniMax’s experiments, while vision understanding did not converge well.
• Partway through pre-training: Introducing multimodality during continued pre-training made results sensitive to the architecture, data mixture and learning rate. A recipe that worked in one experiment was difficult to carry into a larger model.
• From the first step: Joint training offered the most natural route to scaling, but it introduced a stability problem: text-and-vision training could collapse after only a few steps.

MiniMax’s response combined work on ViT, the vision transformer, with work on the training data. Interleaved data retains images and videos within naturally occurring material instead of removing the visual content. Cleaning and masking then shape that data, and Song also identifies reward modeling as part of the training effort. The distinction matters: preserving visual material does not mean feeding all raw material through unchanged.

Song reports that this work let joint training scale without collapsing. The account names the areas the team changed, but does not provide a complete stabilization recipe or isolate each change’s contribution. Its practical lesson is about the training decision: starting both modalities together can avoid the difficulties of retrofitting vision, provided the team solves the resulting stability problem.

The ambition extends to model size. When Wolf asks about going beyond one trillion parameters, Song says that is a future direction, because some tasks remain difficult for smaller models. Sparse attention’s intended ability to scale model size and context length therefore matters beyond the current release.

MiniMax’s product story begins with the model. Song describes multimodal intelligence as an ambition that preceded the company itself. Apps followed as a way to let people experience model capabilities through usable interfaces and concrete scenarios. An API alone would leave many potential users unable to try the technology.

Song estimates that those apps reached more than 300 million people across around 200 countries, and over a million companies. These are her broad reach estimates, without a defined activity period. Their role in the conversation is to explain why interfaces and applications are a substantial part of MiniMax’s work alongside foundation models.

Wolf then raises the persistent business question: how does giving models away fit with earning revenue? Song’s answer focuses on the research value of open releases. The model team plans to keep open-sourcing because performance feedback and community pull requests help improve later versions. The conversation explains that improvement loop without resolving the revenue model or identifying which models power particular apps.

Useful feedback takes two forms:

• Failures in real use: Multimodal issues are especially valuable because combining these capabilities is new territory for the team, and Song acknowledges that the model may still have flaws.
• Requested controls and features: Users have asked for thinking-effort controls, for example. These requests tell the team what to pursue in future models; they are not a promise that every requested feature already exists.

The capabilities come together in a concrete application: give an agent a long video, let it understand the material, then ask it to act through tools. PowerPoints and loosely structured reports pose related problems. The input is richer than a neatly formatted text prompt, and the desired result extends beyond describing what the input contains.

Wolf makes the video example personal: could an agent watch his YouTube tutorial and understand how to use his coding tools? The proposed change is from consuming instructions to doing the work those instructions describe. First, multimodal understanding lets the model interpret the video. Long context provides room for the lengthy input. Tool-use capability then lets the application turn that understanding into an action, whose response can add information for subsequent rounds. Song agrees that this is a plausible use case; it is a proposed workflow rather than a demonstrated tutorial-to-action result.

This example gives the million-token window a purpose. The agent may need the source material and the accumulating interaction in the same task. Vision supplies access to the tutorial’s content; coding and tools supply ways to act on it; context carries the information those steps require. Increasing any one capability alone would leave part of that workflow unresolved.

Inside MiniMax, model-assisted work already reaches research. The team builds its own research harnesses—the surrounding systems that put model capabilities into automated workflows. Song says many workflows are automated and identifies kernel optimization, post-training other models and generating data as examples of research work increasingly within frontier models’ capabilities.

M3’s long-horizon and kernel-optimization capabilities are useful here because research work often requires more than one isolated answer. The harness combines those capabilities into daily routines, which Song says makes iteration faster. When Wolf asks whether M3 is already building M4, she corrects the destination: “building M3.1.” The claim is that M3 helps develop its successor, not that the entire development process runs autonomously.

The closing direction is multi-agent systems and model routing. Song is excited by applications that combine models or agents to tackle more complex tasks. Such systems also expose what individual models can and cannot do. That adds an application-level research question to the architectural ones: once a model can handle long context, visual inputs and tools, how should a system organize those capabilities across multiple participants?