How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

Um, all right. Um, so hi, everyone. I-- My name is Olive. I work at MiniMax on reinforcement learning research. And today, I'm gonna talk a little bit about, um, our new model, MiniMax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called, uh, which is MiniMax Sparse Attention. Um, a little bit introduction of our team. Uh, we are one of the few

labs over the world that, uh, works on multi-modalities, which means that we have published our, um, of course, LLM models before. Um, we worked on our speech models, we have our own video generation models, and we have our coding agentic models.

And lately, we just released our MiniMax M3, which is open-weight, uh, with frontier performance. So it is actually the first open-weight model to combine the following three, um, frontier capabilities. The first one is what we call SODA, uh, coding and agentic capabilities. The model is able to do a lot of the, uh, agentic tasks that you imagine. Um, it, it can do automatic task decomposition, it can work in your coding

agents, it can drop in for ninety percent of the coding works that you are currently doing. And the second pillar is that the model has one million context window, um, so which is very long, that enable, like, unlocks pretty much base, uh, all basic user, long user interactions, um, of like it handles full code bases, and we trained it with MSA, MiniMax Sparse Attention, so it is nine times faster prefill and fifteen times

faster decode versus full attention. And the third one is that the n- the model was trained native multimodal from step zero, which means that from step, from scratch, we combined visual understanding with, um, language generation. So the model would understand both capabilities very well from the start and, uh, so that we can build many more applications on that, say, computer use agents, right?

So first, I'll briefly explain what sparse attention does. Our core design principle is that a simpler architecture can always lead to better scaling properties. This is the principle we have followed throughout the entire design. So our goal is to always pursue a cleaner and simpler architecture. The sparse attention consists of two branches. The first one is what we call the index branch, which performs a lightweight full attention

computation. It uses four query heads and one key head, and it is trained with a KL loss to match the, the attention scores of the main branch. The index branch is used to identify the tokens with the highest attention scores. These selected tokens are then attended s- by, by the main branch. In other words, the sparse branch does not attend to all of the tokens, but only to a subset of them.

This subset is selected through the lightweight computation of the index branch.

Then, the sparse branch would use the blocks selected by the index branch to de- to determine which keys and values are needed for attention computation.

Uh. All right. And what I wanna emphasize today is that our design is not purely algorithm, like algorithm-driven. Instead, we took an algorithm, algorithm and infrastructure co-design approach. Um, a good e- good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for DeepSeek's own architecture, especially the combination of MQA and the

large head dimension. However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It introduces several practical issues, es- especially from the infrastructure and implementation perspective. So this motivates our design choice. Instead of simply borrowing an existing sparse attention design, we rethink the architecture under the constraints of GQA and real system efficiency.

So what are the issues, um, if we directly apply the existing sparse attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups. However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the diversity that GQA is,

is supposed to provide. Second, from the instr- infrastructure perspective, this design is not very friendly to GPU memory access. GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV, uh, layouts as one KV head with a large head dimension. For example, one by five hundred and twelve. This is relatively contigu- contiguous and efficient to read. However, in GQA,

the same total dimension may be split across multiple KV heads. For example, it's four by one twenty-eight. These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead and reduces the com-compute to memory access ratio. And another issue here is the top K computation itself. To select the most important

tokens, the indexer needs to compute top K scores over a large number of token-level candidates. However, top K has inherent data dependencies, so it is difficult to fully be parallel on GPU. As a result, token-level selection introduces non-trivial computation and merging overhead. So how did we address these issues in, um, MSA? From the algor- uh, from the algorithm side, we remove the

aggregation of the index branch outputs. In the original design, the results from all index branch ha- uh, in-index branch heads are merged into a single set of selected tokens. But as we discussed earlier, this would remove the diversity brought by GQA. Instead, we keep the selection different, different across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend to different sets of tokens. This

helps preserve the diversity of GQA rather than forcing all heads to share the same sparse attention. For the infrastructure side, we move from token-level selection to block-level retrieval. Compared with token-level selection, block, block-level retrieval significantly reduces the number of candidates and lowers the overhead of top K computation and result merging. At the same time, block-level retrieval is more hardware friendly,

friendly. It improves memory access efficiency and increases the compute-to-memory access ratio. Importantly, we also show that the design does not introduce a significant performance drop.

Finally, we design efficient kernels for both training and inference. I, I w- I won't go into the kernel details here today, but this is the system-level component that makes the sparse attention design practical and efficient. So with this design, we were able to extend the training of our M3 model to one million context very efficiently.

Um, and as we talked about earlier, the third pillar is the native multimodality. Our model, MiniMax M3, was trained from scratch with multimodal data. What does that mean? Our model was trained with both text and vision understand-standing from step zero, which is very different approach from current open weight models. Our starting point is very clear. At the current stage, a standalone VL model has relatively limited application

scenarios. But co-combining it with, with, you know, text understanding during the training can be challenging. So the key question and challenge there was, how can we introduce multimodal capability without hurting the core text performance? To answer this, we actually explored several training strategies at scale. The first, and most common, approach is to add multimodal training after the

pre-training decay stage, which is essentially a CPT style approach. However, at that point, the model has already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation. Another approach is to ensure multimodal training before pre-training deca-decay stage. For example, after the model has already seen a number of amount of

text tokens. We also tried this direction but found that the results were highly sensitive to the data mixture and training hyperparameters, such as learning rate, for example. In one setting, a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run. More importantly, during a large-scale pre-training, these parameters cannot be arbitrarily changed just for multimodal

training.

And this led us to our final strategy, native multimodal training from scratch. In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budgets, we find that the pro-- uh, this approach almost does not hurt text capability.

We actually did many fun an-analysis during these expe-experiments. Uh, one of them is this interesting observation, uh, from the attention maps.

You can see from this picture that for CPT style multimodal models, the text tokens often pay very little attention to visual tokens. The visual token regions are mostly dark in the attention map. This suggests that the multimodalities are not fully integrated. While in contrast, with native multimodal training, the attention map looks much more integrated. We do not see here a clear

separation between text tokens and visual tokens. This suggests that the model learns to fuse visual and textual information more naturally from the beginning. So overall, our training strategy is not just about adding vision capability. It is about integrating multimodality into the model while prese- preserving its strongest text capabilities.

And of course, uh, we did many experiments on multimodal architecture choices. Our overall architecture is relatively standard. The image or video is being processed by a ViT, which produces visual tokens. These visual tokens are then inserted into the language model input by re- replacing the original image placeholders in the text sequence. After that, the visual embeddings and text embeddings are fed together

into the language model. One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding. In a standard ViT setting, attention is, is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other

directly. This gives the model stronger temporal and cross-frame visual modeling capability. And we found that this improves visual understanding and the benefit also transfers to some image benchmarks, suggesting that it strength the overall visual representation rather than only improving v- video specific, uh, tasks.

And one practical detail is that we cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges. In our exploration, using four frames gave us the best, best trade-off between performance and efficiency. So overall, with our architecture design, with our training strategies that we explored, uh, throughout the stages, and with our

analysis and ViT explorations, we were able to train the model from scratch, uh, with multimodality understanding. And that will be the end of the presentation today. Thank you.