How MiniMax M3 Was Built: Sparse Attention and Native Multimodality — Olive Song

0:01 [music] >> Um all right. Um so, hi everyone. I My name is Olive. I work at Minimax on reinforcement learning research. And today, I'm going to talk a little bit

0:22 about um our new model, Minimax M3, and how we trained it natively multimodal from step zero, and how we trained it with MSA, which is called uh which is Minimax sparse attention.

0:36 Um a little bit introduction of our team. Uh we are one of the few labs over the world that uh works on multimodalities, which means that we have published our um of course LLM

0:51 models before. Um we worked on our speech models. We have our own video generation models, and we have our coding agent models. And lately, we just released our Minimax

1:06 M3, which is open weight uh with frontier performance. So, it is actually the first open weight model to combine the following three um frontier capabilities. The first one is

1:19 what we call soda uh coding and agent capabilities. The model is able to do a lot of the uh agentic tasks that you imagine. Um it it can do automatic task

1:31 decomposition. It can work in your coding agents. It can drop in for 90% of the coding works that you are currently doing. And the second pillar is that the model

1:42 has 1 million context window. Um so, which is very long that enable like unlocks pretty much base all basic user long user interactions um of like it handles full code bases.

1:56 And we trained it with MSA Minimax sparse attention. So, it is nine times faster pre-fill and 15 times faster decode versus full attention. And the third one is that the the model

2:09 was trained native multi-model from step zero. Which means that from step from scratch, we combined visual understanding with um language generation.

2:19 So, the model would understand both capabilities very well from the start. And uh so that we can build many more applications on that, say computer use agents, right?

2:35 So, first I'll briefly explain what sparse attention does. Our core design principle is that a simpler architecture can always lead to better scaling properties.

2:46 This is the principle we have followed throughout the entire design. So, our goal is to always pursue a cleaner and simpler architecture. The sparse attention consists of two

2:58 branches. The first one is what we call the index branch. Which performs a lightweight full attention computation.

3:06 It uses four query heads and one key head. And it is trained with a KL loss to match the attention scores of the main branch. The index branch is used to identify the

3:18 tokens with the highest attention scores. These selected tokens are then attended by by the main branch. In other words, the sparse branch does

3:30 not attend to all of the tokens. But only to a subset of them. This subset is selected through the lightweight computation of the index branch.

3:43 Then, the sparse branch would use the blocks selected by the index branch to to determine which keys and values are needed for attention computation. All right.

4:04 And what I want to emphasize today is that our design is not purely algorithm like algorithm driven. Instead, we took an algorithm algorithm and infrastructure co-design approach.

4:19 Um a good good example to start with is DSA. DSA was a very strong design, but it is also highly optimized for Deep Seek's own architecture, especially the combination of MQA and the large head

4:33 beam. However, when we move to a more commonly used GQA architecture, directly applying the same design becomes less straightforward. It

4:44 introduces several practical issues, especially from the infrastructure and implementation perspective. So, this motivates our design choice. Instead of simply borrowing an existing

4:55 sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency. So, what are the issues if we directly apply the existing sparse

5:12 attention architecture? First, from the algorithm perspective, GQA has multiple KV heads, which should ideally provide more diversity across different KV groups.

5:24 However, if we directly reuse the same token selection strategy, multiple KV heads will end up attending to exactly the same selected tokens. This essentially cancels out the

5:35 diversity that GQA is is supposed to provide. Second, from the infrastructure perspective, this design is not very friendly to GPU memory access.

5:47 GPUs are much more efficient when reading contiguous data. In the MQA setting, we can think of the KV 8 layout as one KV head with a large head dimension, for example, 1 by 512.

6:01 This is relatively contiguous and efficient to read. However, in GQA, the same total dimension may be split across multiple KV heads. For example, it's 4 by 128.

6:15 These are no longer one single contiguous read. Instead, the GPU needs to perform multiple independent reads across different KV heads. This increases memory access overhead

6:27 and reduces the compute-to-memory access ratio. And another issue here is the top-k computation itself. To select the most important tokens, the

6:38 indexer needs to compute top-k scores over a large number of token-level candidates. However, top-k has inherent data dependencies.

6:48 So, it is difficult to fully parallel on GPU. As a result, token-level selection introduces non-trivial computation and merging overhead.

6:59 So, how did we address these issues in MSA? From the algo- From the algorithm side, we remove the aggregation of the index branch outputs.

7:10 In the original design, the results from all index branch In index branch heads are merged into a single set of selected tokens. But, as we discussed earlier, this would

7:21 remove the diversity brought by GQA. Instead, we keep the selection different different across KV heads. Under our sparse attention design, when using GQA, different KV heads can attend

7:35 to different sets of tokens. This helps preserve the diversity of GQA rather than forcing all heads to share the same sparse tension. For the infrastructure side, we move

7:46 from token level selection to block level retrieval. Compared with token level selection, block block level retrieval significantly reduces the number of

7:56 candidates and lowers the overhead of top key computation and result merging. At the same time, block level retrieval is more hardware friendly friendly. It improves memory access efficiency and

8:10 increases the compute to memory access ratio. Importantly, we also show that the design does not introduce a significant performance drop.

8:21 Finally, we design efficient kernels for both training and inference. I I I won't go into the kernel details here today, but this is the system level component that makes the sparse

8:34 attention design practical and efficient. So, with this design, we were able to extend the training of our M3 model to 1 million context very efficiently.

8:56 Um and as we talked about earlier, the third pillar is the native multi-modality. Our model, Minimax M3, was trained from scratch with multi-modal data.

9:08 What does that mean? Our model was trained with both text and vision understand standing from step zero, which is very different approach from current open weight models.

9:19 Our starting point is very clear. At the current stage, a standalone VL model has relatively limited application scenarios. But combining it with

9:30 with, you know, text understanding during the training can be challenging. So, the key question and challenge there was how can we introduce multimodal

9:40 capability without hurting the core text performance? To answer this, we actually explored several training strategies at scale. The first and most common approach is to

9:54 add multimodal training after the pre-training decay stage, which is essentially a CPT style approach. However, at that point, the model has

10:04 already largely converged. Adding multimodal data at this stage tends to interfere with the learned text capability and causes degradation. Another approach is to enter multimodal

10:19 training before pre-training the decay stage. For example, after the model has already seen a number of amount of text tokens. We also tried this direction, but found

10:30 that the results were highly sensitive to the data mixture and training hyper parameters, such as learning rate, for example. In one setting,

10:40 a smaller learning rate might reduce the performance drop, but this conclusion does not necessarily generalize to another run. More importantly, during large-scale

10:51 pre-training, these parameters cannot be arbitrarily changed just for multimodal training. And this led us to our final strategy, native multimodal training from scratch.

11:04 In this setting, multimodal capability is introduced from the very beginning rather than being added after the text model has already formed its structure. Under the same text token budget, we

11:17 find that the this approach almost does not hurt text capability. We actually did many fun analysis during these experiments.

11:31 Um one of them is this interesting observation uh from the attention maps. You can see from this picture that for CPT-style multimodal models, the text tokens often pay very little attention

11:45 to visual tokens. The visual token regions are mostly dark in the attention map. This suggests that the multimodalities are not fully integrated.

11:58 Well, in contrast with native multimodal training, the attention map looks much more integrated. We do not see here a clear separation between text tokens and visual tokens.

12:12 This suggests that the model learns to fuse visual and textual information more naturally from the beginning. So, overall, our training strategy is not just about adding vision capability.

12:25 It is about integrating multimodality into the model while preserving its strongest text capabilities. And of course, uh we did many experiments

12:40 on multimodal architecture choices. Our overall architecture is relatively standard. The image or video is being processed by a ViT, which produces visual tokens.

12:53 These visual tokens are then inserted into the language model input by re- replacing the original image placeholders in the text sequence. After that, the visual embeddings and

13:04 text embeddings are fed together into the language model. One important finding is that applying 3D attention inside the ViT is very helpful for vision understanding.

13:17 In a standard ViT setting, attention is is applied within each individual image or frame. For video inputs, this means that visual patches from different frames do not

13:28 directly interact at the ViT stage. But with 3D attention, patches across different frames can attend to each other directly. This gives the model stronger temporal

13:40 and cross-frame visual modeling capability. And we found that this improves visual understanding, and the benefit also transfers to some image benchmarks.

13:52 Suggesting that it strengthens the overall visual representation rather than only improving video-specific tasks. And one practical detail is that we

14:05 cannot simply extend 3D attention to an unlimited number of frames because that would introduce additional system and load balancing challenges. In our exploration, using four frames

14:18 gave us the best best trade-off between performance and efficiency. So, overall, with our architecture design, with our training strategies that we explored throughout the stages,

14:32 and with our analysis and ViT explorations, we were able to train the model from scratch with multimodality understanding. And that would be the end of the

14:43 presentation today. Thank you. >> [music]