Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax

Uh, I'm Morgan from MiniMax. Uh, how many of you knows MiniMax before? Oh, whoa, I like that. Yeah, because, uh, I joined this min, uh, company for three, almost three years. But when we first, like, stepped into the American market, like, uh, almost no one knows us. But at that time, we have the first video generation model, so I will show you something of that at the end of this speech. But first, I will show you, like, about our, the latest large language model, which will be released next week.

So, uh, today, uh, um, I would like to... My topic is the hidden cost of a faster model. So when people are thinking about, like, if you have a AI application, your product built in, in your own pro- workflow. So when a new model arrives, when I say, "Oh, please, we have a new model. Do you want to try it?" It can generate a task faster, but your, I think your question is, should I switch to that model?

So you try a research task, and the report just comes back very soon, and, but it misses a very important source, so someone has to find that source and fix that report. I think this is the thing we did every day, every day. So did the job really get faster with this faster model?

Today, we will look at how we reduce the model computation, and then we will ask how to test the value of those changes of this model. So let me show you the job that we will use to answer this question. So at MiniMax, we build different multimodal models and pro- products that people use. And a quick chat reply and a research report actually have different needs. So the task tells us what good performance means.

Today, I will focus on our latest, uh, large language model, M3.1 Flash, and one research job. So here's the analyst question. As AI server demand is growing, are the companies supplying these circuit boards turning this growth into cash? The PCB there means printed circuit board. I think you guys know that. And unco- it, it connects electronic components that inside devices that, such as the

AI servers. So our research case looks like companies in this supply chain. The internal review describes work across fourteen company reports, and the model has several jobs. It must find the right reports, it must compare the same reporting periods, and it must keep different businesses separate. Then it must explain what the numbers mean. Think about the analyst who receives the report. They need to open the source behind an im-

important claim, and they need to check a calculation. They need to see what evidence could change the conclusion. So a useful report gives the analyst something that they can check, and that is our standard for a finished research job. So one finding shows why those checks matter.

This internal review highlighted a very useful distinction. So profit can grow, but cash flow tells a different story. A company sells more, so its reported pro- profit rises, but customers have not paid yet. Or it has, uh, bought materials that are still in storage. So an ana- uh, the analyst needs to know why. The model should point to the cash flow statement and the relevant its

disclosures. So a report earns trust when its claims can be checked. And the internal review here describes the research output. So we still need to verify its sources and calculations, and also, we also need to measure the time and cost on this same job. But the co- technical tasks ahead use separate tasks, so they help us to understand s- specific trade-offs. With this job clear, so let's look at what changes inside M3.1 Flash.

This table shows the main changes inside M3.1 Flash. So I will connect them to a few simple questions. First, where can we reduce the attention work? And second, where can we store and move less data? And third, how does, uh, the serving setup affect the time a user waits? And fourth, we also control how much reasoning the model does. These

choices affect different parts of the job, and each knows a check against the result we want. So every optimization has to help the finished job. And let's begin with how the model uses a long input.

So nowadays, I think every model says, "Oh, I have a very big context window." Like we said, we have one million. So among this one million, how it can help you to find the output, find your result quickly, faster, and correct.

So some passage, uh, passages, like in this report, that matter just much more than others for this question. So sparse attention itself means, like, we uses a very small selections that first, like the blue, uh, box there. So it scores the b- blocks of the text and the main attention, then calculation, uh, calculate the uses of the selected blocks. And the model also keeps the nearby text in view.

And this selection, actually, they can change with the query. And different questions can also needs different passages of the, the context. And the number on, at the bottom show how much text this published example selects. And the full input can be much longer. For our research job, the goal is to reduce this attention work across many of, uh, these documents. But just within this document, a very short footnote there

may change the entire conclusion. So for a cash flow question, the footnote may matter more than the pages of industry outlook. So we want the model to include that passage too. Less attention work still has to find the important evidence. So the next question is simple: Can we still find the facts we need?

M3.1 Flash also uses like the, the sparse attention in its first, uh, three layers. That makes this retrieval an important check. So this separate task places facts at different op- positions in long inputs. So in one run, the model passes all 39 requests, and the longest inputs were about, uh, 830k, uh, tokens. So re- the result covers the c- cases in this test.

And for our research job, we need tests with real reports. Can the model find the cash flow foot- footnote? Does it use the right reporting period? And can it spot a conflict between different documents? So move this important passage to different places in the input and then ask the same question again. A long input is useful when the model can find a key fact, so finding the evidence is one check too. Next, we look at the numbers we

used inside this model.

So the model keeps the internal information from earlier tasks, which we call this KV cache. And the M3.1 Flash changes its keys and values from 8 bits to 4 bits, but the query still stays at eight bits. And this can reduce the data we store and move. It also changes the position of that internal information. Look at that table. So the average scores actually is almost the same, but the sign score there, it

drops like 6.6 percentage. So some, but some other scores raises. So the average score actually can hide a mistake which your user cares about. The, this compares the two full setups, but it does not tell us which changes caused each score. So for our research job, we should check the output, uh, output again. Are the sources right? Are the calculations right? And does the explanation match the

evidence? So the same ideas applies to another part of the model, its experts. Uh, our model is, uh, the Mo- in MoE architecture. So, uh, it has the computing parts that we call the experts, which is the, uh, M of E, E, experts. So it selects some of them for each token. In M3.1 Flash, these selected experts use 4-bit weights and ac- activations.

The shared expert is outside this change, and weights are stored model, uh, values and activations are values the model calculates. And lower precision can reduce data m- movement and support more efficient computation. But the full calculation has several steps. So we scale the numbers, we run the calculation, and restore the scale. The position of these steps really matters. And the table shows like where scale restore happens.

And getting that order right is the part of getting the answer right. So fast math still has to be correct. And while we check the complete result, we, we measure the complete imple- uh, implementation. And for this research job, we run the same questions again. We inspire whether answers still meet our checks. And now we look at inside. We have already look at inside the model, and now we can look at how it generates an answer for you.

So a serving method called speculative decoding works in two stages. A faster draft process. It can help you to propose a short sequence of tokens and the, then the target model can checks it. And the system keeps this first part that the target accepts and makes a correction when it's needed, and then continues this whole process. And sometimes it keeps several proposed tokens, and sometimes

the draft does work what it cannot ke- keep. So the co- opportunity here is to get several tokens from one target check. And but drafting takes time, checking takes time, rejecting these tokens also takes time. So count the time spent checking the draft too. These checks propose tokens and, but financial claims and sources need their own checks. So for our research job, we should also include the source retrieval, the tool

calls, and retries. The final measure is the time to a report that an analyst can really use in their own, own work. So even when one w- request runs well, many users can change this diff, uh, experience.

Here's a separate serving test. So it's uses the story generation with low reasoning effort. But the concurrent requests now, like we rise from the 32 to 64. So the total output per second grows about 1.9 times. But look at the wait time. Actually, it, uh, the P95 wait for the first token rises. It f- goes from about 1.5 seconds

to [unclear] seconds. So for 95% of this request, the first token arrives by this time, and the wait now is becomes, like, eight times longer. So the median stay l- as, uh, the, the time that we can satisfied, but when we cannot only look at the median time, so it can changed everything that if we only, like, uh, look at that one. So the

users feel the wait, and even when the throughput re- you see like, "Oh, we have throughput, like, more than 300 per second," but still you need to care about the final wait time. So the first token may be reasoning, and we will still need to measure when the usable report arrives. So set a deadline for that report, then we choose a load that meets it. And there's one more setting that change the work we ask the model to do. So this is the first time we, uh, just use this

different, uh, reasoning, uh, uh, control. So, uh, from this part, uh, so this is the default max changes that we, uh, use in the request. And it can controls how many tokens we use for this model thinking. So in the document set up here, so s- thinking stays on, but if we leave the effort blank, the default is max. So default can change the amount of work we request. And

one coding test there shows the trade-off. So high ch- reasoning and the max reasoning. So the high uses fewer op- output tokens than the max, but the accuracy actually fell from 80% to 68%. So the fewer tokens came with lower accuracy on those task items, so give this task the thinking budget it really needs. So for our research job, compare these settings on the same

reports and check whether each setting finds the evidence and the support, the calculations. And the r- quick first pass on the report ready for review may need different budgets. So we need to record this setting, uh, which reasoning control we need to put and th- then used in our later work.

Okay. Then we shape this research workflow with all these, um, new updates we have with this model. First, we can fix this question on the source reports and use the same information cutoff and gr- agree on what makes a report acceptable before running this comparison. And then we can check three things. The first is, can the analyst trust the report and open the resources behind the

important claims and check the reporting periods, recalculate the financial, financial numbers, and look for the evidence that could change the conclusion? And second, what did the finished job cost? How long did they take? And include retrieval, tool calls, and the retries. So if a person must fix the report before use, count that work, too. So include all the failed attempts in the cost of producing, uh, of producing these

acceptable reports.

So tests are the number of these users you expect, and check whether reports arrive within the promised time, and measure the cost of report you can actually use. And the third one is, can we repeat this result? Save th- this model and runtime versions, and also save the prompt settings and sources. So keep these examples of failures, and we can start with a small rollout and keep a way to switch back.

So before we finish, here's a short recap of today's talk. It's created with MiniMax A3, our AI video model, and you can check here.

Okay, so, uh, this is another model that we are working on, and it's opened with, so everyone, uh, who wants to build this, actually it's only, uh, [unclear]. Okay. So let's return to this, uh, our opening question today. Should we switch to a faster model? And look at this finished job that we're talking about for the last, uh, 18 se- minutes. So for our a- analyst, that means a report with resources they can open, and it has numbers

they can check, and it must arrive when they need it. And for our, uh, this M3.1 Flash changes how we spend computation, and the complete task, uh, shows that those changes means for the user. So measure what we cost to finish job, m- and on time. So faster ge- models, uh, I think, like, f- faster mo- models really matter, but what matter most is what we can do with them. Thank you, guys.