▶ Watch ↗17:42
Is a Faster Model Actually Faster? Test the Whole Job — Morgan Suo, MiniMax
Read the full talk →Key ideas
Scroll to read ↓Morgan Suo uses a financial research task to connect model optimizations to the accuracy, cost and wait time of a report an analyst can actually use.
- Measure time and cost to an acceptable report, including retrieval, retries, failed attempts and human repair.1:13 ↗
- Sparse attention reduces work by selecting text blocks, so test whether important evidence remains findable at different positions in real documents.5:31 ↗
- Lower precision can reduce data movement, but near-identical average scores can conceal changes on individual tasks; full-setup comparisons do not isolate their causes.7:59 ↗
- Test serving load and reasoning effort against the deadline and quality needed for the actual job, rather than relying on throughput or first-token latency alone.11:47 ↗