From Mixture of Experts to Mixture of Agents … with Super Fast Inference

How do you keep making language models more capable? One approach changes the architecture before training: Mixture of Experts (MoE) increases a model’s capacity while activating only part of it for each token. Another combines models that have already been trained: Mixture of Agents (MoA) coordinates separate LLM calls and combines their answers. Daniel Kim and Daria Soboleva’s workshop builds the second approach. To participate, attendees first obtain a Cerebras cloud API key; the hands-on work will produce an application, followed by a configuration challenge and Q&A.

Soboleva introduces herself as a Cerebras research scientist focused on MoE and hardware-efficient training, with previous work at Google and on the SlimPajama dataset. Kim handles growth, developer activities, startup sales, and access to tokens for non-enterprise research projects. His introduction includes a fondness for hot pot; the more consequential credit goes to Kevin, the high-school intern who built nearly all the workshop and is absent for a final exam.

Cerebras supplies the inference infrastructure. Kim introduces its custom silicon beside an NVIDIA H100 and attributes its speed to hardware design and scaling across larger models. Kim claims roughly 15.5× faster inference for Llama 3.3-70B than the fastest GPU inference provider in the comparison he presents. That is a historical presentation claim; the recording does not specify the comparator configuration or measurement methodology.

The architectural motivation begins with GPT-3 and the relationship between parameter count and capability described in Language Models are Few-Shot Learners. Llama 3 adds the importance of dataset curation and training on more tokens. DeepSeek-V3 then illustrates the need for architectural changes as total model capacity grows. Soboleva’s spoken size sequence is 13, 400, and 600 billion parameters; the published specifications are 175B for flagship GPT-3, 405B for the largest Llama 3 model, and 671B total for DeepSeek-V3. The underlying engineering question remains: how can a larger model be served efficiently?

Start with the transformer’s embedding, attention, and feed-forward layers. Soboleva locates the bottleneck in the feed-forward network, which must process representations arriving from earlier layers across languages and domains. Her example is text mentioning the Golden Gate Bridge: which neurons should respond to that concept, among everything else the model must represent?

MoE replaces a monolithic feed-forward network with several feed-forward networks called experts. Math and biology specialists provide an intuitive analogy, although those names are not a guarantee that learned experts acquire neat human-readable specialties. A router selects which experts process a particular token. Adding experts therefore increases total parameter capacity without requiring every expert to run on every token.

Total parameters and active parameters are different quantities. Soboleva describes matching a dense model’s active parameter count while gaining the capacity of a larger trained model. DeepSeek-V3 makes the distinction concrete: its report specifies 671B total parameters but 37B activated per token. Sparse activation reduces computation relative to activating the whole model; matching active parameter counts alone does not establish identical wall-clock latency across hardware and architectures.

Soboleva closes by describing MoE as an increasingly common scaling approach and names GPT-4 and Claude as adopters. Those particular architecture claims should remain attributed: the GPT-4 Technical Report withholds architectural details, and the workshop does not establish Claude’s internal architecture. The mechanism itself does not depend on those examples: a model can have substantially more stored capacity than it uses for any one token.

Architecture is not the only place to spend compute. Referencing Ilya Sutskever’s NeurIPS discussion of a potential limit to unique pretraining data, Kim turns to inference-time compute: additional work after training that may improve the answer. An AIME competition problem supplies the example. These problems can require multiple dependent reasoning steps, making a quick answer less useful if it is wrong.

Kim reports the following results for one AIME problem, contrasting the non-reasoning GPT-4o with the reasoning model o3:

The comparison does not provide the exact problem identifier, prompts, reasoning settings, repeated trials, or timing boundaries. Its immediate product implication is still clear: 293 seconds means four minutes and 53 seconds of waiting. Kim’s joke about a “three business days” timeline points to the difficulty of putting that delay inside an interactive application.

MoA moves the extra computation into an application-level workflow. Send the input to multiple LLMs with different system prompts, collect their responses, and ask a final model to combine them. MoE provides an analogy, but the units differ:

Kim attributes the approach and selected benchmark improvements to Together AI’s Mixture-of-Agents Enhances Large Language Model Capabilities. Its evaluated response-quality benchmarks are separate from the AIME example. The workshop’s proposal is to use fast inference to make many calls practical, potentially producing a stronger system from less capable component models.

Kim next plays a NinjaTech.ai demonstration of a production chatbot using Cerebras. Kim reports that NinjaTech’s system answered the same AIME question correctly in 7.4 seconds. The result belongs to the demonstrated system as a whole, rather than to one underlying model.

The workflow is more involved than asking several models and taking a vote:

1. Propose: a planning agent produces eight possible answers.
2. Critique: another agent evaluates feasibility and rejects the entire first set.
3. Revise: the planner produces eight more proposals, described as using 16K context worth of thinking tokens.
4. Select: the critic identifies two plausible candidates.
5. Synthesize: a summarization agent combines those candidates into the final answer.

The rejection step matters. The first set of outputs does not become the answer merely because it exists; critique changes what the planner does next.

Kim reports over 500,000 generated tokens and 32 LLM calls for this roughly seven-second process, with calls running both in parallel and sequentially. The completed slide labels the total as 510k tokens. These are the presentation’s accounting figures for this demonstration, without a supplied call trace or token breakdown. The example shows why low latency can enable a different workflow: the system spends substantial intermediate computation on alternatives that the user never sees. Long single-model reasoning may remain appropriate for asynchronous tasks, where waiting is less disruptive.

Kim’s hardware explanation begins with the H100 diagram. He describes roughly 17,000 compute cores and a bottleneck between compute and external memory holding weights and intermediate state. Generating the next token requires access to those values; larger models increase the pressure on memory channels, including movement associated with the KV cache. This is the simplified hardware account used in the presentation, rather than a complete description of every GPU serving configuration.

Cerebras takes a different approach to locality. Kim describes 900,000 cores paired with 900,000 distributed memory stores, with each core directly accessing its local memory. In his account, weights remain assigned to those stores instead of being repeatedly fetched from external memory for each computation. The important distinction is where data resides relative to the compute that consumes it.

For larger models spanning chips, Kim says Cerebras transfers activations between chips while retaining weights locally. He illustrates the communication volume with a single Ethernet cable and contrasts it with the connectors and switches used in a DGX system. This is the basis of his linear-scaling claim: reduce recurring data movement as more hardware is added. It does not establish an unlimited, cost-free scaling law, but it explains why memory placement and communication topology matter to the many-call workflow just demonstrated.

Soboleva translates the architecture into a familiar interaction problem. A single model fails to solve a complex task, so the user keeps refining the prompt until the conversation becomes unwieldy or exhausts its context. Her alternative resembles a surgical team: assign different portions of the task to specialists, then combine their work. Prompt engineering defines those responsibilities. The goal is one user question followed by internal collaboration, rather than repeated manual correction; her shift from “zero shot” to “one shot” refers to that interaction pattern, not a formal claim about few-shot evaluation.

The workshop setup follows a short sequence:

1. Obtain your own Cerebras API key.
2. Fork the supplied cerebras-moa-workshop repository; the presenters also ask attendees to star it.
3. Choose hosted Streamlit deployment or local Python execution.

The workshop Slack channel contains the repository link, slides, and peer support. Slides 50–54 repeat the setup instructions. Kim recommends Streamlit when bandwidth permits because it avoids local Python installation problems.

For hosted deployment, sign in to Streamlit with GitHub, select the forked repository, set the app’s main file path, and open Advanced Settings to enter the Cerebras API key. Use the repository’s setup instructions for the entrypoint and secret name; the spoken walkthrough does not spell out a complete launch configuration.

Setup exposes a real credential-handling problem: an attendee notices an API key committed to the repository. Kim says the key is rate limited, asks everyone to use their own credentials to avoid collective throttling, and announces that he will rotate it immediately. Rotation is announced, not shown completing. A shared committed credential would couple everyone’s requests to the same limit even before considering the exposure itself.

For local startup, Kim points to the README’s requirements installation and launch command. Once the application is running, the top-level summarization agent controls how individual results are aggregated. Under Agent Management, each agent has an editable name, model, temperature, and prompt. Agents can be added or deleted, and the layer count controls how many successive rounds the system performs.

The first prompt is deliberately low stakes: plan a San Francisco trip for Kim and Soboleva from 3 PM to 9 PM. The application runs multiple LLM calls across three layers. When the response takes time to appear, the presenters attribute the delay to workshop Wi-Fi; this run does not supply a measured inference latency.

While Kim helps an attendee configure cloud secrets, Soboleva explains the execution graph. Agents within a layer run in parallel; layers run sequentially. The outputs of one layer become the input to the next. After the third layer, a final summarization model combines the results into the user-facing answer. The intermediate tokens broaden the material available to that final call.

A layer is a set of agents, not an agent itself. Its outputs are finalized, concatenated, and passed onward. The following Python implementation captures that scheduling pattern. complete is an asynchronous model-call function supplied by the application; model and prompt choices remain in the agent configuration.

import asyncio
from collections.abc import Awaitable, Callable, Sequence
from dataclasses import dataclass


@dataclass(frozen=True)
class Agent:
    name: str
    model: str
    system_prompt: str
    temperature: float = 0.2


Complete = Callable[[Agent, str], Awaitable[str]]


async def mixture_of_agents(
    question: str,
    layers: Sequence[Sequence[Agent]],
    summarizer: Agent,
    complete: Complete,
) -> str:
    context = question

    for layer in layers:
        if not layer:
            raise ValueError("Each layer needs at least one agent")

        responses = await asyncio.gather(
            *(complete(agent, context) for agent in layer)
        )
        combined = "\n\n".join(
            f"{agent.name}:\n{response}"
            for agent, response in zip(layer, responses)
        )
        context = (
            f"Original request:\n{question}\n\n"
            f"Previous layer responses:\n{combined}"
        )

    return await complete(summarizer, context)

For the travel example, question remains the San Francisco itinerary request throughout the workflow. asyncio.gather waits for every agent in the current layer before the next layer begins. Increasing width adds parallel calls; increasing depth adds dependent rounds that cannot start early.

The AI Configuration Challenge turns the interface into an evaluation loop. Open Configure AI and Generate, choose a configuration, generate code, and submit the solution for automated grading. The challenge’s maximum score is 120 points. Kim starts with presets designed to produce a C-level baseline. In the demonstration, changing the main agent to a Qwen model improves the grade from C to B. The available tuning variables include the main model, layer count, temperature, system prompt, and each agent’s model and prompt; Kim says Soboleva and Kevin have already reached perfect scores.

A brief clarification establishes what is being generated. Soboleva expects an existing buggy Python function; Kim explains that the current UI generates the function from scratch from its task input. The target is calculate_user_metrics, whose displayed requirements describe the intended input and output. Soboleva then offers to share the buggy, unoptimized baseline in Slack as another way to inspect the problem. In either workflow, the grader consumes the generated function, and the objective is correctness plus optimization under that grader’s rules.

The preset responsibilities divide the engineering work:

• Bug specialist: find defects and edge cases.
• Performance specialist: remove unnecessary work, including repeated computation.
• General reviewer: consider the implementation more broadly.
• Final synthesizer: turn the preceding agents’ contributions into one function.

These roles give the aggregation step different kinds of evidence. A performance suggestion should not silently override an edge-case fix; the final model must reconcile their implications in the implementation it returns.

Soboleva’s concrete prompt example is an empty list accessed by index. An engineer who spots that failure can give one agent a targeted instruction, rather than asking every agent to improve everything. For example, the bug specialist’s prompt can identify values[0] on an empty values list and ask it to handle the case according to the function’s required behavior. Inspect the default output, locate what remains wrong, and revise the responsible prompt. Soboleva emphasizes that diagnosis and engineering judgment still matter when vibe coding.

The exercise produces multiple reported perfect scorers. Kim first mentions three, then asks attendees who reached the maximum to raise their hands and acknowledges several winners. He asks them to come forward after Q&A with an email address and suggestions for modest prizes. The observed outcome is success on the workshop’s particular grader, rather than evidence that one configuration transfers unchanged to other coding tasks.

The Q&A begins with a gap in the manual tuning exercise: how would you automate prompt design when searching by hand becomes prohibitively expensive? Kim points to Devin as an example of longer-running coding systems that return proposed solutions and build larger applications. His answer focuses on making those workflows faster—reducing hours of waiting toward minutes—rather than supplying an AutoML or prompt-search algorithm. Faster execution can make a search more affordable, but the workshop has not specified how to choose the next configuration automatically.

Questions then turn to the service behind the application. The availability and roadmap answers describe the service at the time of this workshop, not its current API contract. Kim estimates that Cerebras had opened six US data centers in the preceding year, with France planned for that year and Canada also on the roadmap.

New-model onboarding depends principally on kernel availability. Kim says Qwen 32B was comparatively straightforward because its required operations were similar to Llama’s and the kernels already existed; the remaining work included QA and API-level features. A different architecture can require custom kernel engineering before it runs efficiently. There is therefore no single onboarding duration that applies to every model release.

Power consumption needs a workload-level comparison as well. Kim reports approximately one-third the power consumption of NVIDIA GPU systems for an equivalent workload, citing Cerebras’s observations. He explicitly qualifies the comparison as workload dependent: one large Cerebras system and one GPU do not represent equivalent capacity. The claim concerns systems delivering comparable work, and the presentation supplies no detailed measurement configuration.

Asked where MoA belongs relative to state-of-the-art models, Kim returns to configuration. Poor prompts can produce poor results; model selection, role design, and aggregation must be engineered for the use case. MoA is a system to optimize, not an automatic quality upgrade. This work changes the inference workflow rather than training new model weights.

Soboleva connects the approach to ensemble learning and decision trees: several models can sometimes produce a more robust aggregate than one model. She suggests that even identical configurations and prompts could yield a better final answer. That is a possibility, not a guarantee that duplicated calls add useful diversity.

An attendee sharpens the objection with an XGBoost analogy: what happens if one agent is right and 2,000 are wrong? Could homogeneous answers overwhelm the useful minority? Soboleva responds that excessive agents are likely to create redundant outputs, some of which will not contribute to the final solution, while still consuming generation time. She relates that to expert utilization in MoE; the attendee connects it to feature importance. This answers the wasted-computation concern, but it does not establish a mechanism that protects a correct minority from an incorrect consensus.

For a fine-tuned model such as a customized Qwen 32B, Kim distinguishes enterprise arrangements from the public offering. He says enterprise customers already run custom fine-tuned models in Cerebras cloud, while general LoRA support is still on the roadmap in the service being discussed. Supporting a base model does not, by itself, imply that the public API accepts arbitrary customer adaptations.

An attendee asks whether diffusion text generation could compound Cerebras’s speed advantage, suggesting a tenfold speedup for diffusion models. Soboleva treats diffusion and state-space models as promising research directions, with training recipes, scaling, and robustness still under investigation. She says public inference model selection prioritizes proven, robust models. Kim mentions an impressive internal diffusion demonstration but supplies no benchmark or release commitment; the audience’s multiplier cannot be carried into a performance estimate for the service.

For architectures with unsupported layers, Kim describes collaboration rather than a drop-in conversion. Mistral is his example: customer engineers and Cerebras engineers work together to provide the required kernels. The hardware’s distributed compute and memory can accommodate diverse designs in principle, but the implementation must support the model’s actual operations and any additional model-specific behavior, including the custom reinforcement-learning behavior raised in his answer.

The multimodal question exposes the same distinction between an existing partner deployment and a public feature. Kim says some image queries in Mistral’s Le Chat already run on Cerebras through a non-public multimodal API, and describes public cloud access as forthcoming. His answer also names Mistral Large 2, without clearly identifying the model handling those image queries. Real-time APIs remain a subject of interest: he asks about the attendee’s use case rather than announcing their availability.

The final research question asks what becomes possible when a model is designed specifically for Cerebras, rather than ported from existing hardware. Soboleva identifies unstructured sparsity as a promising pretraining direction. Kim says inference models specially designed for the hardware have not yet arrived in the offering he describes; inference itself had launched only about nine months earlier. Closer work with partners such as Mistral is proposed for both current and future chip generations. Soboleva also highlights very large models spanning several chips, where reducing weight movement and orchestration is especially valuable.

The last technical exchange makes the customer-facing abstraction precise. An attendee asks about model capacity on an instance, initially thinking in terms of VMs. Kim reframes capacity as physical systems that can grow by adding chips, citing models from eight billion parameters through the recently onboarded Maverick, with larger Meta models planned. When the attendee points out that this capacity has a cost, Kim agrees. His broad scaling claim does not mean that customers receive unlimited capacity under a fixed allocation.

The public-cloud contract is an API with rate limits, not a customer-managed hardware partition. Kim says users cannot partition a Cerebras chip or instance through the cloud offering described. Outside on-premises arrangements, Cerebras provisions the backend systems, and public API users consume a shared pool subject to per-user limits. For a mixture-of-agents application, that distinction is practical: adding parallel calls increases demand on the service allocation even when those calls fit neatly into the application’s execution graph.

The presenters remain available for questions and startup inference discussions. Participants who reached the maximum challenge score are asked to bring proof, provide an email address, and suggest a prize—the final step after configuring, generating, and grading their own system.