Elie Bakouch

Elie Bakouch is a research engineer at Prime Intellect and a co-creator of SmolLM, Hugging Face’s family of compact open language models. His work connects two practical questions: how to build capable models that people can run and study, and how to measure whether AI agents are becoming useful researchers.

Building small models with an open recipe

At Hugging Face, Bakouch worked with Loubna Ben Allal, Anton Lozhkov and a broader research team on compact models built through careful choices about training data and methods. The original SmolLM release, published in July 2024, offered models with 135 million, 360 million and 1.7 billion parameters. Their training corpus combined educational web pages, synthetic textbooks and stories, and educational Python code. Releasing that corpus alongside the models gave other developers material they could inspect and use in their own experiments.

The next generation made data-centric training more explicit. Bakouch was an equal-contribution co-author of SmolLM2, whose 1.7-billion-parameter model was trained on roughly 11 trillion tokens. The team adjusted the proportions of web, mathematics and code data between training stages, using earlier results to guide the next mixture. It also developed FineMath, Stack-Edu and SmolTalk where existing datasets proved inadequate. Small models still required substantial training investment; the benefit was a compact model at deployment time, accompanied by datasets and methods others could build on.

In July 2025, Bakouch and his collaborators released SmolLM3, extending the family to a three-billion-parameter model with six-language support, a context window of up to 128,000 tokens, and selectable reasoning and non-reasoning modes. The engineering choices served concrete purposes: grouped-query attention reduced the memory needed to retain attention keys and values during inference, while selective removal of rotary position embeddings helped extend context. The release included the training recipe so other model builders could examine how architecture and data choices shaped its capabilities.

His collaborative work also reached beyond text. He co-authored the SmolVLM research, which examined how architecture, image tokenization and data selection could make image and video understanding practical with smaller memory budgets. He also co-authored the December 2024 INTELLECT-1 technical report, documenting a ten-billion-parameter model trained through a distributed network of community compute providers.

Bakouch’s educational work complements the models. He helped produce the Smol Training Playbook, covering pre-training, post-training and infrastructure. In his public introduction to the playbook, he emphasized sharing failures alongside successful choices, including the work required to make training run reliably.

Making automated research measurable

At Prime Intellect, Bakouch applies his experience with training experiments to autonomous AI research. His optimizer-speedrun research gives agents a constrained problem: improve the optimization procedure so a small language model reaches a specified validation loss in fewer training steps. Restricting changes to optimizer-related parameters makes proposed improvements testable while exposing how agents choose experiments and interpret uncertain results.

Several concerns distinguish this work:

• Experimental judgment: A failed experiment can reflect a bad implementation, an unlucky run or an idea that only becomes useful after another change. Bakouch’s research examines whether agents preserve promising weak signals, repeat borderline tests and revisit earlier failures as the training recipe changes. These behaviors help explain why agents that discover similar ingredients can achieve different results.

• Statistical verification: Record claims must survive repeated training runs with fixed seeds and a stricter acceptance threshold. This reduces the chance that random variation looks like progress. The experiments also examine whether agents measure uncertainty themselves and use it to decide which ideas deserve further tests.

• Novelty beyond leaderboard gains: Bakouch distinguishes improving an existing recipe from discovering a fundamentally new method. In his earlier experiments with Claude Code and Codex, both agents surpassed the human record, but they could consult human submissions and combine existing techniques. Those experiments and his subsequent research produced useful improvements without yielding a fundamentally new optimizer. He treats that result as a question about the task and the models’ capabilities, rather than proof that automated discovery is solved or impossible.

Bakouch has outlined an AlphaEvolve-inspired discovery loop in which models propose ideas, speedruns test them, and human judgment helps select methods for larger-scale evaluation. Scaling matters because an improvement in a small training run may not carry over to larger models or longer training. He presented this as work being explored, with discovery still an intended outcome. His case for conducting the research openly is practical: researchers outside major laboratories need ways to understand and independently evaluate how AI systems perform research.

Supporting sustained agent work

Bakouch is also a co-author of Prime Agent, an open-source system for sustained coding and evaluation workflows. Its persistent Python environment, retained memories and communicating subagents let models construct tools and carry work across longer trajectories. This extends his research from training individual models to building environments in which models can conduct experiments, retain what they learn and expose their work to human inspection.