RAG Evaluation Is Broken! Here's Why (And How to Fix It)
AI Engineer World's Fair 2025 · 10:58
Enterprise AI agents and foundation models
AI21 Labs develops AI systems for enterprises, with Maestro helping teams optimize agents for production. Its tools adjust execution strategies, optimize the combination of models and agent harnesses, and dynamically route calls among models to balance cost and output quality. The company’s work spans agent orchestration and foundation models, although its 2026 restructuring concentrated the business around Maestro.
Founded in 2017 by Ori Goshen, Yoav Shoham and Amnon Shashua, AI21 is led by co-CEOs Goshen and Shoham. Its lasting technical contributions include Jamba, which combines Mamba state-space models, Transformer attention and mixture-of-experts layers to address memory use and throughput in long-context inference. The original model supported a 256,000-token context window. AI21 announced a halt to Jamba development in May 2026; Shashua subsequently expressed his intention to leave the company and board in July.
The May 2026 restructuring reduced AI21’s workforce from 180 to about 70 employees. Its Nebius relationship is a technology licensing partnership; both companies denied characterizations of a sale or acquihire. AI21 said it had signed Maestro contracts worth tens of millions of dollars with international customers, including Nebius. Earlier, its completed $208 million Series C in 2023 brought total funding to $336 million at a $1.4 billion financing valuation.
AI Engineer World's Fair 2025 · 10:58
Affiliations reflect their AIE appearances, not necessarily current employment.
Start here to understand how questions answerable from individual chunks can make benchmarks poor tests of aggregation, illustrated through financial and FIFA World Cup examples.
Yuval Belfer · Niv GranotAI Engineer World's Fair 2025
Use this talk to compare routing, tool use, MCP, and ReAct with dynamic planning for complex instruction following.
Yuval BelferAI Engineer World's Fair 2025
Belfer and Granot describe clustering documents and populating schemas during ingestion so queries can run through SQL. Their examples expose normalization challenges, including changing country names and jointly hosted tournaments.
Belfer's account of AI21 Maestro connects dynamic planning and replanning with Best-of-N generation and decisions informed by expected cost, latency, and success probability.
Affiliations reflect each recorded session, not necessarily current employment.