Will Agent Evaluation via MCP Stabilize Agent Networks?
AI Engineer World's Fair 2025 · 14:11
LLM evaluation and AI quality control
Root Signals, now operating as Scorable, builds evaluation and monitoring software for teams developing AI applications, chatbots and agents. Its calibrated AI judges score responses for quality and policy adherence, helping teams identify failures in production and test behavioral changes when models or prompts are updated. Developers can create custom evaluators from their own policies and examples, then integrate scoring into applications and release pipelines.
Founded in 2023, the company counts Ari Heljakka among its founders. Its engineering work includes Root Judge, a locally deployable evaluator fine-tuned from Llama-3.3-70B-Instruct. The model detects context-grounded hallucinations in retrieval-augmented generation systems and compares responses using customizable rubrics, with explanations for its scores. Its local deployment option lets organizations evaluate sensitive material without sending it over the public internet.
Scorable serves enterprise AI teams, software vendors building specialized bots and LLM consultants. Its commercial offering includes hosted evaluation software sold through AWS Marketplace under contracts with evaluation allowances. Alongside that service, Root Judge's available weights let developers integrate and customize the model for their own evaluation workflows, providing a choice between managed software and local model deployment.
AI Engineer World's Fair 2025 · 14:11
Affiliations reflect their AIE appearances, not necessarily current employment.
Start here to understand how single-turn and multi-turn evaluation fit into a structured approach to assessing agents.
Ari HeljakkaAI Engineer Summit 2025
Use this demonstration to learn how MCP connects agents to persistent judges, including a Cursor example that optimizes marketing copy.
Ari HeljakkaAI Engineer World's Fair 2025
Heljakka frames EvalOps as improving both the agent's operative LLM workflow and the separate workflow that judges it. His evaluation map spans grounding, faithfulness, tool-use validation, and API-specific metrics.
A hotel-reservation demonstration adds an MCP-accessible policy evaluator to a Pydantic AI agent, preventing it from recommending a competing hotel.
Affiliations reflect each recorded session, not necessarily current employment.