# The Build-Operate Divide: Bridging Product Vision and AI Operational Reality Jeremy Silva · Chris Hernandez, Chime AI Engineer World's Fair 2025 [Talk page](https://ai.engineer/talks/1__V4KTv_Gw-build-operate-divide-bridging-product-vision-ai) · [Video](https://www.youtube.com/watch?v=1__V4KTv_Gw) ## The Build-Operate Divide: Bridging Product Vision and AI Operational Reality A promising AI product needs more than a launch: evaluations, human feedback, and clear quality ownership turn rapid development into reliable improvement. ### What happens after the first version ships? [0:00](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=0s) How do you turn a promising AI concept into a product that reliably delivers its intended value? Getting something built is only part of the problem. Operational challenges can prevent a good concept from reaching its potential, and addressing them requires three connected capabilities: **evaluations, human review, and a team structured to act on what they reveal**. Jeremy Silva approaches this problem as the product lead at Freeplay, which helps teams operationalize AI products. Chris Hernandez brings the perspective of leading speech analytics at Chime, with a background spanning customer experience and machine learning. Their shared concern is what it takes to keep improving quality after the initial build. ### Faster development increases operational demands [1:12](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=72s) The transition from traditional ML to generative AI changes how much work is needed before a team can begin learning from a product. In Silva’s comparison, traditional ML demands substantial starting data and long training cycles. Generative AI lets teams use the intelligence already present in base models to make smaller organizational data assets useful. | Development constraint | Traditional ML | Generative AI | |---|---|---| | Starting data | Substantial data needed to begin | Smaller organizational assets can be useful | | Model preparation | Long training cycles | Existing base-model capabilities | | Iteration | Slower development cycles | Faster experimentation | Lower barriers to entry make iteration faster. That speed increases the need for a quality operations function: the ability to change a product quickly must be matched by the ability to assess those changes. ![Slide titled “The Evolution from Traditional ML to Gen AI,” with a downward arrow beside “Barriers to entry” and an upward arrow beside “Speed of iteration.”](https://ai.engineer/api/talk-essay-media/1abb76e6804d6a5d39b986a437bb58f6c96e481ee6ce95fd53be6de7fc78cdd3) The shift to GenAI lowers barriers to entry and increases iteration speed. ### Crossing the quality chasm [2:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=129s) Silva reports a recurring pattern across dozens of enterprise teams at Freeplay: an initial prototype, sometimes already shipped as a production V1, encounters a quality chasm before it becomes a reliable V2 that delivers real customer value. The missing ingredient is reliability, and his prescription is repeated iteration. The loop connects three kinds of work: 1. **Monitoring:** observe the product’s behavior. 2. **Experimentation:** develop and try changes. 3. **Testing and evaluation:** assess those changes through human review, automatic evaluation, or both. Silva connects product quality to a team’s ability to move through this loop repeatedly and quickly. Evaluation is broad here; it includes the human work of examining outputs as well as automated checks. At scale, keeping that loop moving becomes an operations responsibility. Someone must make the review, experimentation, and testing work happen consistently. The irony is that delivering high-quality AI still requires substantial human effort. Faster model development makes that work more central to the product. ### Confident errors need human judgment [3:34](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=214s) Hernandez illustrates the problem with a joking anecdote: he asks an LLM who invented Wi-Fi, and it answers Abraham Lincoln. He immediately qualifies the example as perhaps extreme. Its purpose is to make a familiar failure vivid: even capable models can produce wrong answers with considerable confidence. Unchecked hallucinations can mislead customers and misinform decisions. In healthcare, even a small error can carry substantial risk. **Human-in-the-loop review** gives people a place to guide the system while generative AI handles much of the work. The need becomes especially clear when a response depends on nuance, empathy, or context—qualities that fast content generation alone does not guarantee. Scale changes the consequences. An isolated mistake may appear manageable; repeating that failure across many interactions can make it dangerous. Hernandez’s warning is that teams can end up scaling risk alongside output when they expand automation without human judgment. ### Feedback needs reviewers [5:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=309s) A human flag or correction has value beyond catching the immediate mistake. It supplies a signal for refinement, reinforcement, or retraining, helping future behavior move closer to human expectations. **Review is a feedback mechanism**, not just a safeguard. Collecting a correction and changing the system are separate steps: a label supplies evidence for improvement; it does not automatically update model weights. Hernandez describes teams without enough people to review thousands of model outputs, limiting both improvement and measurement of current quality. Model-graded evaluations help, but he argues that they still need a human element. Without review capacity, the team struggles both to identify what should change and to understand how well the current system works. Existing quality and customer experience teams offer a practical source of that capacity. Contact-center staff already evaluate interactions at scale, identify edge cases, and define what good service looks like. Their expertise can inform AI behavior directly. As their responsibilities expand, they move from measuring outcomes toward shaping what the system should do next. ### Expanding QA beyond retrospective audits [6:52](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=412s) As generative AI enters operations, quality assurance gains a forward-looking responsibility. QA teams become prompt testers, model shapers, and AI performance monitors as well as auditors. Their ability to assess nuanced conversations and surface edge cases gives them a useful starting point for this work. Hernandez traces that expansion from the familiar work of auditing, coaching, and compliance. Looking back roughly a quarter-century, he describes QA professionals listening to calls and evaluating interactions. Those skills remain the foundation. The diagram’s overlap contains auditing, scorecards, insights, and edge-case surfacing; its surrounding scope includes strategic AI impact and model monitoring on one side, and prompt testing, annotation, and human-in-the-loop design on the other. Testing prompts and tagging outputs extend the existing discipline into shaping model behavior. ![Venn diagram titled “QA at the Center of GenAI Adoption.” The left circle lists strategic AI impact and model monitoring; the overlap lists auditing, scorecards, insights, and edge case surfacing; the right lists prompt testing, annotation, and HITL design.](https://ai.engineer/api/talk-essay-media/756682128d7788cdb4d565bddea9d5535fa95571bab397a58b392a51c3333bc5) QA spans model monitoring, auditing, prompt testing, annotation, and HITL design. This work also opens participation beyond ML and engineering teams. Recognizing a good output does not require knowing how to build the model pipeline, just as being a good wine connoisseur does not require knowing how to make wine. Domain expertise remains valuable even when its owner does not implement the system. Contact-center and CX teams already possess much of the judgment needed to evaluate the conversations AI products generate. ### Give AI quality an owner [9:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=549s) Silva calls an emerging function the **AI quality lead**, while noting that he rarely sees that exact job title. He expects the role to become more formalized; Chris and his team are an example of the function already being performed. People can enter it from product, operations, or engineering. What matters is deep customer and domain understanding combined with the ability to diagnose quality problems systematically. The daily work makes that ownership concrete: - **Evaluation:** label data, define evaluation criteria, and maintain testing datasets. - **Experimentation:** run experiments and tests, and optimize prompts. - **Coordination:** identify quality issues and prioritize fixes with product managers and engineers. These responsibilities connect an understanding of customer needs to a repeatable process for improving outputs. ![Slide titled “The AI Quality Lead Role,” with columns for key attributes and day-to-day duties, including identifying quality issues, defining evaluation criteria, managing testing datasets, optimizing prompts, and prioritizing fixes with PMs and engineers.](https://ai.engineer/api/talk-essay-media/eaef0fec15b12e69573ec1d4195ae8f3f0cf83624355d4274bb519d428711cb9) The AI quality lead role combines customer understanding with practical evaluation and improvement duties. Writing production code is often outside this role. With suitable tools and team structure, a quality lead can make hands-on contributions through prompt engineering, evaluation, and experimentation without implementing the production application. The team design must make those contributions possible, rather than treating coding as the only direct route to improving the product. Silva reports success with one or two empowered people performing this function in smaller deployments, while larger enterprises need a substantial quality team. This is an observation about staffing at different scales, not a workload-based sizing rule. A small organization can begin by assigning clear ownership; increasing scale calls for enough capacity to sustain the same quality work. ### Place review where it matters, then keep operating [11:13](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=673s) When review resources are scarce, Hernandez recommends prioritizing high-risk areas and interactions where trust matters most. Put people at actual decision points, where their judgment can affect the outcome, rather than adding review merely for appearances. Bring operations and CX teams into the product lifecycle early so they can help define quality, build **golden sets** of reference examples, and test real-world edge cases. **Launch begins the operating responsibility.** Track performance, flag hallucinations, measure impact, and iterate. Celebrating a release is reasonable, but the team must be ready to determine whether deployed outputs continue to meet expectations. That ongoing work is what makes the improvement loop part of the product’s operation. Scaling therefore depends on people as well as technology. QA, operations, support, and frontline teams need to be strategic partners in the AI system’s development and operation. Hernandez closes by framing generative AI scale as an operational liability and responsibility alongside the technical challenge: embedding quality, human feedback, and the right people in the loop is how faster building becomes better building. ## Resources - [Human Labeled Evaluations](https://docs.freeplay.ai/core-concepts/evaluations/human-evaluations) — Freeplay’s current guide to expert labeling, production spot checks, ground-truth datasets, review queues, and calibration of automated evaluators. - [Sam’s Club and Chime put people first in their AI implementations](https://www.customerexperiencedive.com/news/sams-club-chime-people-balance-ai-qualtrics-x4/743156/) — Contemporaneous reporting on Chris Hernandez’s approach to targeted contact-center automation, downstream impact, and access to human support. ## Slides ### The Build-Operate Divide Operational challenges can keep promising AI product concepts from reaching their potential. The talk explores evals, human review, and team structure as ways to bridge that gap. [0:18](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=18s) [Title slide: The Build-Operate Divide: Bridging Product Vision and AI Operational Reality.](https://assets.ai.engineer/sites/aiecode2025/sha256/61bdf422245cb3fd980e9796cf356a547cd2b23ab5007d284b01d98c30531b10.webp) The Build-Operate Divide: Bridging Product Vision and AI Operational Reality AIE aws ### Introduction Jeremy Silva leads product at Freeplay. Chris Hernandez leads the speech analytics team at Chime. [0:48](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=48s) [Jeremy Silva, product lead at Freeplay, and Chris Hernandez, manager of AI speech analytics.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0f66aa28610e7a121315aa660d89a9b370f34e3990c39f3c8f5a16bb3e670d9.webp) Introduction Jeremy Silva Product Lead at Freeplay Chris Hernandez Manager of AI Speech Analytics AIE AI Engineer 2025 Microsoft smol ai ### The Evolution from Traditional ML to Gen AI Jeremy Silva describes how base models let teams use smaller data assets, lowering the barrier to entry and increasing iteration speed compared with traditional ML. [1:12](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=72s) [Barriers to entry decrease while the speed of iteration increases.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd61f15774e49fbf2d0e58fd9a5aafd604cadf50b68dfbc15bcbd960406e42ae.webp) The Evolution from Traditional ML to Gen AI Barriers to entry ↓ Speed of iteration ↑ AIE AI Engineer 2025 Microsoft smol ai ### Gen AI Quality is a function of iteration speed After a prototype and V1 launch, teams cycle through monitoring, experimentation, and testing and deployment, with evaluation at the center, to improve quality toward V2 and beyond. [2:38](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=158s) [A timeline from prototype to production leads to an evaluation cycle of monitoring, experimentation, and testing and deployment, followed by iteration and improvement.](https://assets.ai.engineer/sites/aiecode2025/sha256/485e830895e852e45d741704c60ad010ddfbfaffb789f8c9db2b1a947d250702.webp) Gen AI Quality is a function of iteration speed 1 Build A Prototype 2 Launch V1 Into Production Test & Deploy Monitor Evaluation Experiment V2 And Beyond: • Fast Iteration • Continuous Improvement • Customer Adoption • Business Impact AIE Microsoft smol ai ### The Role of Human-in-the-Loop (HITL) Chris Hernandez jokes about an LLM naming Abraham Lincoln as Wi-Fi’s inventor, then argues that confident hallucinations need human oversight. [3:32](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=212s) [Title slide introducing the role of human-in-the-loop, attributed to Chris Hernandez.](https://assets.ai.engineer/sites/aiecode2025/sha256/4dd6d12e54f4e8c30637629bbb5ea58387798297f740388268c18805796d716e.webp) The Role of Human-in-the-Loop (HITL) Chris Hernandez AIE AI Engineer 2025 aws ### Why Human-in-the-Loop Is Essential in GenAI LLMs generate content quickly, but can fall short of real-world reliability when nuance, empathy and context matter. Human oversight helps bridge that gap. [4:43](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=283s) [Slide highlights that human oversight bridges the gap between LLM capability and real-world reliability.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2ed2f3bef28f1922369441781103d3032c0af08fd4fb83eb0fffa74523eb0fc.webp) Why Human-in-the-Loop (HITL) Is Essential in GenAI Human oversight bridges the gap between LLM capability and real-world reliability. Without HITL, we risk automating errors at scale. Feedback from humans improves the model over time. Model Output Human Review Feedback Model Refinement AIE AI Engineer 2025 Microsoft smol ai ### Why Human-in-the-Loop Is Essential in GenAI Without human review, model errors can scale. The speaker also treats flagged or corrected outputs as feedback for improving models over time. [4:57](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=297s) [Slide highlights the risk of automating errors at scale without human oversight.](https://assets.ai.engineer/sites/aiecode2025/sha256/7b743250d45936ec6eb72566d48e9e3453efa6df6d7a32f55bb8b1b42cad5a28.webp) Why Human-in-the-Loop (HITL) Is Essential in GenAI Human oversight bridges the gap between LLM capability and real-world reliability. Without HITL, we risk automating errors at scale. Feedback from humans improves the model over time. Model Output Human Review Feedback Model Refinement Interface text: AIE AI Engineer 2025 Microsoft smol ai ### Quality in the Age of GenAI As GenAI enters operations, the speaker sees QA teams moving beyond auditing toward shaping models, testing prompts, and monitoring AI performance. [6:50](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=410s) [Section title: Quality in the Age of GenAI, above colored wave lines.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3220e9626fa1aebbbacef4b2b9108abdb828041eeb1a0fd387b850330d78087.webp) Quality in the Age of GenAI AIE AI Engineer 2025 aws ### The Evolving Role of QA in Contact Centers The timeline traces QA from call listening and compliance to insight reporting and LLM evaluation, then casts QA as an AI quality gatekeeper in 2025 and beyond. [7:39](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=459s) [Timeline from manual QA through insight-driven QA, LLM evaluation, and strategic AI enablement.](https://assets.ai.engineer/sites/aiecode2025/sha256/8622bf7d3bd086f29ede6eeacced20d7282f311ee57d35c92573631ee3054f0e.webp) The Evolving Role of QA in Contact Centers 2000-2018 Manual QA Scorecards, compliance, audits Call listening 2019-2023 Insight-Driven QA Sentiment, behavioral tagging Trend reporting 2024 - Now Hybrid QA + LLM Evaluation Prompt testing, labeling golden sets Model feedback loops 2025+ Strategic AI Enablement HITL design, model monitoring QA as AI quality gatekeeper AIE aws ### QA at the Center of GenAI Adoption The diagram places auditing, scorecards and edge-case insights between model monitoring and prompt testing. The speaker describes QA staff expanding into testing prompts and tagging outputs. [8:07](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=487s) [Overlapping circles show model monitoring, auditing and scorecards, edge-case insights, and prompt testing and annotation.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc604776b6ed4fca35f9b91300bca07ccc662da334dcbc59d236e1b16802fe38.webp) QA at the Center of GenAI Adoption Strategic AI Impact & Model Monitoring Auditing & Scorecards Insights & Edge Case Surfacing Prompt Testing, Annotation, HITL Design AIE Microsoft smol ai ### The AI Quality Lead Role AI quality leads pair customer knowledge with systems thinking: they diagnose production issues, define evaluations and improve prompts. The speakers say people can do this hands-on work without writing production code. [9:07](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=547s) [Two columns list AI quality lead attributes and duties, including evaluation criteria, testing datasets, and prompt optimization.](https://assets.ai.engineer/sites/aiecode2025/sha256/b034e7f9fd8045d84c7b39508375ab9fa04655afe0d5d6bd7670d6ed280722fe.webp) The AI Quality Lead Role Key Attributes • Deep understanding of the customer need • Ability to identify systemic quality issues • Systems thinker and natural problem solver • Strong motivation to learn technical skills • Effective cross-functional communicator Day to Day Duties • Analyze production data to identify quality issues • Define evaluation criteria • Build and manage testing datasets • Optimize prompts • Work with PM/Engineers to prioritize fixes Interface text: AIE Microsoft smol ai [9:17](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=557s) [Two columns list the attributes and day-to-day duties of an AI quality lead.](https://assets.ai.engineer/sites/aiecode2025/sha256/bf4e403fd971b882739d3faec80c67d17895afe5dca543f83e5051658098ea1c.webp) The AI Quality Lead Role Key Attributes • Deep understanding of the customer need • Ability to identify systemic quality issues • Systems thinker and natural problem solver • Strong motivation to learn technical skills • Effective cross-functional communicator Day to Day Duties • Analyze production data to identify quality issues • Define evaluation criteria • Build and manage testing datasets • Optimize prompts • Work with PM/Engineers to prioritize fixes Interface text: AIE aws ### Scaling Responsibly: Human-in-the-Loop When resources are limited, prioritize human review at high-risk, high-trust decision points rather than adding it for show. [11:12](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=672s) [Scaling Responsibly slide highlights human-in-the-loop where it matters.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ae9099c540d751e938ec7ae57bb4921e9418a3ed05870e59855fa2dcee86aa8.webp) Scaling Responsibly Human-in-the-Loop Where It Matters Monitor, Measure, Improve Operationalize Quality from Day 1 Empower the Right Teams AIE Microsoft smol ai ### Scaling Responsibly: Operationalize Quality Bring operations and CX teams into the lifecycle early to define what good looks like, build golden sets, and test real-world edge cases. [11:29](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=689s) [Scaling Responsibly slide highlights operationalizing quality from day one.](https://assets.ai.engineer/sites/aiecode2025/sha256/ad6c28f20c36027f6da4ab39e183f9b1d957afec228b5e842019735181409271.webp) Scaling Responsibly Human-in-the-Loop Where It Matters Operationalize Quality from Day 1 Monitor, Measure, Improve Empower the Right Teams AIE Microsoft smol ai ### Scaling Responsibly After launch, teams should track performance, flag hallucinations, measure impact, and iterate. The slide places monitoring alongside human review, early quality work, and empowering the right teams. [11:45](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=705s) [Scaling Responsibly slide highlights monitoring, measuring, and improving.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ae1ab253494533032f84a59eaac987defffa2ef4c1551d018c598f68380f467.webp) Scaling Responsibly Human-in-the-Loop Where It Matters Monitor, Measure, Improve Operationalize Quality from Day 1 Empower the Right Teams AIE Microsoft smol ai [11:53](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=713s) [Scaling Responsibly slide highlighting “Monitor, Measure, Improve”; three other points are dimmed.](https://assets.ai.engineer/sites/aiecode2025/sha256/79d3fdcaf678a6a275a2089aa518fa7187fb262f50b207a004038d2867be2e7f.webp) Scaling Responsibly Human-in-the-Loop Where It Matters Monitor, Measure, Improve Operationalize Quality from Day 1 Empower the Right Teams AIE Microsoft smol ai ### Scaling Responsibly: Empower the Right Teams Scaling GenAI calls for QA, operations, support and frontline teams as strategic partners, embedding quality and human feedback into the system. [12:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=729s) [Scaling Responsibly slide highlighting “Empower the Right Teams”; three other points are dimmed.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a9ddc040ce03806f246a0269274c0b7c3fef51edea21f36cd5b7c367223d8bc.webp) Scaling Responsibly Human-in-the-Loop Where It Matters Monitor, Measure, Improve Operationalize Quality from Day 1 Empower the Right Teams AIE Microsoft smol ai ## Full transcript [0:00](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=0s) [upbeat music] Um, we titled this talk the build operate divide, 'cause we've ob-observed this troubling trend where, like, good AI product concepts fail to reach their full potential due to operational [0:25](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=25s) challenges. So today, Chris and I are going to be talking about some of the learnings we've gleaned from, like, our experiences on the front lines operationalizing AI products at scale. [0:34](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=34s) By the end of this talk, we hope you're going to leave with an understanding of how you can kind of bridge that gap between product concept, um, and operational reality by understanding how to deliver quality through evals, human review, and how you might build your teams around that. [0:48](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=48s) So a little bit about ourselves. My name is Jeremy. I lead product at a company called Freeplay, um, which exists to help solve a lot of these operational problems and, and help companies ship great AI products. [0:59](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=59s) Hi, everyone. My name is Chris Hernandez. I lead our speech analytics team at Chime. Uh, I've been in the industry for about ten years, uh, from a CX perspective and, uh, about nine years on the ML space. [1:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=69s) So excited to be here. [1:12](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=72s) So both Chris and I kind of came from the traditional ML world into the GenAI world, and I thought it was, like, helpful to maybe take a note and talk about, like, what we feel like has changed in this transition. [1:24](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=84s) Um, so I think the biggest difference in my mind is this decreased barrier to entry, right? In the traditional ML world, like, you need tons of data to even get started. [1:32](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=92s) There are these long model training cycle times, and that barrier to entry just kind of like goes down significantly, um, in a GenAI world, right? Like, you can now use the intelligence of the base models to be able to, like, make use of and leverage, uh, smaller data assets within your organization. [1:49](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=109s) And then what comes along with that is this, like, increased iteration speed, which is, like, as the barrier to entry goes down, the speed at which you can iterate starts to go up. [1:59](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=119s) And this increased iteration speed starts to accentuate the need for, like, a high-quality ops function. Um, so we want to take a look at why. [2:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=129s) Working across, like, dozens of different enterprise teams at Freeplay, we've noticed this common trend emerge, which is companies will build an initial prototype, maybe they'll even ship a V1 of that thing into production, but they inevitably hit this sort of quality chasm as they try and go from a V1 to a V2 that really drives true value [2:27](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=147s) for the customer. And there's this reliability problem in there. And what we've seen is, like, the only way that you really cross that quality chasm that, and get to a reliable V2 is through iteration. [2:41](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=161s) And so we talk about this iteration loop as you move from monitoring to experimentation to testing, evaluation, and I, I use that word broadly speaking, human review, auto-evaluation, like all these things all kind of come into play. [2:55](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=175s) Um, but what happens is your product quality becomes a direct function of your ability to move through this loop, right? The faster you can iterate, the more times you can move through this loop, the better your product quality becomes. [3:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=189s) So it, ops kind of sits at the foundation of this, and especially as you start to scale, right? Like, your ability to move through this loops is, like, ends up being an ops function. [3:19](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=199s) And the, like, not so subtle irony here is that to deliver high-quality AI products, you actually need a ton of human elbow grease. And I'll pass it to Chris here to talk a little bit more about the importance of human experts in this process. [3:34](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=214s) Cool. Awesome. Um, you know, I made the joke that one day I was like, you know, the first time I saw LLMs, we started playing around with it. You know, I typed in like, uh, "I wonder if it actually knows like basic things like, uh, who invented Wi-Fi?" [3:45](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=225s) And then it spit out Abraham Lincoln. I was like, "Oh my goodness- [laughs] ... we're in some trouble." But, uh, all jokes aside, and maybe not that extreme, it is a good reminder that LLMs, and even the smartest ones, can make mistakes, uh, and they often do it with a lot of confidence. [3:59](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=239s) Um, so, uh, that's of course what we call hallucination. And, uh, in the GenAI space, um, it, it, at times, like you got to make sure that, you know, we don't leave it unchecked and that these hallucinations can actually be tricky and risky. [4:11](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=251s) Uh, they could also mislead customers, and they could misinform decisions. And when you think of like companies or industries like healthcare, uh, just a small hallucination could have a lot of risk in it. [4:21](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=261s) Um, so that's where human-in-the-loop comes in. That's why it's important to talk about this today. And human-in-the-loop kind of ensures while GenAI does the heav-heavy lifting, uh, humans are still there to steer the ship and, and drive, drive in the right direction. [4:34](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=274s) Um, so today we're going to talk a little bit more about like human-in-the-loop, why it's important. I don't think this is like groundbreaking, like everyone understands human-in-the-loop. It's been around forever, but just reiterating on the importance of it. [4:45](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=285s) So LLMs right now are just, you know, we know that they're good at, uh, generating content, and they do it really, really quickly. But without humans, they often fall short of real-world reliability, especially when nuance, empathy, and context is required. [4:58](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=298s) Without human-in-the-loop, you're also not scaling productivity, you're scaling risk. And so a [REDACTED:marital_status] hallucination in isolation might not seem like a big idea or big deal, but at scale it could be dangerous. [5:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=309s) Um, so, uh, the last point I want to make on that is also each time a, a human flags or corrects an output, it's a signal that we need to be using to help retrain, reinforce the models that we see. [5:21](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=321s) And that's of course how, uh, you know, models are evolving over time. So I want everyone to think of human-in-the-loop not as like a safeguard, it, but as a feedback mechanism or feedback engine, if you will. [5:32](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=332s) So every model output that goes through a human review, that feedback then gets fed for refinement and, and of course, uh, improving the model. And over time, this loop brings AI closer to real human expectations and behaviors. [5:45](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=345s) But there is a challenge and, and speaking with many different teams, the challenge is that we don't have enough people to do all these reviews. Like model-graded evals and evals in general are great, but you still need to have the human element in there. [5:56](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=356s) So most teams don't have enough people to review thousands of the different outputs that you have. And that makes it really, really hard to improve on the models and even measure, uh, the current state of the models that you have. [6:06](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=366s) But there is good news, and so this might not seem like a, a clear path in some of the organizations that you all might be, be part of. Um, but we already have teams that are kind of trained to do this type of work of human-in-the-loop, and those teams are like your quality folks or CX folks within [6:21](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=381s) the operations side of the business. And so especially when you look at call centers, they're already experts in doing the jobs that you need them to do, which is evaluate interactions at scale, spotting a, spotting edge cases and defining what really, what good looks like. [6:35](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=395s) So in this age of A, uh, GenAI, their roles from my perspective is not shrinking, it's continuing to evolve. Um, and they're not just, right, they're not just measuring outputs anymore, uh, or outcomes. [6:47](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=407s) Uh, I think they're really shaping, like, the future and what's next. [6:52](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=412s) So as GenAI becomes, like, more embedded in the operations, quality is also evolving. It's no longer about auditing what's already happened, uh, but it's also about shaping what's gonna happen next. [7:02](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=422s) So the shift change, uh, in the QA teams who are in this ops space, they're not just scorekeepers anymore. They're becoming model shapers, prompt testers, and AI performance monitors. [7:12](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=432s) And if there's any group that's already built for this kind of work, as I mentioned before, you might want to look at your ops team and see if there's anyone in the QA contact centers that could also help expand your projects, uh, even further. [7:23](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=443s) Um, they al- already know how to evaluate the nuanced conversations. They also know how to surface these edge cases like I mentioned before. [7:30](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=450s) So talk a little bit about, like, the evolution of just quality in general or CX. Um, they've been for the most part focusing on auditing, coaching, compliance, uh, but as automation continues to scale, these roles aren't disappearing, but they're actually transforming. [7:46](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=466s) When you look back, you know, 25 years ago, um, right, you know, the, the main roles were just QA professionals were listening to phone calls and evaluating interactions. But we see that as, like, continu- like, a continued evolution where automation is now coming, uh, into play, and these folks are, like, transforming their skill sets to help solve [8:05](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=485s) for larger problems. [sniffs] The diagram that we have here kind of just shows, like, where the QA scope is today and how it's expanding to the GenAI space. At the core, we have what QA has always done, uh, which is auditing interactions, capturing quality behaviors. [8:20](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=500s) But now, as you can see, the scope... The, the scope itself is expanding, and QA professionals are testing prompts now, and they're tagging outputs, and they're really helping shape the model, uh, behavior that we're expecting. [8:31](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=511s) Uh, and there is a beauty in the GenAI space, uh, which is it opens the doors to non-technical folks. Uh, traditionally, uh, only the ML teams, the engineers are the ones that are heavily involved in the outputs and deciding what's gonna, you know, uh, come from it. [8:44](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=524s) Uh, but now, um, uh, you don't need to know how to build, uh, the model pipeline to know what, you know, a good output looks like, uh, just like you don't need to know how to, um, make wine to be a good wine connoisseur. [8:54](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=534s) Uh, the, the outputs or the expertise still matters and just as valuable. Uh, and then the contact center CX teams are, again, already equipped with this. And so, um, it might be something to, to look into and just see how you could expand that reach even further. [9:09](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=549s) So one of the things Chris is talking about here is, like, something we have observed, like, working across a number of custom- customers as well, which is this stor- sort of emerging role of, like, the AI quality lead. [9:20](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=560s) And importantly, like, I've almost never seen it actually called this, but it seems to exist at companies who are having, like, a lot of success in the GenAI space. [9:29](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=569s) And I expect this role to actually become more formalized and gain more traction, right? Like, Chris and his team are an example of this role. Um, and importantly, like, people in this role can come from a variety of different backgrounds, product, ops, engineering. [9:43](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=583s) But, like, the key attributes of what makes someone a good AI quality lead is someone who first and foremost has a deep, deep understanding of the customer need in the domain, and then importantly is a systems thinker and is able to, like, systematically think about how to diagnose and solve these quality problems. [10:01](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=601s) What that looks like day to day is this person is doing a lot of these just kind of, like, this new skill set, labeling data, writing evaluation criteria, running experiments and tests and things like this, and prompt engineering. [10:15](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=615s) And you'll notice that, like, there is something notably missing from that, which is, like, these are often not the people who are writing production code. But I think what has changed so significantly is writing production code is not the only way now that you can contribute in, like, a really hands-on way. [10:33](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=633s) All of these things, like, given the right tool set and the right structure of your team, you can contribute to, like, this iteration loop, the prompt engineering, the evaluation, all this kind of stuff, without necessarily being the one, like, writing the production code day to day. [10:47](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=647s) Chris is talking about, like, how you do this at scale, but we've also seen a lot of success where companies will have one person or two people in this role, um, especially when you have a smaller footprint, and that goes a long way, you know? [10:58](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=658s) So I think what, what Chris is painting is, like, in larger enterprises and as you start to scale this stuff up, you really need, like, a meaningful quality team. [11:05](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=665s) But you can do this by, like, kind of empowering a [REDACTED:marital_status] or couple individuals in this role. [11:13](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=673s) Cool. So I'm just gonna leave you all with a, a, a few last points. You know, I think while human-in-the-loop is important, and, like, we've just spent the last few minutes talking about it, uh, if you don't have the resources, I think, you know, looking at high-risk, high-trust areas is abs- like, an absolute must. [11:28](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=688s) And so insert human-in-the-loop at decision points and not just for show. The next piece is, like, continue to bring your co- like, your ops teams and CX teams into the life cycle early to help what, you know, define what good looks like, uh, so they can help build out golden sets and, uh, tests against real world, real [11:45](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=705s) world edge cases. Um, and then this is another point I wanted to make too, is just, like, the, the fact of, like, launch is not the finish line. Uh, track performance, flag hallucination, me- measure impact, and iterate. [11:55](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=715s) Uh, time and time again, I see teams, like, celebrating, which is great to see celebration of, like, a product actually launching or solution being actually launched. Uh, but the important part there that it's not the end. [12:04](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=724s) It's the beginning, where you need to set up your teams to make sure that the outputs are performing how you expect them to perform. Um, and last but not least, scale is not just about tech anymore. [12:13](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=733s) I think it's about people. And so leveraging QA, ops, uh, and support, and frontline teams, I think as a strategic partner in the GenAI space is gonna be what makes teams successful. [12:23](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=743s) And if there's just one key takeaway today, it's that scaling GenAI isn't just a technical challenge anymore. It's an operational, uh, liability and responsibility. Um, when you embed quality and then human feedback into that loop, the right people, uh, into your GenAI systems, you're not just building faster, but you're also building better. [12:41](https://www.youtube.com/watch?v=1__V4KTv_Gw&t=761s) Um, so anyways, thank you, guys. Appreciate it. [audience applauding] [upbeat music]