1 00:00:02,421 --> 00:00:09,701 GoLive proves that a system exists, but it doesn't prove the people behind the work will actually use it. 2 00:00:10,341 --> 00:00:14,181 or that it will ever deliver a single dollar from the deal model. 3 00:00:14,641 --> 00:00:18,061 I am Michele Grant, creator of Leading at Scale, 4 00:00:18,401 --> 00:00:24,341 and the question every enterprise is confronting right now is the agent works. 5 00:00:24,601 --> 00:00:25,981 Does anyone trust it? 6 00:00:26,421 --> 00:00:30,441 I have spent over 20 years in enterprise transformation 7 00:00:30,441 --> 00:00:38,741 and more than a decade leading M&A integrations across media, tech, and consumer goods. 8 00:00:38,741 --> 00:00:45,941 On those deals, trust was not something we crossed our fingers and hoped for after launch. 9 00:00:46,241 --> 00:00:54,041 We engineered it before day one through early change management and risk management. 10 00:00:54,941 --> 00:01:01,821 So when the platform went live, the teams on the ground actually relied on it. 11 00:01:02,681 --> 00:01:06,621 AI agents demand the exact same muscle. 12 00:01:06,621 --> 00:01:14,581 An agent earns trust when it delivers the value the business budgeted for while staying 13 00:01:14,581 --> 00:01:21,981 safely within the company's approved risk appetite, policies, and regulatory guardrails. 14 00:01:22,861 --> 00:01:29,121 In my experience, leadership gets there by answering three straightforward questions. 15 00:01:29,721 --> 00:01:31,621 What value does it create? 16 00:01:32,201 --> 00:01:34,421 Who is on the hook for it? 17 00:01:34,421 --> 00:01:35,021 And 18 00:01:35,021 --> 00:01:39,581 And what evidence gives it permission to run with greater autonomy? 19 00:01:40,981 --> 00:01:49,081 The moment an agent steps out of a sandbox and into a core business workflow, the entire test changes. 20 00:01:49,081 --> 00:02:00,961 A pilot tests whether the software functions, but daily operations is where we test whether it creates value within the controls set by leadership. 21 00:02:02,341 --> 00:02:06,221 Established AI risk frameworks give you strong principles. 22 00:02:06,661 --> 00:02:11,661 What they do not give you is how to apply these principles to each agent. 23 00:02:12,101 --> 00:02:16,201 For every agent, leadership has to decide four things. 24 00:02:16,781 --> 00:02:18,701 What is it permitted to do? 25 00:02:19,081 --> 00:02:20,381 Who owns it? 26 00:02:20,701 --> 00:02:23,221 What evidence it records? 27 00:02:23,581 --> 00:02:26,181 And what triggers an escalation? 28 00:02:27,181 --> 00:02:30,001 That work is not an engineering task. 29 00:02:30,461 --> 00:02:33,681 We find it is an executive operating mandate. 30 00:02:34,481 --> 00:02:37,241 And today, I'll show you how to meet it. 31 00:02:38,341 --> 00:02:42,041 Let us ground this in a real-world scenario. 32 00:02:42,041 --> 00:02:52,321 own. Picture a $15 billion acquisition targeting 400 million in net synergies. The deal has already 33 00:02:52,321 --> 00:02:58,761 closed and the integration efforts are now underway. The integration management office, 34 00:02:59,001 --> 00:03:07,421 which we'll also refer to as the IMO today, is driving the synergy roadmap across 40 work streams, 35 00:03:07,421 --> 00:03:14,441 covering everything from rationalizing vendor contracts to consolidating ERP systems. 36 00:03:14,781 --> 00:03:21,281 And every single initiative has a business owner accountable for its bottom line savings. 37 00:03:22,521 --> 00:03:31,361 Every week, analysts in the IMO are going to reconcile the numbers against the general ledger 38 00:03:31,361 --> 00:03:36,161 as they track down status from all 40 leads. 39 00:03:36,561 --> 00:03:40,181 They're also comparing actuals against the plan, 40 00:03:40,381 --> 00:03:44,901 then finally assembling the executive deck for the steering committee, 41 00:03:45,301 --> 00:03:50,421 which is going to include the CFO, the chief risk officer, 42 00:03:50,921 --> 00:03:54,241 business unit heads, and the integration lead. 43 00:03:54,841 --> 00:04:00,321 The engineering team builds an agent to automate that first draft. 44 00:04:01,361 --> 00:04:12,181 The agent then ingests the weekly updates, reconciles the ledger, runs variance math against the plan, and flags the initiatives that are falling behind. 45 00:04:13,501 --> 00:04:17,041 Everything looks good, and they decide to run a four-week pilot. 46 00:04:17,661 --> 00:04:22,941 The agent's drafts match the analyst's reports, line for line. 47 00:04:23,221 --> 00:04:25,801 So we can say that the tech works. 48 00:04:26,601 --> 00:04:36,001 Now, the question is, what value does that steering committee need before they can trust it to run the business? 49 00:04:36,741 --> 00:04:46,141 First, the steering committee needs a written value case, agreed upon before the agent ever touches production data. 50 00:04:46,841 --> 00:04:51,541 In our example, that comes down to three specific metrics. 51 00:04:52,021 --> 00:04:54,821 First, let's look at cycle time. 52 00:04:55,801 --> 00:05:04,441 If that weekly report eats five business days to compile right now, you're going to want to set a hard target such as two days. 53 00:05:05,041 --> 00:05:08,581 Your second metric will be analyst capacity. 54 00:05:09,161 --> 00:05:18,441 If four analysts spend half of their week building the stack, that is 80 hours a week the agent gives back. 55 00:05:19,461 --> 00:05:32,061 That time counts as value only when the integration lead assigns it to the work streams that are behind on their synergy targets and need help getting back on track. 56 00:05:32,721 --> 00:05:43,281 Then you only count that value after netting out the compute, licensing, and human review costs of running the agent. 57 00:05:44,321 --> 00:05:49,261 Your third metric is the most important one, detection time. 58 00:05:49,761 --> 00:05:56,281 That is how long it takes from the first sign of a slip in the data to the moment the steering committee sees it. 59 00:05:56,981 --> 00:06:04,721 Cutting the report from five days to two days does not guarantee that the committee sees that slip any sooner. 60 00:06:05,501 --> 00:06:11,241 The agent has to catch it, and the value case measures that directly. 61 00:06:12,381 --> 00:06:16,381 Every single metric requires a pre-deployment baseline. 62 00:06:16,841 --> 00:06:21,701 If the IMO cannot state the baseline today, do not deploy. 63 00:06:22,161 --> 00:06:28,041 If you can't measure where you started, you can never prove the agent moved the needle. 64 00:06:29,121 --> 00:06:35,961 Next, the steering committee needs to know who owns the risk the agent creates. 65 00:06:36,521 --> 00:06:40,641 We are going to apply the three-line model to the agent. 66 00:06:41,241 --> 00:06:45,261 In the first line, management owns the controls. 67 00:06:45,801 --> 00:06:49,061 The integration lead is the accountable executive. 68 00:06:49,951 --> 00:06:54,901 She owns the agent's output exactly the way she owned the manual deck. 69 00:06:55,801 --> 00:07:02,721 In the second line, risk and compliance oversee the agent and challenge how to run it. 70 00:07:03,811 --> 00:07:07,281 How deep they go depends on the agent's exposure. 71 00:07:07,281 --> 00:07:18,601 An internal draft scrubbed by a senior analyst warrants a very different review standard than an autonomous agent publishing figures straight to the board. 72 00:07:19,801 --> 00:07:31,821 In the third line, internal audit provides assurance deciding based on material risk when and how the agent enters the annual audit plan. 73 00:07:31,821 --> 00:07:38,601 The agent also needs one employee assigned to oversee it every day. 74 00:07:39,101 --> 00:07:42,541 In our example, that is the senior analyst. 75 00:07:43,321 --> 00:07:46,221 reviews the draft before it reaches the community. 76 00:07:47,361 --> 00:07:50,161 Her role requires four specific capabilities. 77 00:07:50,721 --> 00:07:54,641 She has to first understand the operational boundaries. 78 00:07:55,101 --> 00:07:58,741 She has to interpret its output with context. 79 00:07:58,741 --> 00:08:11,901 She must have the authority to override or reverse it, and she needs a clear kill switch to shut it down safely the second it behaves unexpectedly. 80 00:08:14,301 --> 00:08:18,881 Accountability also means defining where the agent's authority stops. 81 00:08:19,841 --> 00:08:25,361 I document this in a one-page policy called the Human Works Standard. 82 00:08:25,361 --> 00:08:37,581 For every fork in the process, it maps the decision the human role authorized to make it, the required evidence, and the escalation path. 83 00:08:38,601 --> 00:08:41,001 These decisions exist at every tier. 84 00:08:42,361 --> 00:08:49,001 senior analyst decides whether a variance is an accounting timing difference or a real cash shortfall. 85 00:08:49,941 --> 00:08:54,361 The work stream lead decides how to remediate that shortfall. 86 00:08:54,601 --> 00:09:03,001 And the integration lead is going to decide whether to formally flag an initiative as at risk to the steering committee. 87 00:09:03,781 --> 00:09:11,601 The agent prepares the analysis behind every one of those decisions, and it stops there. 88 00:09:12,101 --> 00:09:16,161 The person who makes the decision owns the consequence. 89 00:09:17,021 --> 00:09:22,221 So keeping these decisions with your team does not slow the agent down. 90 00:09:22,541 --> 00:09:27,221 It can go into use while each role keeps its authority. 91 00:09:28,481 --> 00:09:34,381 one reason to delay deployments is a decision with no assigned owner. 92 00:09:35,741 --> 00:09:42,101 This means if leadership has not agreed who declares an initiative off track, 93 00:09:42,101 --> 00:09:52,121 the agent's flags lead to no action. The third question is where the engineers in this room 94 00:09:52,121 --> 00:10:01,781 come in. The evidence for greater autonomy comes from how the agent is built. I look for three 95 00:10:01,781 --> 00:10:09,861 artifacts. The first is an auditable decision log. For every work stream the agent evaluates, 96 00:10:09,861 --> 00:10:18,361 it records the raw data ingested, the tool calls executed, the prompt and the model versions, 97 00:10:18,821 --> 00:10:27,941 output generated along with the human sign-off. Two engineering choices make that log airtight. 98 00:10:28,741 --> 00:10:37,561 First, your core math and reconciliation rules must run in deterministic finance-approved code, 99 00:10:37,561 --> 00:10:42,301 not generated as conversational prose from an LLM. 100 00:10:43,621 --> 00:10:49,481 Second, treat the model's written summary as a helpful explanation, not proof. 101 00:10:49,921 --> 00:10:52,921 Your real proof lives in the telemetry. 102 00:10:53,981 --> 00:10:55,441 What records were pulled? 103 00:10:55,781 --> 00:10:58,021 What deterministic tools ran? 104 00:10:58,261 --> 00:11:00,401 And who signed the review? 105 00:11:02,321 --> 00:11:06,081 If reported savings do not match the general ledger, 106 00:11:06,081 --> 00:11:09,201 Agent does not touch the number. 107 00:11:10,421 --> 00:11:12,861 flats discrepancy and roots it up. 108 00:11:14,421 --> 00:11:17,601 second artifact is the model inventory record. 109 00:11:19,141 --> 00:11:21,841 captures the accountable executive. 110 00:11:23,281 --> 00:11:23,841 lineage. 111 00:11:25,321 --> 00:11:28,261 Dependencies across source systems. 112 00:11:28,741 --> 00:11:30,381 Validation results. 113 00:11:30,801 --> 00:11:31,841 Escalation path. 114 00:11:32,401 --> 00:11:34,041 And the kill switch runbook. 115 00:11:34,041 --> 00:11:35,021 Most 116 00:11:35,341 --> 00:11:48,781 Most importantly, it documents your regression triggers, which are any updates to our prompt, base model, or source API that require full revalidation before running in production again. 117 00:11:51,261 --> 00:11:55,441 third artifact, this is going to be the Autonomy Scorecard. 118 00:11:56,321 --> 00:12:01,121 The steering committee uses it to grant autonomy in four stages. 119 00:12:02,361 --> 00:12:04,461 Stage one is shadow mode. 120 00:12:04,801 --> 00:12:08,581 The analysts build the report as they do today. 121 00:12:09,561 --> 00:12:13,141 agent runs right alongside them and the team compares the two. 122 00:12:14,001 --> 00:12:17,041 Stage two is the supervised draft. 123 00:12:17,561 --> 00:12:24,241 The agent compiles the first draft and the senior analyst reviews every single line item. 124 00:12:25,481 --> 00:12:36,681 Stage 3 is the exception and sample, where the analyst reviews land initiatives plus a random sample of the claimants. 125 00:12:37,181 --> 00:12:41,741 Then we have stage 4, where it's exceptions only. 126 00:12:42,441 --> 00:12:49,281 The analyst reviews only high-risk anomalies, supported by periodic independent sampling. 127 00:12:49,281 --> 00:13:00,961 Pay close attention to stage 4, however, because once humans only look at exceptions, your biggest vulnerability is what the agent failed to flag. 128 00:13:02,111 --> 00:13:12,761 So I advise tracking missed material issues through independent sampling, monitoring error severity, and watching your override rate carefully. 129 00:13:13,581 --> 00:13:18,801 A falling override rate does not automatically mean the agent is getting smarter. 130 00:13:19,281 --> 00:13:22,501 It can also mean the depth of review has split. 131 00:13:23,041 --> 00:13:27,941 So independent sampling tells you which one of these you are seeing. 132 00:13:28,741 --> 00:13:34,621 The scorecard puts those control metrics side by side with your value metrics, 133 00:13:35,021 --> 00:13:39,741 cycle time, analyst hours deployed, and your detection time. 134 00:13:40,081 --> 00:13:47,701 And with that single page view, the steering committee can make clean, efficient data-backed decisions 135 00:13:47,701 --> 00:13:57,681 to either promote the agent, hold it where it is, step it back, or down to supervised review stage. 136 00:13:58,521 --> 00:14:02,281 So let's come back to the three questions. 137 00:14:03,501 --> 00:14:07,501 Ask them of every agent you build, fund or govern. 138 00:14:08,641 --> 00:14:10,641 What value does it create? 139 00:14:11,361 --> 00:14:13,621 Who is accountable for it? 140 00:14:13,621 --> 00:14:18,041 and what evidence permits it to operate with autonomy. 141 00:14:19,181 --> 00:14:25,421 In our M&A example today, the agent does not magically create a 400 million in synergies. 142 00:14:26,401 --> 00:14:32,681 Procurement, supply chain, and operating leaders capture that value through the gritty work of integration. 143 00:14:34,101 --> 00:14:36,021 what the agent delivers is clear. 144 00:14:37,081 --> 00:14:40,161 Verified evidence weeks ahead of schedule, 145 00:14:40,161 --> 00:14:47,201 giving leadership the runway to intervene while the deal's synergies can still be captured. 146 00:14:48,321 --> 00:14:52,101 Building an agent that works is just the first step. 147 00:14:52,861 --> 00:14:59,181 Answering those three questions is what earns it the trust to run at scale. 148 00:14:59,861 --> 00:15:00,881 Thank you.