# Trusting Agents in Policy-Driven Workflows Evan Kormos · AI Engineer NYC 2026 · October 14 [Watch recording](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3124s) [00:52:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3124s) Hi, everyone. [00:52:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3125s) My name is Evan Kormos. [00:52:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3127s) I've been delivering, designing, building software for 20 years, the last 15 enterprise [00:52:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3133s) financial services. [00:52:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3135s) I am a player coach at Coinbase. [00:52:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3139s) and we're here today to talk about trusting agents and policy-driven workflows. [00:52:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3150s) So let's start with the policy, and it's pretty boring. [00:52:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3154s) I'm going to go over here for my notes. [00:52:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3158s) So our starting point centers around the policy, [00:52:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3161s) and this sets the rules for how work gets done within the organization. [00:52:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3167s) and including especially risk and compliance. [00:52:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3172s) Think of policy as the original institutional intent [00:52:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3176s) or the original system prompt. [00:52:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3179s) ideally we have this written down. [00:53:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3181s) It's a little bit boring, so stick with me. [00:53:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3185s) Using language like must, may, and should. [00:53:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3189s) And again, these help define the rules and decisions [00:53:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3192s) about how the system operates in the organization. [00:53:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3197s) Separate from policy, we have the actual running system. [00:53:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3203s) So the first thing we're going to do is separate these two, both the build state and the run state, at two different moments in time. [00:53:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3213s) So typically we're either defining or changing our policy, but the system is constantly running and it's constantly moving and changing. [00:53:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3225s) So we could then sort of imagine a compilation event that, oops, sorry, I'm skipping ahead. [00:53:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3232s) It's constantly changing. [00:53:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3237s) So it's evolving quickly with these product-driven changes. [00:54:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3240s) Everything is expected to still be roughly bound by the policy, but ultimately we separate [00:54:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3246s) this from the run state and the build state. [00:54:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3251s) So inside the system, what does it look like? [00:54:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3255s) We have a familiar model, [00:54:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3257s) if we have any enterprise folks in the room here. [00:54:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3261s) People, process, and technology. [00:54:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3264s) But where really do agents fit into this mold? [00:54:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3269s) So I see them as an operational model [00:54:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3272s) cutting across all three. [00:54:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3274s) They work with people, [00:54:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3276s) execute parts of the process, [00:54:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3279s) themselves, and they use and are made up of technology. [00:54:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3284s) So for our talk today, we're going to be focusing here where agents meet process, [00:54:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3291s) what work we give them, what boundaries we set, and how we build trust in their outputs. [00:55:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3303s) So let's dive into your workflow. [00:55:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3307s) And it's high level and seems basic, but if you haven't, there are good reasons to map this out. [00:55:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3315s) So maybe a change has to be approved before it goes live, or a customer has to go through verification before their account is opened. [00:55:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3324s) We expect work to flow in a specific and gated direction. [00:55:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3331s) So I believe it makes, agents make it worth revisiting what we can do in parallel. [00:55:41](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3341s) Previously, some work used to cost too much or be too complex to coordinate in this way. [00:55:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3346s) But now we can run independent reviews alongside each other, bring the results together, but [00:55:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3352s) keep the required checks in place. [00:55:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3357s) So this is a directed acyclic graph, or a DAG. [00:56:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3361s) It is a way to represent this workflow or process, [00:56:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3364s) the steps, the dependencies, [00:56:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3367s) for a flow that moves forward without looping back on itself. [00:56:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3372s) Apache Airflow, if you're familiar, [00:56:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3374s) is an example of open-source software [00:56:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3376s) that represents workflows, [00:56:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3379s) dependencies as DAGs in Python code. [00:56:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3391s) So, this Python code, these DAGs can be used to define and manage a number of policy-driven [00:56:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3398s) processes like, for example, commercial lending. We could also use a DAG to represent the workflow [00:56:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3408s) flow to list a token on the exchange in a specific region, if you're into that. Or we [00:56:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3416s) could use a DAG to represent a regulated flow of delivering software, which is a hot topic. [00:57:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3422s) One we'll spend a little bit of time on, I think that we've seen a lot of adoption within [00:57:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3428s) agents, within software delivery, and ultimately we're looking to leverage that same efficiency [00:57:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3434s) and scale in other areas of the business [00:57:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3436s) and close the adoption gap. [00:57:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3442s) So let's stick with the software delivery example [00:57:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3446s) for a little bit. [00:57:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3448s) What happens inside one of these nodes of the process? [00:57:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3452s) And I'll give you a hint. [00:57:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3454s) It's a loop. [00:57:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3464s) oversimplified, but we have a basic version [00:57:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3466s) of what's happening within the, [00:57:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3468s) let's say this is the code review process, right? [00:57:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3473s) So we're going to essentially do a number of steps, [00:57:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3477s) and essentially once we realize the work, [00:58:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3480s) we recognize that the work is good enough, [00:58:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3482s) we're going to allow it to pass on to the next step. [00:58:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3485s) And that's essentially the function of the code review node. [00:58:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3491s) The work doesn't always end up okay. [00:58:14](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3494s) So sometimes we have to take the feedback from the code review [00:58:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3497s) and iterate on it until the work is good enough to pass. [00:58:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3508s) So this is a representation of a directed cyclic graph, or DCG. [00:58:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3514s) So, as far as open source software, something like LangChain, LangGraph would be what I would [00:58:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3522s) say is helpful to manage the state of DCGs. [00:58:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3528s) And this is an oversimplification. [00:58:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3531s) These graphs can be very complex. [00:58:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3536s) So zooming back out, the inner flow of the node, we have people and agents working together. [00:59:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3542s) together, on the outer flow, we have a policy boundary. [00:59:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3548s) But not everything goes to plan. [00:59:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3551s) So we have to handle exception flows. [00:59:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3553s) We don't want the work to get stuck and we don't want to design for every edge case either. [00:59:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3559s) The exception flows will monitor and see kind of what happens if you're shipping code that [00:59:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3565s) gets cancelled or doesn't meet security standards. [00:59:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3569s) that's something you should want to know about. [00:59:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3574s) Volume shifts across the flow once we start to add agents into the mix. [00:59:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3579s) And we've seen this with software delivery, [00:59:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3582s) and we'll see it in other processes as we bring agents into those processes as well, [00:59:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3586s) where now all of a sudden if you're reviewing code faster, [00:59:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3590s) you need to design it quicker, or you need to do the security scan faster. [00:59:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3595s) So we see the volume shift across the flow. [01:00:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3600s) And this is where it's really important that we have visibility into the process overall from a metric standpoint. [01:00:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3606s) Because as we start to supercharge and parallelize and add more efficiency to the process, we start to develop more and more bottlenecks. [01:00:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3615s) And so the operational visibility of the process is extremely important. [01:00:20](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3620s) Much more important than the authoring and the building of your workflow tool. [01:00:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3625s) so we want to visualize the process [01:00:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3630s) so we want to add parallel capacity [01:00:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3632s) by still keeping the controls [01:00:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3633s) and so this is where we'll add more additional capacity [01:00:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3643s) okay [01:00:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3643s) so we're doing some work [01:00:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3646s) we're checking it [01:00:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3647s) we're using the feedback again to refine it [01:00:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3649s) until the work is good enough to move on [01:00:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3651s) but why is that hard? [01:00:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3653s) That seems like a pretty easy thing to do. [01:00:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3655s) But as I mentioned before, these graphs get actually pretty complex, [01:01:01](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3661s) and we'll dive a little bit more into why this is difficult. [01:01:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3666s) And if you're familiar, one challenge with AI engineering [01:01:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3669s) is the variability of inputs and outputs. [01:01:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3672s) And I believe that higher quality inputs [01:01:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3677s) will result in better outputs at a lower compute cost. [01:01:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3683s) but what are quality inputs? [01:01:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3694s) So we'll generally speak to it as context model, maybe. [01:01:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3698s) We can nitpick that. [01:01:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3699s) But this isn't our area of focus today. [01:01:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3702s) We're going to focus more on outputs. [01:01:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3704s) But we can't really talk about quality of outputs [01:01:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3707s) without talking about the quality of inputs. [01:01:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3713s) So ideally, we've established a trusted context for our agent, [01:01:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3717s) and there's some references on that. [01:02:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3720s) But one thing we can do in this model is inject the DAG state into our agent [01:02:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3726s) and let it know kind of where it fits into the bigger picture. [01:02:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3732s) Okay, so now we've added even long time horizon goals, [01:02:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3738s) and that emphasizes the need to optimize the cost and efficiency of this loop. [01:02:30](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3750s) So the check is where we decide whether the work is good enough to move forward. [01:02:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3755s) We need to define what good looks like. [01:02:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3757s) That's the hard part. [01:02:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3759s) And we need to have a reliable way to evaluate it. [01:02:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3763s) Otherwise, more passes don't necessarily mean better results. [01:02:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3768s) the loop is really only as trustworthy as the check. [01:02:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3771s) So if we want to talk about quality, [01:02:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3773s) let's talk about the check and let's talk about quality. [01:02:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3778s) Okay, I think building with AI is an opportunity [01:03:03](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3783s) to improve product quality overall. [01:03:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3785s) I like some of the conversation about durable software. [01:03:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3790s) So I'm a believer in quality. [01:03:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3792s) And in traditional software, [01:03:16](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3796s) we've built confidence mechanisms in place [01:03:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3799s) to give us confidence that the software is of high quality. [01:03:24](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3804s) And that's done with a variety of mechanisms, [01:03:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3807s) but primarily systematically with test cases, unit tests, [01:03:32](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3812s) checking these as a part of our delivery flow [01:03:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3814s) to ensure that when we make code changes, [01:03:37](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3817s) mistakes aren't made that break other things. [01:03:39](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3819s) So this is a confidence mechanism that we've put in place [01:03:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3822s) with traditional software. [01:03:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3826s) The agents are probabilistic, and so the same request can produce different answers. [01:03:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3834s) than one answer may be acceptable. [01:03:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3838s) And we still need evidence, but exact output comparisons alone really aren't good enough. [01:04:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3845s) need to build representative data sets and evaluate them and define what good looks [01:04:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3852s) like. [01:04:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3852s) we need to run these evaluations in repeated series, [01:04:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3859s) and this is a statistical analysis, right, [01:04:22](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3862s) where we actually look at the outputs in volume [01:04:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3866s) and not just as a direct comparison. [01:04:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3871s) So combining with evals as sort of your runtime logs [01:04:34](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3874s) of what happened, or sorry, traces, [01:04:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3876s) which are sort of your runtime logs of what happens, [01:04:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3880s) we've built this confidence mechanism [01:04:42](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3882s) to complement traditional software. [01:04:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3889s) And ultimately, this is what we're going to be talking about [01:04:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3892s) a little bit more in terms of driving confidence in the output, [01:04:55](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3895s) not only putting this mechanism in place, [01:04:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3897s) but how do we begin to turn it on. [01:05:05](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3905s) So where does the agent belong? [01:05:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3907s) Back to the lending example. [01:05:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3909s) we can do KYC [01:05:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3911s) financials in parallel and sequence [01:05:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3913s) like credit memo, risk review [01:05:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3915s) documentation, decision, all these things [01:05:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3917s) in our DAG [01:05:19](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3919s) looking at each node we want to ask [01:05:21](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3921s) some certain questions like how valuable [01:05:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3923s) would it be to have an agent [01:05:26](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3926s) drive this node [01:05:27](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3927s) versus a human drive this node [01:05:29](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3929s) and we can sort of look [01:05:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3931s) at some of these basic questions like [01:05:33](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3933s) can we check it against the policy easily [01:05:35](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3935s) can it run in a bounded loop [01:05:36](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3936s) Can we trace it? [01:05:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3938s) Can we fall back to a human? [01:05:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3940s) If we can answer yes to these type of questions, [01:05:43](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3943s) we can kind of have a good idea going in [01:05:46](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3946s) that this particular node may be better suited for automation. [01:05:49](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3949s) But throughout the DAG, we should look to drive agent assistance. [01:05:54](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3954s) So what we're really talking about is not whether or not AI is involved, [01:05:57](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3957s) but whether or not AI we expect to take over at some point. [01:06:06](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3966s) on the decision you would probably agree that we probably need that to be human driven this is [01:06:11](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3971s) essentially the check against the policy um so this is going to be a little bit more difficult [01:06:17](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3977s) i think to hill climb but what do we mean by hill climb all right how do we turn this on [01:06:28](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3988s) So day one, we start supervised. [01:06:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=3991s) And as we have agents and humans working together as a part of the process, we can manually verify the output to ensure that it is of high quality. [01:06:44](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4004s) This is where we refine our data sets and ensure that we have the right confidence mechanisms in place. [01:06:50](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4010s) We've defined what good looks like with the domain subject matter experts. [01:06:53](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4013s) and that they're beginning to verify it. [01:06:59](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4019s) When measured confidence reaches a policy threshold, [01:07:02](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4022s) we can move towards sampling. [01:07:04](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4024s) And what that means is humans are looking at a slice of the output [01:07:08](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4028s) and not necessarily all of it. [01:07:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4030s) We begin more systematic evaluation through traces [01:07:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4035s) in order to ensure that what the humans aren't seeing [01:07:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4038s) is still meeting the quality bar. [01:07:23](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4043s) And then finally, once we clear the confidence threshold, humans handle more of the exceptions of the process, but still act as participants in refining the data sets and ensuring the quality outputs. [01:07:40](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4060s) Okay, so a few takeaways from this. [01:07:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4065s) DAG and DCG. [01:07:47](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4067s) Basic graph theory. [01:07:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4068s) but ultimately how do we build these processes that exist today [01:07:52](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4072s) and where people are working them and bring agents into the mix? [01:07:56](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4076s) And the recipe really is to keep it deterministic on the outside [01:08:00](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4080s) and make it adaptive on the inside. [01:08:09](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4089s) The next is quality through evaluation. [01:08:12](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4092s) And this is really establishing a complementary confidence mechanism. [01:08:18](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4098s) So it's not a feeling about a model. We can actually provide research and statistics [01:08:25](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4105s) about the quality of our agents. [01:08:31](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4111s) So this builds confidence and inherently trusts output. [01:08:38](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4118s) Quality is a team sport. AI engineers own the pipeline to taking product signals [01:08:45](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4125s) from all directions [01:08:48](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4128s) and deliver applications that are well-built [01:08:51](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4131s) and utilize ML operations behind the scenes. [01:08:58](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4138s) Finally, agents accelerate across the operating model. [01:09:07](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4147s) I've included some references on the final slide, [01:09:10](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4150s) things that I've spoken about today. [01:09:13](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4153s) This concludes my presentation. [01:09:15](https://www.youtube.com/watch?v=jiUC6IUMdvc&t=4155s) Thank you very much.