[
  {
    "startMs": 12440,
    "text": "All right. Hi, everyone. I'm Shy. I am a developer advocate at Temporal. Uh, I cannot see anyone in the room, so if you boo me or, like, wanna throw tomatoes or something, I, I will have no idea. Um... All right. Oh, clickers are difficult. There we go."
  },
  {
    "startMs": 33640,
    "text": "I guess I'm not using the clicker today. So, uh, agents, they are everywhere. I'm guessing if you are in this room, you are probably building an agent or shipping an agent. Does that sound about right? Yeah? Um, so last year KPMC, uh, conducted a survey, and they figured out that, uh, we have seen $3 trillion in terms of corporate productivity gained in 2025, and that's from their, uh, agentic AI report. Uh, and from that three trillion, it's really fin- uh, financial services that are"
  },
  {
    "startMs": 63500,
    "text": "leading the way here. So 94% of fintech are saying that AI is core to how their businesses operate, and they've decided that AI isn't just a side project, but, but it's like, it's part of the business now, like, this is how we operate. Uh, and 62% of those teams lose measurable time, uh, or revenue to reliability issues. So they haven't deci- you know, deciding that AI is, uh, that something we're doing isn't necessarily the same as shipping it. There's definitely still a gap between"
  },
  {
    "startMs": 93420,
    "text": "those two things there. Now, to break it down, uh, part of this is the shape of the work. Finance runs on long-running workflows, not, not on individual models. Um, you know, for some examples of their process, you can think of loan origination, KYC and AML, fraud review, uh, paying people. Each of these processes tan- spans multiple days, multiple systems, and usually multiple people. Uh, so there's many, many steps. There's many, many systems, and there's real money involved,"
  },
  {
    "startMs": 123460,
    "text": "so they can't mess anything up. And now imagine that we're gonna go ahead and put an agent into the middle of that process. So, uh, at Temporal, we wanted to understand how things are actually going, so we conducted a survey. This is our second year running this survey, the State of Development Survey. Uh, we surveyed 554 engineers and engineering leaders who are actively using AI agents right now. Every number on this slide deck is either from that report, uh, or it'll be qualified in the bottom left from the"
  },
  {
    "startMs": 153400,
    "text": "report that I'm pulling it from. So to start things off, we can talk-- take a look at the demographic of that survey. 48.4% have 11 plus years of experience working in industry, so we're not talking about hobbyists. And then the remai-- another third of that segment also has six to 10 years. So none of these people are hobbyists. They're all professionals working, um, every day. Um, it's a mix of engineers, AI engineers, data engineers, platform teams, IT leaders. Two-thirds of them were from the US. A third of them were from the UK and"
  },
  {
    "startMs": 183040,
    "text": "EMEA. Uh, and what we found, uh, is four gaps. I promised you all that this talk was about gaps in the title. Uh, and the data shows that there are four major gaps, and I wanted to walk through each of them and then close with kind of one big idea. So let's go into gap one, and that's the gap between, uh, usage versus production. So daily agent use from our 2025 version of this survey w- had us at about 47.3% of the folks that we talked to were using agents daily. Uh, today, uh,"
  },
  {
    "startMs": 213120,
    "text": "nearly a year later, we've doubled that to 80.8%, uh, of users are using agents in their day-to-day. Um, now, using agents isn't necessarily the same thing as shipping agents. Uh, and so if we take a look at those teams again and ask them what they're doing, about half of them are running those agents in production, or they say that those agents are core to how they ship. And then the other half of that is still experimenting or still only really using assistants. So we've got"
  },
  {
    "startMs": 243300,
    "text": "49.1% of the folks that we talked to, half of the folks that we talked to are running agents in production or saying that those agents are core to how, how things ship. Um, so that's a huge gap, right? Like, that's a 30% gap between 80% of the people are using agents every day and 50%, 49.1% are actually shipping them and using them in production, so far fewer people in production than actually using them daily. A- and something seems to keep breaking on the way to production, and let's talk about what that is, and"
  },
  {
    "startMs": 273340,
    "text": "that brings us to gap number two, which is trust versus reality. Now, one of the things we asked folks in this survey was, how much do you actually trust your agent output? And trust is really high, actually. I'm really impressed by this number. Uh, a quarter of the folks that we, we surveyed trusted the agent output almost completely. Most people trusted it somewhat, and almost no one disrupt it-- distrusted it. I did not get asked to, to participate in this survey because I'm in that complete somewhat distrust category myself. So I'm actually really surprised by this"
  },
  {
    "startMs": 303300,
    "text": "number from that survey. Um, yeah, 85.5% trust agent output at least somewhat, uh, and, and that's gonna be an important number. Let's come back to that 85.5 in a little bit. So we also asked, how often do your agents have issues? Um, the pink bars are daily or worse, and that's a lot of pink for a number that, uh, 85% of people trust. That's a lot more pink than I would expect for an 85% trust rating. Uh,"
  },
  {
    "startMs": 332640,
    "text": "41.1% of those folks hit those agent issues daily or more, so that's four in 10 of the people that we talked to see issues every day, and then 16.4 hit them every hour. So one in six of the people that we surveyed were getting an agent issue every hour. And then that brings us back to that trust, that 85.5% gap versus the 41.1%, uh, system breaking. Uh, so,  so we trust the output and the system, but it still breaks every day. Uh, both of those things are true at the same time."
  },
  {
    "startMs": 362908,
    "text": "And getting good at agents doesn't seem to, to make this go away. You can't skill your way out of this. Um, we were seeing teams that call themselves successful were actually 1.7 more times likely to hit issues continuously, so be in that hourly or daily category. Uh, and it's not because they've run more agents, right? The report found that agent count and issue frequency aren't actually correlated in any way. I'm guessing, and the team at Temporal are guessing, that maybe those folks are just looking harder"
  },
  {
    "startMs": 392948,
    "text": "for issues, and so they're finding those issues some more. That's definitely something we need to dig into a little more when we do next year's version of this survey. But the important, I guess, takeaway is that even with how much you trust your agents, failure isn't going away. Uh, you have to account for it. You have to plan for it. So the goal isn't zero failures, right, when we're using agents in production. It's just how do we make it so these failures don't cost us anything? Now, a really interesting stat is saying that 40%, uh, of the teams that we talked to says security is truly the thing that"
  },
  {
    "startMs": 422948,
    "text": "is blocking them from autonomous agents, and that's its own talk. Mason, uh, from the Temporal team and Kim from the Keycard team actually gave a talk about this yesterday on the workshop day, so I'm gonna defer to the recording of that talk, uh, rather than going to it myself, um, as well. And let's jump into that third gap, the prototype, uh, that we- we're probably all seeing and working on last year versus the enterprise deployment. You know, customers are actually touching it. That prototype to production-ready code system. And most teams going from an AI prototype"
  },
  {
    "startMs": 453048,
    "text": "wanna go to production-ready code, uh, really fast. We're seeing it in, like, really low numbers. Uh, 51.3%, uh, are able to get a- an AI prototype to production ready in hours or faster. Uh, that's like startup speed, right? That's traditional startup speed. Uh, this falls apart though if you work somewhere big. Uh, so if you work somewhere, uh, five thousand plus employees, it tends to take months, uh, and the data breaks down at that point. The biggest companies, about one in five says it takes months."
  },
  {
    "startMs": 483008,
    "text": "Everywhere else, it's about one in twenty saying that it takes months to get an AI agent deployed into production. So big companies, uh,  are shipping at very different speeds. We're using the same tools, uh, at very different speeds. Half are shipping in hours, and then at these large companies, we're doing it one in five, taking five months. Um, big companies, regulated work, uh, we're talking about banks, we're talking about insurers, we're talking about payment companies. I'm guessing there's a lot of those in the room right now, and those companies all need something really--"
  },
  {
    "startMs": 512908,
    "text": "need a bunch of things that are really important, right? They need approvals. They need audit trails. They need systems that will not lose a payment. My read is that's where the five months are going, into the auditability and the trust of, of all of those things. And the survey doesn't break that down, right? But it's not bureaucracy for its own sake, or at least I hope it's not. I hope it's, it's bureaucracy for something productive and, like, getting to that reliability bar. Which brings us to that last reli-reliability gap, uh, seeing versus recovering, and this is one of the ones that was in the title, right?"
  },
  {
    "startMs": 543108,
    "text": "So we asked, \"What limits your team's productivity?\" Number one is tracking state, and number two is debugging. The models aren't in the top two, uh, at all. Um, 35%-- .7% were saying, uh, state, uh, tracking state across services is their top blocker, and then 30.9% says that it's isolating and debugi-debugging issues. 22.2% are lo-- uh, of that survey are losing progress and redoing work because of crashes and restarts. So that's one in five losing work. And then"
  },
  {
    "startMs": 572848,
    "text": "last year, in a different survey, um, 27% called their state management fragile. This year it's the number one blocker. Uh, different surveys, different questions, um, so it's not a trend line, but the story is consistent. State got more important as agents got more real. And one of the wildest responses that we got was when, when an agent breaks, where you go for help, and the number one answer to that being YouTube is kind of wild to me. I don't really code with YouTube, and so this is very"
  },
  {
    "startMs": 602768,
    "text": "different from me, uh, and the way that I build things. Um, but that's what debugging looks like when your system can't tell you what happened, is you end up going to YouTube and trying to find someone that can. Observability right now just tells you your agent failed. Most teams have that, right? Traces, logs, LLM observability tools. But reliability means that failure doesn't cost you anything."
  },
  {
    "startMs": 628328,
    "text": "Uh, only 13% of the people that we talked to were very confident that they can observe and debug AI workflows at scale, and then 22% of the people that we talked to lose progress because of crashes. And so the gap between seeing, uh, and recovering is a person reading a trace at two AM in the morning and then fixing their state by hand. And that's not realistic, right? Like, that's not a realistic thing to expect, uh, is someone fixing things at two AM in the morning. So I wanna show a quick little demo, right, of what this looks like, and I'm gonna move twenty-five"
  },
  {
    "startMs": 658288,
    "text": "thousand-- twenty-five hundred dollars. This is a synthetic customer, but it's a real temporal workflow. It's a real LLM running on this laptop. I have Qwen three six B, uh, nine bi- nine B running on this laptop right now. Uh, and I'll show you a demo of what we're about to go through. But basically, we're running a payout workflow. So this is gonna be a nine-stage workflow, uh, which is gonna be much shorter than I think it would be in, like, in production. Uh, money is gonna move. Uh, the money is gonna look kind of weird. Uh, and so an agent is gonna jump in and is gonna do a fraud"
  },
  {
    "startMs": 688248,
    "text": "investigation. The agent is gonna just take a look at the transaction and decide, do we need to escalate this to a human being, or is this good to just wave it through? Uh, and then we're gonna have a human in the loop that actually makes the decision and decides what to-- uh, whether or not to make the final call of this transaction getting approved or not. We're gonna validate the request, reserve the funds, check in with the FX quote, and those funds will be reserved, so the real state will exist in another system. And then I'm gonna go ahead and kill-- uh, sorry.  And we're gonna have that agent do the"
  },
  {
    "startMs": 718128,
    "text": "real work, do those tool calls, uh, and then I'm gonna kill the process. Um, so I'm gonna just tell the worker to die. I'm gonna stop the worker, uh, and so our process will die, uh, and that could be for any reason, right? Like, it could be a timeout, it could be a deploy error, we could run out of memory. I'm gonna just intentionally kill it to simulate a worker crash, and we'll see what we lose. Um- What we would lose traditionally is that the funds would be reserved with no payouts, uh, and a brief would get produced that never actually makes its way to the approver. So we're gonna have to re-spend our"
  },
  {
    "startMs": 748052,
    "text": "LLM tokens traditionally to regenerate that AI, uh, draft or summary for the check. Um, in that tech- in, like, a traditional stack, all of that is gonna live in process memory. And when that worker crashes, it's all gone. And so the reserve funds are still reserved, it's just not in our system. It's in the bank system. So traditionally, we'd have to write recovery code, we'd have to build state tables, we'd have to build retry logic, we'd have to create reconciliation jobs, and that's all code that's unrelated to the core business logic that I'm building to just do a payment transaction,"
  },
  {
    "startMs": 778492,
    "text": "and that's something that 30% of our respondents complained about, is that they're building all of this code unrelated to bus- business logic. Or you could use a platform that remembers you, and, uh, that's where Durable Execution comes in. So Durable Execution is a temporal concept that code essentially should run to completion even when the process doesn't. We know that crashes happen, but with Temporal, state survives, progress survives, uh, and every step is recorded in an event history that looks like this. And so we've also recorded"
  },
  {
    "startMs": 808412,
    "text": "the AI in this state. So this is a timeline that'll- we'll see in a second live that's a run from r- the payout demo that we're gonna run in a second. Uh, each of those steps, uh, each of those AI steps, choosing the investigation, the tool calls that the agentic loop is gonna make, uh, for the lookup of the customer and then drafting the brief, each of those is its own individual activity with a recorded result, and the yellow bar is gonna be our, our essentially activity waiting for a human approval. Um, the workflow code is deciding what's next, and it's"
  },
  {
    "startMs": 838152,
    "text": "deterministic. Anything non-deterministic or side effecting like the LLM is gonna be run in an activity. So a tool call or a bank API call is gonna be an activity. And then the person when they approve or reject the transaction is gonna be an event too. It's a signal, is what we call them in Temporal, which is, like, recorded in history, uh, like everything else is recorded in the Temporal history. Even though the worker's not running, that signal is still gonna get captured, uh, and get processed. So when the worker dies, another worker is able to resume from the last previous success. Uh,"
  },
  {
    "startMs": 868731,
    "text": "yeah. And the history lives in the Temporal Service, uh, and we're not running our LLM tool calls multiple times, so we're not wasting tokens, uh, on multiple tool calls. We're gonna retain the agent's work, uh, and not redo it. Uh, if the call was mid-flight while the worker died, we would retry that activity, but everything before that is, uh, is kept. Um, and that matters. Eight in 10 people say that cost limits how they use their agents, or being able to hold onto the agent's memory, uh, is really helpful. Rerunning an agent after every crash is a lot of money that you're"
  },
  {
    "startMs": 898552,
    "text": "burning right there. Uh, so let's go into the quick demo and..."
  },
  {
    "startMs": 905212,
    "text": "Yeah. So this is my, uh, payment transfer. So, uh, I'm gonna go ahead and set it to $2,500. I'm gonna say that it's a new recipient. I'm gonna say it's an unusual amount for this customer. And just to make sure that, uh, our AI flags it as something that needs an investigation, we're gonna go ahead and set the customer record to say that there's an unresolved account alert. I'm gonna go ahead and start this job, which is gonna trigger this new workflow, and you can see down here in the timeline, all of those nine steps that we talked about are happening. Um, so we're gonna reserve our funds, validate, uh, the AI is doing an agentic"
  },
  {
    "startMs": 935212,
    "text": "loop and deciding, okay, we're gonna look at the customer profile, we're gonna look at the payouts, and then we're gonna draft a breach-- uh, a brief for a human being to review. So I've got an Approve or Reject button that I can click right here, but before I click them, uh, like I said, I am going to intentionally kill my worker. So I'm just gonna hit Stop, and you can see that there are no workers running at this point. So I'm gonna go ahead and approve my transaction, and we can see the approval gets logged as a signal. Now, traditionally, that would be something that just gets lost. Um,"
  },
  {
    "startMs": 965172,
    "text": "we'll get a timeout in a second, but that's not really a big deal, uh, 'cause we're gonna have a retry policy that'll handle this automatically. So as soon as I turn on my worker, it's just gonna pick up that signal and resume the behavior and continue on from that last successful event. So we'll start a worker. This will take a second."
  },
  {
    "startMs": 984472,
    "text": "Of course it takes a little longer  when I'm demoing it live. Usually, it's instantaneous. Um, let me just go ahead and give it a refresh."
  },
  {
    "startMs": 994952,
    "text": "You know what? I have a video for just this scenario, so we can go ahead and just skip straight forward in the video and see what it's like, uh, if the video plays."
  },
  {
    "startMs": 1008012,
    "text": "Yeah. Let me see if I can skip to the middle of this."
  },
  {
    "startMs": 1013792,
    "text": "Oh. Oh, it's in the other tab. What am I doing?"
  },
  {
    "startMs": 1022172,
    "text": "So I'm gonna approve while the worker is dead, uh, and then I'm gonna go ahead and resume the worker right below. Uh, and it should go ahead and just, uh, pick things up and, like, resume, uh, resume. So the agent is gonna investigate, write the brief, uh, and then I give the approval, and then, you know, I can go ahead and crash the worker again while we're waiting for the bank to approve the transaction or mark that it was completed. Again, the signal logs, and then whenever I turn the worker back on, it's gonna pick it up and rerun again. So, so this is what happened according to the Event"
  },
  {
    "startMs": 1052212,
    "text": "History, and let me full screen that again for y'all. Oh. There we go. Um, so reading from top to bottom, each of our AI activities ran once, they attempted once, they worked once, and they didn't run again. We didn't rerun that workflow. We didn't respend those tokens. We just ran it once, and so we got that one-time cost. Uh, our approval signal came through while there were no sig- while there were no Workers running. Uh, and then our bank submission also, you know, came through while no Workers were running, and both of those things happened on"
  },
  {
    "startMs": 1082000,
    "text": "new Workers. We stopped the Workers each time, uh, and then we got new Workers that just picked things up. So zero Workers, uh, the payout waited, uh, the approval landed, and we didn't have to rerun anything from the beginning. So what are the full list, uh, of what  those engineering leaders are saying, uh, slow them down? Um, our demo just handled seven of the top, uh, use cases right there that they gave us for problems. Um, so state, debugging, cost, retries, approval, long-running work, and crash recovery. Um, and we can take a look at the code for each"
  },
  {
    "startMs": 1112000,
    "text": "of them. So state management in Temporal is the number one blocker... Or sorry, was the number one blocker in that survey. In a workflow, state is just fields. It's plain variables. There's no database that I built. There's no cache. After a crash, the replay just rebuilt them exactly. Uh, our LLM calls, uh, would cost us normally for multiple runs. In this run, the agent, we maintained the state, so we only ran it once. The tool run was held, was sustained, so we didn't need to spend any new tokens. Retries"
  },
  {
    "startMs": 1141540,
    "text": "themselves were a policy, not a loop. We declared the retry policy, uh, in our, uh, configuration once, and then it applied to every single call in that transaction. There's no loop anywhere in that code. Um, the, uh... This is probably the most important slide for the room. So the agent itself gave us advice. The workflow itself owned the money and the state, and the AI had a really bounded, uh, attempt to, to jump in, and if the model had failed, the payout would have continued"
  },
  {
    "startMs": 1171420,
    "text": "without the brief. A model outage wouldn't have degraded anything. Um, we had our human approval, uh, and we could have set that timer for as long as we wanted, uh, and it would have been the same code. Um, we could have waited for a year and resumed that Worker, and it would have been fine. And then I didn't write any crash recovery code. That was all built into Temporal. Um, so debugging. Uh, in most stacks, observability and recovery are two separate recor- records that drift apart, and then a human brings them together. Here, we were able to have the history that we debugged in the"
  },
  {
    "startMs": 1201320,
    "text": "UI be the same history that our Worker replays, which closed gap four. So we had seven blockers, and we solved them with one workflow. And this isn't just a demo, so there are people using this stuff in production for real right now. OpenAI's Codex runs on Temporal. Uh, the coding agent that you use a lot of the day is running on Temporal. Uh, Replit runs their Replit agents on Temporal. Their engineering team says it's a bad user experience when the agent gets super far into something and then hits a catastrophic error, and that's just what we ha- watched happen, or"
  },
  {
    "startMs": 1231140,
    "text": "watched not happen. Um, and I'm gonna skip through this because I know I'm running out of time. Uh, so from the AI engineer's side, you write the prompt, you run it, and the out- if the output's bad, you throw it away, and then Temporal retries it for you. Um, bad model output is just an activity failure, so same similar retry policy, uh, as well. Um, so back to the four gaps. One record, uh, closed them all. Um, and just to skip through what those four gaps were, uh, again,  uh, and re-summarize that, uh, that yeah,"
  },
  {
    "startMs": 1261180,
    "text": "reliability and observability, uh, were just one record. And so every number of this talk that I, I, um, mentioned is in that temporal.io report State of Development, and I'll leave that up screen if anyone wants to take a photo and come check it out. Uh, and there's a bunch of us that'll be at the Temporal booth running this demo live if anyone wants to come see it or say hi. So, um, yeah, that's, that's my talk. Let's hope."
  },
  {
    "startMs": 1291440,
    "text": "Yeah."
  }
]