We Trust AI Agents That Break Every Day - Shy Ruparel, Temporal

A payment can leave funds reserved in a bank even after the application handling it has disappeared. Temporal developer advocate Shy Ruparel described that mismatch at AI Engineer New York on October 13, 2026: financial work crosses systems and people, and its consequences outlast any one running process. Adding an agent puts model decisions into an already long-running coordination problem.

Loan origination, know-your-customer and anti-money-laundering checks, fraud review and payouts can span days, multiple services and several people. A useful model response is only one contribution. The surrounding application still has to remember which actions succeeded, wait for decisions and finish the transaction without losing track of money.

Temporal’s second State of Development survey supplies the talk’s organizing evidence. It covered 554 engineers and engineering leaders actively using AI agents. Of those respondents, 48.4% had at least 11 years of industry experience, and roughly another third had six to ten years. The sample included engineering, AI, data, platform and IT roles; about two-thirds were in the United States, with the remaining third in the UK and EMEA. These are reports from an agent-using professional sample, rather than a measure of adoption across all engineers.

The first gap separates using agents from shipping systems that depend on them. Daily use rose from 47.3% in the survey’s 2025 edition to 80.8% in the edition presented here. Yet only 49.1% reported running agents in production or treating them as core to how they ship. The other group was still experimenting or mainly using assistants.

Those percentages describe different levels of commitment. An engineer can use an assistant every day without making an agent responsible for a customer-facing process. Production adds consequences: a failed run may interrupt work, leave external state behind or require someone to recover it. The survey establishes the adoption gap; the rest of the talk asks what engineering work sits inside it.

The second gap is more surprising. Fully 85.5% of respondents trusted agent output at least somewhat, with roughly a quarter trusting it almost completely. Ruparel places himself among the skeptics, which makes the result particularly striking to him. But confidence in an answer does not tell us whether the system delivering it can finish its work.

How often do those systems encounter issues? The issue-frequency chart is the useful comparison: 41.1% of respondents reported issues daily or more often, including 16.4% reporting them hourly. Ruparel describes the daily-or-worse results as pink bars on the survey chart. Their prominence explains the tension in the title: people can trust agent output while still encountering failures every day.

Experience did not remove the problem. Teams describing themselves as successful were reported to be 1.7 times as likely to encounter issues in the hourly-or-daily category, while the report found no correlation between agent count and issue frequency. Ruparel offers a possible explanation: successful teams may look harder and therefore find more failures. That is a hypothesis, not a causal finding from the survey.

The resulting design goal is to contain the cost of failure rather than assume competence will eliminate it. A separate 40% of teams identified security as the blocker to autonomous agents. That remains outside this talk’s recovery demonstration: preserving execution progress does not, by itself, answer whether an agent should have permission to act.

The third gap appears between a prototype and enterprise deployment. Across the survey, 51.3% reported getting an AI prototype production-ready in hours or faster. At organizations with more than 5,000 employees, about one in five reported taking months; elsewhere, about one in twenty did. The same broad tool category is operating on very different delivery clocks.

Banks, insurers and payment companies need approvals, audit trails and confidence that a payment will not disappear between services. Ruparel’s interpretation is that meeting those requirements accounts for some of the enterprise delay. The survey does not break down the causes, so it cannot establish how much time goes to reliability work versus other organizational constraints. It does explain why a fast prototype is an incomplete measure of readiness.

The fourth gap gets closest to the implementation problem: seeing a failure and recovering from it are different capabilities. The leading productivity blockers concerned execution state and debugging, rather than model capability:

• Tracking state across services: 35.7% named it as a top blocker.
• Isolating and debugging issues: 30.9% identified this problem.
• Redoing work after crashes or restarts: 22.2% reported losing progress.

A separate survey from the previous year found that 27% called their state management fragile. Ruparel explicitly cautions against treating the two surveys as a trend line: the surveys asked different questions. Still, the current responses make the operational concern concrete. Teams have to know what happened across services, not just what the model said.

YouTube was the most common reported destination for help when an agent broke—a result Ruparel finds bewildering. His interpretation is that people seek an explanation elsewhere when their own system cannot supply one. Traces, logs and LLM observability tools can reveal a failure, but visibility alone does not restore a lost approval or reconcile reserved funds.

Only 13% of respondents were very confident in their ability to observe and debug AI workflows at scale. The gap becomes a person reading a trace at two in the morning and repairing state by hand. The payout demonstration asks a more useful question: can the execution record itself carry enough information to recover, instead of merely helping someone investigate?

The example is a $2,500 payout for a synthetic customer, implemented as a real Temporal workflow with a model running locally on the laptop. Its nine stages compress a larger financial process: validate the request, reserve funds, obtain an FX quote, investigate suspicious transaction details and obtain a human decision. The agent can assess whether a transaction warrants escalation and prepare a brief; the human makes the approval decision in the demonstrated path.

The crash is deliberately placed after useful work has happened. Funds have been reserved in another system, and the agent has made tool calls and produced material for review. Stopping the worker simulates failures such as a timeout, deployment error or exhausted memory. The important distinction is where state lives: killing application memory does not undo the bank’s reservation.

In the memory-only design Ruparel contrasts with Temporal, the application could lose the brief and its knowledge of completed steps while the funds remain reserved. Regenerating the brief repeats model work; repairing the transaction requires persistent state tables, retry logic, recovery code and reconciliation jobs. Those mechanisms are necessary work somewhere in the system, but writing them separately for each business process distracts from the payout logic itself.

Temporal’s answer is durable execution: the code should be able to finish even if the process running it does not. Progress is recorded in an event history held by the Temporal Service, outside the worker. A replacement worker can use that history to reconstruct workflow state and continue from recorded progress.

Three kinds of work have different roles:

• Workflow code decides what happens next. It is deterministic, allowing replay to rebuild the same state from the recorded history.
• Activities perform nondeterministic or side-effecting work. Model calls, tool calls and bank API calls run as activities. Choosing the investigation, looking up customer information and drafting the brief each have separately recorded results.
• Signals bring external decisions into the history. Human approval or rejection is recorded as a signal, including when no worker is running. Processing can resume when a worker becomes available.

What survives the disappearance of a worker? The relationship below separates completed work, persistent history and later execution. Activity results and human decisions both enter the service’s history; a replacement worker reads that history rather than asking the model to recreate completed work. The history is the bridge between two processes that do not share memory.

There is an important limit to the token-saving claim. Completed activities with recorded results are retained, but an activity still in flight when the worker dies may be retried. The demonstration therefore supports reuse of completed model work, not a blanket promise that every external call executes exactly once. That distinction matters when a call costs money or changes another system.

Ruparel connects this to the survey’s finding that eight in ten respondents said cost limited their agent use. Recovering a workflow without regenerating its completed investigation avoids paying again for that work. It also preserves the earlier result, instead of substituting a fresh model response during recovery.

Which inputs make this transaction worth investigating? Ruparel sets the payout to $2,500, marks the recipient as new, identifies the amount as unusual for the customer and adds an unresolved account alert to the customer record. These are deliberately suspicious inputs, chosen to push the agent toward an investigation rather than leave escalation to chance.

Starting the job creates the workflow. The agent examines the customer profile and payout information, then drafts a brief for human review. Approve and Reject controls are available, but before making the decision, Ruparel stops the worker. With no workers running, he approves the transaction. The approval is still logged as a signal.

The intended next step is to start a worker, let the retry policy handle the timeout and resume from the last successful event. The live restart takes longer than expected; Ruparel refreshes and switches to a backup recording. The subsequent recovery walkthrough is therefore a recorded example, rather than an uninterrupted live completion.

The backup example repeats the approval during downtime and resumes the worker. It also introduces a second interruption while the payout waits for the bank to approve or mark the transaction complete. The bank-related signal is recorded during that downtime too, and a new worker picks up the waiting workflow. The two interruptions test separate waits in the same business process: one for a person, another for an external financial system.

What distinguishes recovery from starting over? Ruparel reads the resulting event history from top to bottom: each AI activity shows one attempt and one successful execution, while the approval and bank-related events arrived during periods with no workers. Replacement workers continued the payout without repeating the completed investigation or spending its tokens again. The record connects the cost claim to specific activity attempts, rather than merely showing that the final payout advanced.

The closing code discussion explains what the example moved out of application-specific recovery logic. Workflow state appears as ordinary fields and variables; replay rebuilds it after a crash. Ruparel did not build a separate database or cache for that state—the persistence sits in Temporal’s execution machinery. Completed model and tool results are retained with the workflow.

Retries are declared as a policy rather than implemented as a hand-written loop around each call. In this example, the configured policy applies to the transaction’s calls. Long waits also remain part of the workflow: the human-approval timer could be much longer without changing the basic waiting code. Ruparel uses a year-long wait as an illustration of the execution model, not as a duration exercised by the demo.

The most consequential separation is between advice and transaction control. How does the application keep an unavailable model from taking the payout down with it? The agent has a bounded opportunity to investigate and produce a brief; the workflow owns money movement and state. In this design, model failure can leave the payout continuing without the brief. That is a deliberate fallback choice with a narrower outcome than uninterrupted investigation: execution can proceed, but the AI’s review material is absent.

The same separation clarifies debugging. The history inspected in the UI is the history the worker replays. An engineer does not have to reconcile one observational account with a separately maintained recovery record before resuming execution. This is the concrete sense in which one record closes the seeing-versus-recovering gap: it both explains completed progress and supplies the information used to rebuild it.

Ruparel groups the demonstration’s benefits into seven concerns: state, debugging, cost, retries, approval, long-running work and crash recovery. He also names OpenAI’s Codex and Replit’s agents as production uses of Temporal. Those examples position the mechanism beyond this payout, but the demonstrated result remains specific: recorded work and external decisions survived worker replacement. The survey’s adoption, enterprise-readiness and security challenges are broader than that one recovery result.

One final extension brings output quality into the activity model. If application logic rejects a model response and treats it as an activity failure, Temporal can apply the retry policy to that failure too. The talk does not provide the acceptance check, so retrying cannot substitute for deciding what counts as a good answer. It does give that decision an execution mechanism: retain accepted results, retry rejected attempts under policy and let the workflow—not the model’s current process—carry the transaction forward.