Your Agent Didn’t Fail. Your Harness Did.

Talk transcript

1. 0:00

   [upbeat music] Thank you for choosing to spend this session with me.

2. 0:16

   My goal today is simple. I want to convince you all that most of the production failures are not-- most of, most of the agent failures are not model failures.

3. 0:27

   Those are harness failures. So let's start with one production incident.

4. 0:32

   The user saw the reply. The system forgot it happened. This is a failure shape I want to start with. Not a hallucination, not a crash, not a bad answer.

5. 0:43

   The user-visible edge looked healthy while the durable record ha-had a hole.

6. 0:50

   In this example, the user asked the customer-- the agent to remember a refund for a customer.

7. 0:58

   The assistant said it recorded the fact for the next turn. The interface looked normal, no red screen, no obvious failures. But the next turn cannot reconstruct the fact. The user experienced success.

8. 1:11

   The system inherited incomplete reality. Why this matters? A crash is annoying, but at, at least it gives you a boundary. You know so- you know something stopped. You usually see an error.

9. 1:26

   You can often start from last known good point. Silent success gives you a lie. Delivery can succeed while the persistent fails. The user has no reason to doubt the reply.

10. 1:40

    The operator has no obvious alarm. The next turn can still sound confident

11. 1:47

    because the model is coherent. But it is coherent over a bo-broken history.

12. 1:53

    That is why agent reliability matters, and agent reliability cannot stop at model quality.

13. 2:01

    Hi, I'm Vinoth. I work on core data and AI infrastructure at OpenAI.

14. 2:08

    Before that, I worked on distributed systems at Apple and Uber. Outside of the work, I write the Agent Stack, where I try to explain how the production AI and data systems work under the hood.

15. 2:20

    I'm not a mentor of OpenClaw, and this is not a OpenClaw product pitch. This is a pure system design talk. I'm using OpenClaw as a public case study because

16. 2:31

    its issues, code, docs, makes the harness around the a-agent unusually visible.

17. 2:39

    Here's the production contract for the talk. A model proposes, the harness commits, and the receipt proves it. The model ca-- may suggest a message, a tool, a edit, or an command, but model is not the production boundary.

18. 2:54

    The harness owns the state transition, the authority check,

19. 2:58

    the ordered commit, and the receipt is the evidence that survives the turn. OpenClaw is the case study, and the contract is the takeaway.

20. 3:07

    If you remember only three things from this talk, make it these: own the state, order the mutation, and prove the action.

21. 3:16

    A fact needs only one owner and one replay path. Shared mutable state needs one ordered commit path. And transcript is not the proof. A transcript tells you what the agent said.

22. 3:27

    A receipt tells you what the system allowed, attempted, executed, and what the user-visible edge confirmed.

23. 3:37

    To create a simple mental model, I created this car analogy of the harness. The model is the engine.

24. 3:44

    It matters a lot. But nobody, uh, buys a production car by just looking at the horsepower alone. You also care about steering, brakes, road rules, dashboard, and a black box.

25. 3:58

    The model gives you capability, but the harness gives you control.

26. 4:03

    A powerful engine with no brakes is not autonomy, it is a liability with good acceleration.

27. 4:11

    Here is the harness blueprint I wanted to, uh, discuss today. Every agent we know of, like personal agents such as OpenClaw or Hermes, coding a-agents such as Codex, Cursor, OpenCode, or Claude Code, uses the same underlying architecture.

28. 4:28

    Events enter, enter from many surfaces: a chat, webhook, timer, or heartbeat, or another external system. The control plane maps the events to a session key, and the session key de-determines the state boundary.

29. 4:43

    The session lane gives you one active writer for the mutable state.

30. 4:47

    The runtime calls the models and tools. Tools act through approvals and policies. An audit trail becomes the run receipt. This is the blueprint: event, session key, throttle, tools, audit.

31. 5:00

    The blueprint is the talk, and the incident was the proof that each boundary matters.

32. 5:08

    Context is assembled. Uh, uh, in agent runtime, th-there's not-- does not usually remember in human sense. It is stateless.

33. 5:18

    The harness rebuilds the working state for the each turn. The working set may include the transcript, session state, memory, policy, tool definitions.

34. 5:29

    The model only sees what the harness supplies. If one input is missing or stale, the answer may still sound coherent. Coherence does not prove the working set was complete.

35. 5:43

    These failures are not new. We already know about timeouts, retries, idempotency, logs, ordering, and state ownership. What changed is the a-agent setting.

36. 5:55

    Now these failures sit around a probabilistic planner with dynamic plans. It's rebuilding the context for every turn. There are more event sources, and it can act through more action surfaces.

37. 6:07

    So these failures are familiar.

38. 6:09

    Agents makes them easier to trigger and harder to explain. That's why Agent harness matters.

39. 6:18

    Let's talk about the first failure mode. This is the same failure mode I started this talk with. The user sees a success, the source of truth cannot replay it.

40. 6:35

    Delivered is not remembered. In this state hole OpenClaw issue, a Telegram replay could succeed while the router turn was not returned to the active context or transcript. The user saw the response, the log looked healthy,

41. 6:52

    but the next turn had no durable record of that exchange. A successful send proves transcript. It does not prove the future context. That distinction matters because the model can answer fluently over an incomplete record.

42. 7:06

    The missing boundary was not intelligence, it was the state ownership.

43. 7:13

    By owner, I do not mean a person, I mean the system of record whose persisted state becomes the truth.

44. 7:20

    A calendar event belongs to the calendar system, a support status belong to the ticketing system, while a code change belongs to a workspace or repository, and a conversation turn belongs to a session transcript, and a user preference belongs to a memory store.

45. 7:35

    Storage tells you where the bytes live. Ownership tells you who can reconstruct the reality.

46. 7:43

    A replay is not a reliable memory until, until a named owner can replay it. A system has to persist the turn, it has to name the owner or system of record, and it has to make the replay possible.

47. 7:55

    The real question is simple: For every fact the agent might use later, who owns it, and how would you replay it?

48. 8:04

    If no owner can replay the fact, the s- the system did not reliably remember it.

49. 8:11

    Once we know who owns the state, the next question is: Who's allowed to change it, and in what order?

50. 8:18

    Two correct writes can still produce one wrong outcome, and last writer wins is not a consistency model.

51. 8:28

    In this overlapping writer OpenClaw issue describes a load-modify-save race. Two callers loads the same old state. Each changes a different record. The sile-- The sec- the second save silently erases the first.

52. 8:43

    The user may see a dismissed commitment return or receive a duplicate follow-up. Neither writer is malformed. Both operations are locally correct. The missing boundary is serialization around the commit.

53. 8:58

    The invariant is not no concurrency. That would be too slow, and it would miss the point. You can fan out sub-agents. Parallel reads are fine. Independent retrieval is fine.

54. 9:08

    Many sessions can also run at once. The rule is narrower and simple. One ordered commit path for one mutable state boundary.

55. 9:17

    This mechanism may be a queue, a mutex, a transaction, or a lock. You can use locks or mutex across the sessions and queues or transactions within a session.

56. 9:27

    Be conservative with the commit time and not across the whole system.

57. 9:34

    Users do not see queues or locks, they see behavior. A last correction feels forgetful, a stuck lane feels dead, and completion before delivery feels confused. Ordering is a product feature because users experience ordering books as personalities.

58. 9:55

    Let's-- Now let's talk about time. In production, silence cannot be neutral. Let's review, uh, the next, uh, failure mode is life cycle failure mode. The RAN waits for an event that cannot arrive.

59. 10:08

    Silence is not a terminal state. In this dangling tool call issue, the session contains a tool call but no matching tool result. A process may have died, a connection may have dropped, a timeout might have happened, happened before the results were recorded.

60. 10:26

    The exact cause, uh, matters for debugging. The production failure is much simpler. The RAN is waiting for an event that will never arrive.

61. 10:36

    New messages queue behind that silence. To the user, the agent simply looks stuck.

62. 10:45

    RANs needs deadlines and cancellation. A deadline bounds the wait. Watchdog makes the stuck work visible. Tools needs timeouts and error results. Channels needs recovery commands that do not wait behind the stuck work they are trying to fix.

63. 11:00

    Every external boundary needs an ending: success, failure, timeout, cancel, or max attempts.

64. 11:08

    Most im-importantly, the receipt records the terminal outcome, so the next step does not have to guess. Bound the work before the work bounds you.

65. 11:19

    Now let's, uh, we can move on from state to authority, because a chat becomes risky when it becomes an action. Capability is not execution. The model can request an action.

66. 11:30

    Request ability is not authority. Approval needs a shape.

67. 11:35

    In this approval drift issue, expired approved callback was treated as retrievable. The state call-callback stayed durable, survived restarts, and blocked later channel work. The button click existed, the valid authority did not.

68. 11:50

    This is the mistake. Treating approval as a vague memory that the human was near the system or clicked yes.

69. 11:57

    Approval is a scoped execution state. It must stay bound to the action it authorized, and expiration must terminate rather than loop.

70. 12:07

    A useful approval object answers who approved, in what session and RAN. For wh-which tool and for which arguments, and for how long, and with what outcome.

71. 12:20

    It also point to the receipt. If those fields fall off during a retry, replay, or a channel callback, the harness can no longer prove the action was being executed as the action being approved.

72. 12:34

    The general lesson is simple. Capability is not execution. Least privileges narrows the tool surface. Scoped credentials ensures the right identity is used for the action. Approval and audit decides what happens before and after the execution.

73. 12:49

    The model can reason about the boundary, but it should not be the boundary. The model can request, but the s- still the system decides.

74. 13:00

    Finally, even if the tool says success, the user visible world may disagree. Internal component reports success, the user visible surface shows nothing. This is the inverse of the opening incident we saw.

75. 13:13

    In this missing edging-- edge proof issue, the message tool reported success for a web chat or TUI run, but the message did not render. Normal assistant reply still appeared.

76. 13:23

    The tool pro-proved that the internal path accepted the request. It does not prove the user saw the result. That difference changed the conversation. The agent may later say, "I already sent it," and the user may truthfully say, "I never saw it."

77. 13:37

    Internal success is not external proof. Proof is a chain, not a claim. Model proposed something, policy allowed or denied it, execution attempted it, user-visible edge confirmed or failed to confirm the outcome.

78. 13:54

    The receipt preserves that chain. A transcript records what the agent said. The tool results records what one component claimed.

79. 14:02

    A receipt records what the agent can verify at the boundary that matters.

80. 14:08

    Let me recap all the incidents. Here are the file failure shapes you, you to look for: a state hole, overlapping writers, dangling tool call, approval drift, and missing edge proof.

81. 14:19

    For each one, let's ask the same question: What did the user see? Which boundary it broke? And what would the receipt have caught?

82. 14:29

    Here is the audit I want you to run when you get back to your team. Pick one agent system, not all of them. One.

83. 14:36

    Trace one production-- one real production path and ask for the receipt.

84. 14:42

    The audit has five questions: What woke it up? What state did it inherit? Which authority did it use? What executed? And what evidence survived? These

85. 14:55

    questions expose causality. They turn a few fluent conversation into an inspectable production run.

86. 15:02

    First, what woke it up? A user message, webhook, timer, tool result, sub-agent, or a replay. Name the trigger and its identity. Without that, you cannot reason about deduplication, order, or authorization.

87. 15:17

    Second, which state did it inherit? Transcript, session state, memory snapshot, policy version, and tool surface. The model only reasons over the working set the harness assembled.

88. 15:30

    Third, which auto-authority did it use? Record the actor, session, tool, run, arguments, scope, and lifetime. A model request is not permission. Authority should bind to a one pending action.

89. 15:46

    Fourth, what executed? Record the tool or API call, arguments, attempt number, idempotency key, and external results. This is a side effect boundary, not the poor summary of what the agent intended.

90. 16:00

    Fifth, what evidence survived? Did the ticket get updated? Did the message got rendered? Did the file got changed? Did the calendar even exist? The receipt should end at the boundary the usual-- the user usually cares about.

91. 16:18

    Now let's apply the opening incident, uh, the same audit to the opening incident. What woke it up? A user message. What state it owned that was the broken boundary?

92. 16:28

    What executed the channel send? What evidence survived delivery? What did not save-- survive the durable turn?

93. 16:38

    Delivery survived while the st-state did not. That gap is the harness failure.

94. 16:44

    The agent, uh, do-- did not need a better model. The model did not need a better prompt. The system needed a better harness with complete receipt.

95. 16:55

    Let me recap the same three things I asked you to remember the, from the start of my talk. Own the state, order the mutation, and prove the action.

96. 17:05

    A better model helps inside the turn. Ownership, ordering, life cycle, authority, and proof keep the system sane across turns.

97. 17:16

    A model proposes, the harness commits, and the receipts-- receipt proves it. Once text can become an action, the useful question changes. Do not only ask whether the model can reason, uh, ask whether the system can own the state, order the mutation, bound the work, constraint authority, and preserve evidence.

98. 17:36

    A loop can answer a turn, and harness can serve a production.

99. 17:42

    If you want to go deeper, scan the QR codes. The first points to the agent-- OpenAI Agents SDK, where all these, um, harness are already built in so that you can use to build your own agents.

100. 17:55

     And second points to The Agent Stack, where I write about the production agents systems in more detail. I'll be at the OpenAI booth after this talk if you want to talk about your harness design.

101. 18:05

     Thank you. [audience applauding] [outro jingle]