Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

An AI application often depends on rented inference, limited context windows, and a model whose internal behavior its builders cannot directly control. Tejas Kumar frames the central engineering problem as reliability: an agent should perform its assigned job even when the underlying model is a black box and its outputs are nondeterministic. The harness addresses what the surrounding system can control rather than assuming the model alone will behave predictably. 1:17

The physical analogy is an anchor: a climbing harness connects a person to something stable and limits how far they can drift. In AI engineering, an agent harness is everything around the model that grounds its behavior in a stable, controlled environment. Kumar distinguishes this from a machine-learning harness used primarily to run inputs through models and assess outputs; his subject is the runtime infrastructure surrounding an acting agent. 3:13

This distinction also clarifies that a harness is not simply an agent loop. The loop is one component, while the harness includes the surrounding structures that govern how the agent acts, what resources it can consume, and whether its claimed result is actually true. In some designs, an outer harness loop can even manage repeated runs of an inner agent loop. 4:11

Kumar identifies several recurring components: a tool registry, a model, context-management primitives, guardrails, an agent loop, and a verification step. Coding systems such as Claude Code, Cursor, and Codex illustrate the pattern through tools that can read files, write files, and execute bash commands. A harness coordinates these capabilities around a model rather than treating tool access as sufficient by itself. 4:11

Context management keeps the agent operating within practical limits, while guardrails constrain behavior such as excessive tool calls or repeated attempts. Kumar describes a maximum-step rule that terminates a run once it exceeds a specified threshold. These controls establish boundaries in application code instead of relying on the model to police its own resource consumption. 4:11

Verification closes the gap between an agent saying it succeeded and the external system demonstrating that it did. For a coding agent, Kumar gives examples such as running lint and tests after the work is complete. The broader principle is that success should be checked against evidence available in an environment the application controls, not accepted solely because the model declared the task finished. 5:19

The demonstration assigns a browser agent a specific task: visit Hacker News and upvote the first post. Kumar deliberately selects GPT-3.5 Turbo as an older model and commits to leaving both the task prompt and system prompt unchanged. The experiment isolates the effect of engineering the surrounding runtime: if the outcome improves, the improvement comes from the harness rather than stronger prompting. 6:13

The initial implementation uses Playwright directly to launch Chromium, create a browser context and page, and navigate through ordinary browser automation. A browser session is passed into a set of tools defined with a name, description, parameters, and executable implementation, following the tool structure Kumar attributes to OpenAI's SDK. The initial context consists only of a basic system prompt and the user's task. 7:26

An initial agent loop repeatedly requests a response, stops when the model signals completion, and appends events to a trace. On its first run, the agent reaches Hacker News, clicks an upvote control, encounters a login screen, and nevertheless reports success. The concrete defect is not merely that authentication is missing: the system has no independent mechanism to distinguish an attempted click from a completed upvote. 8:22

The first incremental improvement introduces two explicit guardrails: maximum iterations and maximum messages. Exceeding the iteration limit stops execution; exceeding the message threshold triggers context compression. The loop also records metadata such as context size, making the runtime's interventions observable alongside the existing event history. 9:24

The demonstration's compressor preserves the system prompt, the user prompt, and the two most recent messages while removing the intervening history. Kumar explicitly describes this strategy as basic and naive, noting that better approaches exist. Its value in the demonstration is conceptual clarity: context control is implemented as deterministic application behavior, but aggressively discarding intermediate messages can sacrifice potentially relevant state. 11:15

Kumar then moves the orchestration into a function called run harness, initially changing the organization rather than the behavior. A subsequent revision introduces a verification function and a maximum-attempt setting: an outer harness loop can run an inner attempt up to three times. This layering demonstrates why the harness is more than the agent loop, because policy and retry limits are enforced around the underlying execution. 12:09

The verifier inspects the accumulated tool history rather than trusting the model's summary. It rejects a run when an automatic-login tool reports failure or when the browser remains on a login URL without a successful recovery, even if the agent claims the upvote happened. At this stage the task still fails, but it now fails truthfully: the system can distinguish a real unresolved problem from an invented success. 13:11

The final improvement adds a login handler that checks the browser's current URL during each agent-loop cycle. When the browser is not on a login page, the handler returns without intervening. When the browser reaches a login page, the harness fills credentials into the form and submits it through deterministic browser automation. Kumar notes that the credentials could come from an environment variable rather than being embedded in a prompt. 15:16

This placement matters: authentication is performed by the harness, not by the agent. The surrounding runtime can access the required secrets, execute the login procedure programmatically, and append a message telling the agent that authentication has been handled. The model remains responsible for pursuing the task, while application code handles the predictable, sensitive transition that previously caused the agent to fail. 16:07

With the handler in place, the demonstration opens Hacker News, reaches the login flow, authenticates, completes the upvote, and reports success after six iterations. Kumar additionally checks the resulting page and observes that an unvote action is available, providing external evidence that the upvote occurred. The task prompt and system prompt remain unchanged throughout, so the demonstrated improvement comes from guardrails, deterministic intervention, and verification around the model. 17:03

Kumar argues that a strong harness can make a less expensive or smaller model useful by compensating for nondeterminism with stable application logic. His browser demonstration supports a bounded version of that argument: an older model completes one specific workflow after the runtime gains authentication handling and outcome verification. It does not establish that every task can be made reliable this way, nor does the example compare multiple harness designs or provide broader performance measurements. 6:13

He also describes Open RAG, an open source project at IBM that he says is deployed in enterprise environments to perform RAG operations over sources including Teams, calls, PDFs, and invoices. In this account, harness engineering supports asking questions over sensitive, siloed internal data with enterprise-level security. The transcript identifies the intended role of the harness but does not detail the project's security architecture or provide independently measured results. 18:02

Finally, Kumar presents dynamically generated harnesses as a speculative future direction: before attempting a task such as buying a flight ticket, an agent might create its own task-specific constraints and verification structure. He compares the idea to an expanded form of plan mode and imagines a system anticipating where it might hallucinate before acting. This is a proposed possibility, not a demonstrated capability or a confirmed prediction. 19:07