Building Deterministic Infrastructure for Non-Deterministic AI Agents

What changes when an AI system stops answering questions and starts making decisions that affect production? Models must still reason well, but an agent also plans, calls tools, and coordinates workflows. Nishant Gupta, who introduces himself as a software engineering tech lead at Meta, locates the emerging challenge in reliability: probabilistic decisions need infrastructure with predictable controls.

Modern cloud infrastructure developed around short-lived requests, mostly deterministic services, known execution paths, and bounded failures. Autonomous agents strain those assumptions. They preserve state, run for extended periods, choose their next steps dynamically, and may follow different workflows for identical inputs. Gupta calls this the great mismatch. The slide contrasts stateless and stateful execution, deterministic and probabilistic behavior, request-response and multi-step workflows, and millisecond and long-running execution.

A demo asks whether an agent can solve a problem, use a tool, or complete a workflow. Production asks whether it can do so repeatedly, recover from failures, operate safely, and deliver acceptable outcomes within cost and latency constraints. That shifts engineering effort below the model into orchestration, monitoring, safety, evaluation, and recovery. Task completion is the beginning of the reliability requirement, not its completion.

Hallucinations are only one failure mode. Recursive reasoning loops, overflowing logs, retry amplification, context corruption, memory poisoning, and cost explosions can turn an incorrect model output into a system-wide problem. The distinction matters: the model makes a mistake, but infrastructure determines whether that mistake remains contained or becomes an outage.

Consider Gupta’s invalid tool-call cycle:

1. The agent sends an incorrect request to a tool.
2. The tool returns an error.
3. The agent produces a slightly different request that is still invalid.
4. The cycle repeats, consuming more compute while reasoning depth and GPU consumption rise.

Gupta describes exponential resource growth as a possible escalation of this cycle, without supplying a measured growth curve. Repetition alone does not establish that growth rate; the architectural risk is that recovery has become an uncontrolled source of additional work. A minor API error can then become a compute incident.

The model suggests; the platform decides. Gupta’s strongest architectural recommendation is to keep the model from directly controlling production systems. The boundary has four distinct stages:

1. The model generates a proposal.
2. Infrastructure validates it.
3. A policy engine approves or rejects it.
4. An execution gateway enforces the decision.

Validation, approval, and execution are separate responsibilities. A valid request is not necessarily permitted, and a permission decision does not itself perform the action.

A TypeScript implementation can make that boundary explicit. Here, the proposed action restarts a named service. Model output enters as unknown; only a validated proposal can reach policy evaluation, and only an approved proposal reaches the gateway.

type Proposal = {
  action: "restart_service";
  service: string;
};

type Decision =
  | { allowed: true }
  | { allowed: false; reason: string };

interface PolicyEngine {
  evaluate(proposal: Proposal): Promise<Decision>;
}

interface ExecutionGateway {
  enforceAndExecute(
    proposal: Proposal,
    approval: { allowed: true }
  ): Promise<void>;
}

function validate(input: unknown): Proposal {
  if (typeof input !== "object" || input === null) {
    throw new Error("Expected an action proposal");
  }
  const value = input as Record<string, unknown>;
  if (
    value.action !== "restart_service" ||
    typeof value.service !== "string" ||
    value.service.trim() === ""
  ) {
    throw new Error("Invalid restart proposal");
  }
  return { action: value.action, service: value.service };
}

async function handleProposal(
  input: unknown,
  policy: PolicyEngine,
  gateway: ExecutionGateway
): Promise<void> {
  const proposal = validate(input);
  const decision = await policy.evaluate(proposal);
  if (!decision.allowed) {
    throw new Error(decision.reason);
  }
  await gateway.enforceAndExecute(proposal, decision);
}

This example expresses the separation in application code. The gateway remains responsible for enforcing the approved operation, and production credentials belong behind that boundary rather than in the model’s control.

Containers gave rise to Kubernetes, and microservices created a need for service meshes. Gupta proposes a corresponding agentic control plane responsible for scheduling, memory coordination, policy enforcement, evaluation, monitoring, and workload routing. His operating-system analogy describes a common layer that manages autonomous work, rather than leaving each agent to manage its own execution. He expects organizations that build this layer well to gain a competitive advantage.

That layer needs more than logs of completed actions. Traditional logs describe what happened; debugging an agent also requires reconstructing how its workflow reached that point. Traces should connect planning decisions, tool calls, memory lookups, and state transitions. The final output may reveal that something went wrong while concealing the earlier decision or retrieved state that caused it.

Trace the observable decision sequence. This means recording instrumented decisions and events, not assuming access to hidden model reasoning. Together, the sequence of calls, retrieved memory, and state changes provides the multiple dimensions needed to debug an autonomous workflow.

Once multiple agents share state, familiar distributed-systems failures appear: stale reads, conflicting updates, context drift, and inconsistent views. Memory becomes harder to coordinate when it is also probabilistic and retrieval based. An agent may reason from a different view of the world than another agent, so an apparent reasoning failure may actually be a consistency failure. Improving the model alone does not resolve disagreement about the underlying state.

Safety likewise cannot sit in a single component. Gupta applies defense in depth through several complementary controls:

• Prompt-level controls: Guide the agent’s behavior.
• Tool permissions: Restrict which operations it can access.
• Policy validations: Check whether proposed actions are allowed.
• Human approvals: Introduce review where judgment is needed.
• Audit systems: Preserve records for accountability and investigation.

Each layer addresses a different class of failure; none replaces the others.

Human involvement is not necessarily a temporary concession on the way to full autonomy. Gupta expects successful systems to remain supervised, with people serving as exception handlers: reviewing ambiguous situations, handling novel scenarios, and providing calibration signals. The design objective is to allocate human attention where it contributes the most value, rather than simply minimizing human participation.

Agent workloads increasingly resemble cluster scheduling problems. Demand arrives in bursts, reasoning depth is unpredictable, and workflows may run for minutes instead of milliseconds. Resource requirements also vary dramatically, making GPU efficiency, workload placement, elastic capacity management, and scheduling central concerns. Inference therefore becomes a resource orchestration problem, not just a model-serving problem. The slide’s jagged demand curve and stepped capacity bands bring those concerns together with resource contention and GPU limits.

These requirements do not demand an entirely new infrastructure discipline. Gupta maps established reliability patterns onto autonomous workloads:

The invalid tool-call cycle illustrates why these mappings matter. A retry should be part of controlled recovery, not permission to generate work indefinitely. Tool isolation and agent limits contain failures; quotas bound resource exposure; traces make the resulting behavior explainable. The implementation changes, but the reliability purpose remains familiar.

Gupta describes an industry progression from prompts as the differentiator, to models, and now toward infrastructure as prompts and models become more widely available. His forecast is that the winning organizations will not necessarily have the best prompts; they will have the most reliable systems. The competitive advantage moves into the layers that make model capability dependable.

Treat agents as distributed systems. Their models remain stochastic, while infrastructure must make validation, permissions, execution, and recovery predictable. Observability is mandatory, and control planes become the foundation for coordinating autonomous work. Better models expand what agents can attempt; better systems determine whether those attempts can safely become production operations.