How to Build AI Agents that Actually Work

What has to change when a product is rebuilt around AI agents? Patrick Dougherty opens with that decision at Rasgo, where he was co-founder and CTO: two years before the talk, the team began taking apart its existing product and rebuilding it around agents. The first requirement is a precise definition of what the new system should do.

Dougherty’s working definition has three parts:

1. Accept an objective, supplied by a human or another AI.
2. Call at least one tool and receive its response.
3. Decide autonomously how and when to use tools to accomplish the objective.

That last requirement separates an agent from a predefined prompt chain. If the application always runs one tool followed by another, the sequence is making the decisions. An agent must use what it learns to choose its next action.

The first design shift was to enable reasoning instead of relying on what the underlying model already knew. Rasgo moved context gathering into discrete retrieval calls that the agent could make while working. The practical distinction is upfront context injection versus retrieval on demand; retrieval through tools is still a form of retrieval-augmented generation.

The product searched and queried enterprise data warehouses. Giving the agent a large collection of tables and all their columns seemed like a straightforward way to prepare it to write SQL. In practice, Dougherty reports that the volume of schema information overwhelmed its selection process: it picked the wrong table or produced a query that would not execute.

The replacement was a small set of tools with distinct purposes:

• Search tables: Find candidate datasets relevant to the question.
• Get table detail: Inspect a candidate’s structure before using it.
• Profile a column: Investigate the data in a field rather than relying only on its name.

These are building blocks for an iterative investigation, not a mandatory three-step chain. Each result helps the agent decide what to inspect next and which columns belong in the query.

That investigation also needs a valid stopping condition: the required data may not exist. A useful agent can search, recognize that it lacks the evidence needed to answer, and report the gap so someone can act on it. Dougherty contrasts this with his observations of GPT-4o, which tended to attempt a query even when the available data did not support the requested calculation.

The demonstration supplies GPT-4o with a simplified Salesforce schema: accounts, contacts, and opportunities, each with representative columns. The request is direct: “Write a query to see how many of my customers churned in the last month.” Before writing SQL, the model needs to establish whether those fields can actually identify a customer’s transition to a churned state.

GPT-4o proceeds to generate SQL. Its crucial assumption is that an account type other than customer indicates churn, and that this type would be updated when a customer left. The supplied schema does not establish that business rule. Counting accounts under that assumption can produce a plausible-looking answer without measuring churn at all. Automatically running the query would carry that unsupported interpretation into the analysis; the response does not push back on the missing evidence.

Dougherty then runs the same schema and question through o1. In this example, o1 concludes that the provided schema cannot establish whether an account has churned.

This is a same-prompt demonstration, not a general model benchmark. Its useful lesson is about the agent’s objective: answer the business question correctly, which sometimes requires declining to generate a query until the missing data or definition is supplied.

The Agent-Computer Interface, or ACI, is the contract between the agent and the code that performs its actions. It includes the syntax and structure of tool arguments, along with the content and format of the response. A tool can execute correctly while still returning information in a form the model handles poorly.

Rasgo found that apparently trivial interface changes could substantially affect agent behavior. One example involved the format of search results. With Markdown responses, GPT-4o sometimes reported that a column did not exist even though the tool result contained it. Dougherty reports customer tables with 500 or 1,000 columns and tool responses of roughly 30,000 tokens. The failure occurred in long results where finding the relevant field was itself a demanding part of the task.

After testing alternative formats, the team changed those responses from Markdown to JSON. Dougherty reports that this resolved the missing-column failure for GPT-4o in their tests. A compact illustrative table-detail response makes the interface decision concrete:

{
  "table": "accounts",
  "columns": [
    { "name": "account_id", "data_type": "VARCHAR" },
    { "name": "account_type", "data_type": "VARCHAR" }
  ]
}

Here, the example field names illustrate a structured response: each column is an explicit object with named attributes. Formatting preserves the metadata; it does not supply a missing business definition such as what constitutes churn.

The team later found XML preferable for Claude in its setup. The actionable conclusion is to test the tool’s input and output contract with the model and workload actually in use. Dougherty suggests that training data may explain these preferences, but the observations do not establish a universal rule that one model always needs JSON and another always needs XML.

Think of the model as the agent’s brain: its mistakes become visible as bad decisions in the product. That does not require using the same expensive model for every operation. Cheaper models can handle subordinate prompts or work inside tools, while a more capable controller decides what to do next from the results accumulated so far. At the time of the talk, Dougherty favored Claude 3.5 Sonnet for that controller role because of its balance of speed, cost, and decision quality.

Failures can also reveal how to improve the interface. If an agent repeatedly ignores a tool’s JSON Schema and supplies an argument in a different format, inspect the pattern rather than treating every occurrence as an isolated mistake. The repeated shape may suggest a tool definition the model can use more reliably. Where the intended operation permits it, changing the interface to match that expectation can reduce friction. Dougherty interprets this as working with the model’s learned expectations; the malformed calls are the observable evidence, while the training-data explanation remains a hypothesis.

Fine-tuning did not pay off for Rasgo’s agent work. Dougherty describes cases where tuned models became more likely to repeat a particular sequence of tasks instead of stopping to ask whether the next action made sense. He interprets this as overfitting that reduced reasoning flexibility.

For this setting, his recommendation is to prioritize ACI iteration over fine-tuning the agent’s controller. The distinction is practical: making relevant evidence easier to retrieve and interpret supports a fresh decision at each step, whereas reinforcing a habitual sequence can undermine that flexibility. This is a conclusion from the team’s experience, not a general limit on what fine-tuning or other post-training methods can achieve.

Rasgo did not build on an agent framework. Initially, Dougherty says, options such as LangGraph and CrewAI were not publicly available when the team began. Early research drew on Auto-GPT. As frameworks became more popular, the team reconsidered whether to move its implementation onto one.

The deciding constraint was production authorization. A person querying Snowflake may have granular permissions over the underlying data. The agent acting for that person needs to operate under those same permissions, rather than gaining broader access simply because a service is making the query.

Rasgo needed an OAuth integration that carried the requesting user’s permissions through to the agent’s work. Its own code needed to manage authentication, service keys, and tokens. Dougherty reports that fitting this requirement into an approach like LangGraph was difficult to build and scale. That is an account of Rasgo’s integration constraints, not a claim that LangGraph cannot support authentication; LangGraph Platform has documented custom authentication and access control. The architectural question is whether the chosen framework lets the application retain the required control over identity and delegated access.

For prototypes, Dougherty sees value in abstractions that help validate an idea quickly. For production, he advises deciding the end requirements before becoming dependent on them. His reasoning is that basic single-agent or multi-agent orchestration does not necessarily require much code, while adapting that orchestration to a product’s security and operational requirements can be the harder work.

That production work also changes where differentiation comes from. Dougherty is skeptical that keeping a system prompt secret creates much of a moat. He places more value in the surrounding product: how users interact with the agent, which systems it connects to, and which security protocols govern its work.

Those connections, permissions, and interaction patterns were the most time-consuming parts of turning the agent into a production product. In his view, they are also more defensible than the prompt itself—although he qualifies even that claim because the technology is changing so quickly.

About a year into the transition, as customers became comfortable with single agents, Rasgo introduced a multi-agent concept. The central design decision was a hierarchy: a manager agent owns the final outcome and delegates subtasks to workers with more specific instructions and tools. This distributes specialist context instead of placing every instruction and tool in the manager’s context, where the volume can overwhelm its decision-making.

Team size mattered too. Dougherty invokes the two-pizza rule for human teams as an analogy for keeping agent collaboration bounded. In Rasgo’s experience, roughly five to eight collaborating agents typically worked well. In systems he saw or prototyped with 25 or 50 agents, loops and paths the system failed to return from made completion less likely. These are practical observations, not a measured scaling law or a universal agent-count limit.

The manager’s instructions should emphasize accomplishing the overall objective rather than forcing every worker through a prescribed sequence. Dougherty describes this as incentivizing or rewarding the manager for the outcome, without specifying a reinforcement-learning implementation. The responsibility is concrete: delegate useful work, judge whether the returned output is valuable, and determine whether it can contribute to the broader result. Delegation does not transfer ownership of the final answer. The manager must keep evaluating what the team has actually accomplished.