How to Improve Your Agents: Academic Lit Review

An agent cannot do useful work merely by producing convincing text. It must perceive its environment, reason about possible actions, execute a decision, and reconsider what happens afterward. The relevant inputs might include text, images, audio, video, or touch; the relevant outputs might include a conversation, physical movement, or another action that changes the surrounding world. 1:18

These capabilities do not automatically imply complete autonomy. A useful progression begins with a chatbot that retrieves information, advances to agent assistance where a person approves suggested customer-service responses, and then reaches agents that independently handle bounded workflows such as scheduling meetings or drafting job descriptions. Greater autonomy means coordinating multiple tasks and sharing resources between them; the furthest hypothetical stage resembles Jarvis from Iron Man, an assistant trusted with sensitive credentials and authority to act independently. 2:54

The appropriate deployment boundary depends on consequences. Self-driving systems operate where errors can threaten lives. Filing reimbursements is comparatively low-risk and can begin under human supervision, whereas customer-facing work carries greater exposure. Moving from back-office automation toward front-office responsibilities requires a corresponding increase in trust, not merely a more persuasive model. 4:37

TriPosT treats an attempted solution as part of the training input and teaches the model to produce both feedback and an improved answer. Crucially, the stronger model does not discard the smaller model’s reasoning and substitute an unrelated demonstration: it edits the smaller model’s own critique so the correction remains connected to the original mistake. For mathematical problems, a Python script can also provide more reliable corrective information. 10:56

The interaction follows a specific sequence:

1. The smaller model attempts the problem and generates an initial answer. 11:39
2. The same model critiques that answer. A stronger model edits the critique while preserving its relevance to the smaller model’s reasoning.
3. The smaller model revises its answer using the corrected feedback; this cycle can repeat until a benchmark problem reaches its known ground-truth answer.
4. Successful trial-and-error trajectories are filtered and balanced into training examples that teach the smaller model how to improve on mistakes it actually makes.

The final step uses weighted supervised learning on on-policy data: examples generated from the model’s current behavior rather than a fixed collection of unrelated ideal solutions. Because the model changes during training, its attempts and feedback must be regenerated accordingly. Otherwise, the training distribution drifts away from the mistakes the updated model actually produces. 12:47

The evaluation includes BIG-Bench Hard tasks covering Multistep Arithmetic, Word Sorting, Date Understanding, and Logical Deduction. After three feedback-and-revision iterations, the reported result reaches approximately 48% accuracy on Logical Deduction; the linked TriPosT results table identifies the corresponding task-specific total. This is an accuracy value for that reasoning task, not an improvement percentage or a result averaged across every benchmark. 12:59

The important comparison is whether the model learns to correct itself, rather than merely memorizing gold answers. Later correction rounds contribute additional correct answers across the examined tasks, and training on on-policy trajectories performs better than using straightforward supervised fine-tuning examples alone. However, the approach inherits a ceiling from its editors and verifiers: if the stronger model cannot supply a useful correction, the smaller model cannot reliably distill it. 13:47

Model pretraining depends on available compute, training data, and parameter count. Increasing these inputs becomes difficult when high-quality data is limited and access to large GPU fleets is concentrated. For smaller companies and academic groups, training ever-larger foundation models is often less practical than extracting better behavior from models that already exist. 15:55

Test-time compute scaling shifts that effort into inference. Instead of changing pretrained weights immediately, the system allocates additional steps to reasoning, reflection, or search before committing to an answer. The useful question is not simply whether a model can generate one response, but whether a larger decision-making budget can uncover a better sequence of actions. 17:20

Tree search extends ordinary stepwise reasoning by considering alternative branches rather than following a single chain. The resulting trajectories can eventually become training data, but those are separate interventions: searching at inference improves a particular decision, while learning from accumulated search traces changes the underlying model later. 18:04

Conversation is a sequential decision problem: given everything already said, what response should come next, and how might another person react? In a charity-persuasion example, one participant tries to encourage a donation to Save the Children. The conversation starts socially, then explores whether the other person has previously donated. A response that donation depends on finding the right circumstances changes which persuasive strategy should follow. 18:27

Chess provides the structural analogy. A player proposes a move, anticipates an opponent’s response, evaluates the resulting position, and repeats this process across possible branches. Search algorithms associated with systems such as AlphaGo use this general pattern of proposal, simulation, and evaluation to identify stronger actions through lookahead. 19:38

GDP-Zero adapts Monte Carlo Tree Search, or MCTS, to goal-oriented dialogue without first training a specialized dialogue policy. Prompted language models take several roles: one proposes candidate conversational actions, another simulates their likely consequences, and an evaluator estimates action quality. Another prompted model plays the prospective user, generating possible responses from the existing conversation history. Action estimates are updated as the search explores additional branches. 20:50

The human-response problem makes this different from a deterministic board game. A conversational strategy can produce many plausible replies, including negative reactions, and its estimated prospects can vary accordingly. GDP-Zero therefore uses open-loop MCTS: rather than assuming one fixed next state, it samples possible responses and searches through alternative conversational developments. This explicitly accommodates the stochastic character of human interaction. 22:23

The intended outcome is concrete: whether someone donates to the charity. Evaluation combines language-model judgments comparing conversations generated with and without planning, alongside a human study in which online crowdworkers interact with a persuader chatbot. Yu reports better donation outcomes for the planning-based system, together with stronger perceived persuasiveness, naturalness, and coherence, without giving numerical effect sizes in this recording. 23:32

The resulting strategies also show why planning matters. Asking immediately for a donation can be ineffective; the agent instead builds toward the larger request. It also varies emotional and logical appeals instead of repeating one tactic throughout the conversation. These findings concern dialogue planning specifically, not browser manipulation or mathematical self-correction. 25:06

Search generates more information than the final action selected. Ordinary imitation learning transfers only the best action found in a tree, effectively discarding the alternatives that were explored, the mistaken branches that were rejected, and the evaluations that motivated changing direction. The ExACT implementation uses Exploratory Learning to treat the search traversal as a trajectory that can be used to train the underlying model. 33:26

The key transformation is to linearize the traversal so that exploration, evaluation, and backtracking appear as an ordered learning signal. Instead of seeing only the successful endpoint, the model learns the process of trying an action, assessing the resulting state, recognizing when it fails to advance the goal, and returning to consider another option. Unlike R-MCTS operating solely at inference, this stage explicitly changes the base model through training. 34:01

The ExACT demonstrations show why the search path matters. Consider a task requiring the agent to find a recent, relevant coffee maker with a touchscreen and then comment on the item. An attempted action can lead somewhere unhelpful; the agent must evaluate that outcome, backtrack, and try a different route. The useful training example is not merely the eventual successful interaction but the sequence that reveals how the agent recognizes and recovers from a wrong turn. 34:50

R-MCTS and Exploratory Learning therefore cooperate without becoming interchangeable. The former spends additional inference-time computation to search and evaluate actions; the latter learns from the resulting search examples so subsequent decisions can incorporate exploratory behavior in the model itself. 35:24

Both search and interactive environments impose practical costs. Open research questions include whether reinforcement-learning methods can reduce reliance on expensive search trees and whether model-predictive control can reduce the expense of setting up environments and repeatedly interacting with them. Better agent performance matters only alongside the resources required to obtain it. 35:45

Beyond individual tasks, the orchestration layer must balance autonomous exploration with controllability. The open-source Arklex Agent-First Organization framework enters this discussion through capabilities including task decomposition and continuous learning. Developing such systems also involves systems engineering, human-computer interaction, and security, rather than machine-learning expertise alone. 36:09

Most of the preceding benchmarks assume one agent performing one human-assigned task. Allowing one person to delegate multiple tasks on the same computer introduces scheduling conflicts, shared database access, unwanted side effects, and decisions about when to request human approval or hand control back. Expanding further to multiple people and multiple agents creates additional coordination and adversarial concerns. More realistic evaluations must therefore account not only for task completion but also for efficiency, security, and the behavior of the larger system. 37:18