Build a Prompt Learning Loop

An agent receives a task but no guidance for planning it. Giving it a rigid plan can create a different problem: the next step may depend on information that has not arrived yet. Even an agent with the right tools can choose the wrong one if its instructions never explain when to use each. These are failures of the application around the model, and a stronger model alone may not repair them.

DeLucia groups these gaps around adaptation, planning, and context. Instructions should incorporate lessons from the environment in which the agent operates. Planning needs enough structure to guide execution and enough flexibility to respond to intermediate results. Context must supply the relevant data, available tools, and guidance needed to make the decision. Otherwise, the application is asking the model to act on information it never received.

The work also crosses organizational boundaries. Engineers and data scientists own code, automation, pipelines, performance, and cost. Domain experts and product managers understand what a successful interaction should accomplish. They can identify a response that technically completes a request but violates an important product expectation. A prompt-learning loop needs both kinds of expertise: infrastructure that captures behavior and feedback that explains why that behavior should change.

The workshop explains the learning mechanism through a student taking an exam. The student produces answers, a teacher grades them, and the result influences the next attempt. In the reinforcement-learning version of this analogy, the grade is a scalar reward that contributes to updates to model weights. That requires access to a different part of the system than an application developer usually changes when writing a prompt.

Meta-prompting moves the change into text. An LLM receives information about a prompt's performance and proposes a revised prompt. In the workshop's simplified comparison, the teacher now edits the student's instructions using the evaluation results. But a score alone leaves considerable work for that teacher: it says how an attempt performed without necessarily locating the mistake.

Arize Prompt Learning adds explanatory feedback to that revision process. Which answers were wrong? Why were they wrong? What should the student study? An expert annotation or an LLM evaluation can identify the missed instruction and explain the expected behavior. The revising LLM receives that information and produces better instructions, sometimes with examples. The artifact being learned is the prompt; the process does not require changing model weights.

For example, an expert might identify that a response ignored required context rather than merely marking it incorrect. A judge can likewise return its reasoning alongside a label. Those explanations give the optimizer something specific to address in the system prompt. Labels remain useful for counting outcomes, but retaining the input, output, and reason for failure gives revision a direction. This is the practical distinction DeLucia emphasizes: use the rich text already available in the evaluation process instead of compressing it all into a number.

The Cline coding-agent case study makes this change concrete. The displayed baseline prompt establishes that the agent should perform coding work, but it lacks a rules section. A role description tells the model what kind of assistant it is; it does not necessarily establish how it should behave when engineering decisions become difficult.

The revision adds expectations a good engineer would recognize:

• Identify root causes when handling errors and exceptions.
• Keep changes consistent with the system's design and check for technical debt.
• Accompany changes with appropriate tests.

These are reusable instructions about how to work, rather than solutions to particular issues.

DeLucia reports that adding the rules improved Cline's results on SWE-bench Lite, a benchmark of repository issue-resolution tasks. The intervention changed the system prompt without fine-tuning the model, replacing tools, or changing the agent architecture. The example makes prompt revision a concrete experimental variable: determine whether better instructions improve behavior before assuming that the surrounding system must be rebuilt.

Arize's published results give the scale of that change. The experiment used Cline in ACT Mode with GPT-4.1, dividing SWE-bench Lite into 150 training and 150 test examples; a patch counted as successful when it passed unit tests. Training accuracy rose from 18.67% to 34.00%, a gain of 15.33 percentage points. The highest reported test accuracy occurred at optimization loop 2, rising from a 17.33% baseline to 31.33%, a gain of 14.00 percentage points. Later test accuracies were 28.00% and 30.00%, so the last iteration was not the best.

Failure-driven revision raises an obvious concern: could the optimizer simply learn to accommodate the examples it has already seen? The proposed response is to request high-level coding standards rather than fixes tied to individual failures, then evaluate on separate examples. Those rules need to work beyond the failures used to write them. The coding experiment's data split is distinct from the webpage notebook used later in the workshop.

DeLucia also uses the analogy of hiring an engineer. You want that person to learn the conventions and quirks of your codebase, not remain so general that they ignore its requirements. Application-specific instructions can provide similar expertise. Specialization does not eliminate the need to check generalization, but it explains why local knowledge is valuable. As new problems appear, collect them, explain them, and revise the instructions again. The rules become something maintained throughout the application's life.

GEPA also uses reflection on evaluation feedback to revise prompts. Its search evaluates candidate prompts, asks a reflection model to inspect their results, and proposes mutations. Pareto-based selection retains promising candidates with different strengths, rather than immediately discarding everything except one winner. Merging provides another way to combine candidates as the search continues.

DeLucia reports that Prompt Learning performed better than GEPA with fewer optimization rounds in the comparison she presents. She then draws attention to a dependency shared by these experiments: the quality of the evaluator. An optimizer can only use the signal it receives. If that signal rewards an undesirable behavior, revision can reinforce the wrong lesson. The workshop situates GEPA alongside the MIPRO family in Arize's optimizer comparison.

Evaluator engineering therefore belongs inside the improvement process. Its prompt needs clear instructions, and its judgments need inspection. Improving the agent while leaving a flawed evaluator untouched creates a misleading target. The next question is not simply how to revise the agent faster, but how to keep the feedback useful as both prompts evolve.

An audience question challenges the coding example: how do you evaluate work whose success is less easily quantified? DeLucia starts with the stakeholders who understand the application. Bring domain experts, product managers, security, and leadership together to define what success means. An existing question-answering evaluator may help initially, but the application's own requirements should determine the eventual checks.

For Arize's assistant, those requirements include several distinct decisions:

• Did it find the right data?
• Did it appropriately choose semantic search or structured filtering?
• Did it retrieve information in the right order?
• Was its plan suitable for the task?
• What jailbreak attempts does security need to understand?

Define the desired behavior first, then translate each concern into an evaluation. A single judgment about the final response can conceal which part of the process needs attention.

Ali adds a subjective example from Booking.com: deciding whether a property listing or photograph is good. Different viewers may disagree about its appeal. A coarse good/bad judgment can start the process, followed by more specific factors such as dim lighting or the layout of the room. Those factors explain the judgment in terms that can guide a change.

A follow-up asks whether the additional factors provide a better direction for improvement than a binary label. They make the feedback more informative without requiring a literal continuous gradient. As reviewers discover useful distinctions, they can refine the evaluator and place some of those distinctions directly into the generating prompt.

Another question asks how an evolving rules section avoids redundancy and contradictory instructions. The initial prompt may be handwritten, but subsequent revisions should use accumulated examples. An LLM can consolidate instructions as it incorporates feedback. The intended workflow continually collects failures and produces new prompt versions that can later be adopted by the application.

This does not mean copying every failed example into an ever-growing rules section. The examples are inputs to a process that generates a new set of instructions. The revised prompt is the output of that process, separate from the dataset that motivated it.

DeLucia then separates two co-evolving loops. The agent loop collects unsuccessful responses and explains how the agent should behave differently. The evaluator loop collects judgments that deserve scrutiny and explains how the evaluator should change. Both can revise prompts, but the failure being studied—and the prompt being repaired—is different.

Human review, disagreement among judges, and confidence-related signals can help locate evaluator decisions worth inspecting. These are review cues, not proof that a verdict is wrong. The useful annotation identifies the actual mistake: what the evaluator misunderstood, which criterion it misapplied, and how its instructions should distinguish the case next time.

Consider two teaching cases based on the webpage evaluator's internal-link rule: internal paths must begin with /.

These illustrative cases show why an incorrect verdict should not automatically become training feedback for the agent. The agent-improvement loop depends on the evaluator's ability to distinguish the two situations.

The audience finally clarifies the scope: does this require a simulator of the whole product? Here, the work concerns the agent prompt and evaluator prompt. Building an end-to-end simulator is a separate undertaking, not a prerequisite demonstrated in this workshop.

The workshop moves from slides to the Prompt Learning repository, which participants clone to work locally. Setup provides time for another practical question: should a new agent begin with evaluations, or with a prototype whose failures reveal what needs evaluating?

DeLucia's advice is to avoid making a comprehensive evaluation suite a blocker. Build something small enough to inspect. An existing evaluator can help sort through outputs and reduce the burden of manual review, but it is a starting point. As the feature develops and its failure modes become clearer, refine the evaluator alongside it. This preserves the earlier requirement for reliable feedback without demanding a complete specification before the first experiment.

A further question asks how prompt learning applies to subagents and multi-agent systems. The demonstrated approach treats prompts as independently optimizable tasks. Each specialist can learn more useful instructions, including instructions relevant to handoffs. Ali argues that such local improvements can benefit a larger system, while the workshop remains scoped to isolated prompt experiments rather than joint optimization or simulation of every agent together.

The JSON webpage-generation notebook gives the loop a small, concrete task: generate webpage JSON from a request, evaluate the result, and improve the prompt. The demonstration uses OpenAI, while allowing generation and evaluation components to be adapted to other providers or applications.

The live setup adjusts a dependency version and adds asynchronous execution before configuring the experiment. The central input is a dataset with useful feedback: human annotations, evaluator results, or both. The optimizer uses that information to propose instructions; evaluations then help determine whether to continue and which candidate might be useful in the application.

The configuration controls several different dimensions:

• Sample size: the demonstration starts with 100 rows. A smaller sample shortens exploratory runs; a broader sample can cover more cases.
• Training split: separates examples used to construct revision feedback from examples used to assess candidates.
• Rule configuration: identifies the rules associated with the evaluation experiment.
• Optimization budget: limits the number of rounds of generation, evaluation, and revision.

These settings determine the scope of the experiment. Changing the amount of data is a different intervention from allowing more optimization rounds.

Participants supply their own OpenAI credentials through getpass. The notebook prompts for the key rather than requiring it to appear as a literal value in the displayed cell. With configuration and credentials in place, the next task is to inspect what each dataset row actually represents.

When the data appears, the stated split receives a live correction: the demonstration uses 50/50, rather than the initially mentioned 80/20. The first displayed rows contain inputs and outputs. They establish the task examples before feedback is attached.

Evaluation can add correctness labels, explanations, and rule violations. Several evaluators may contribute columns, or human annotations may accompany model judgments. The structure is flexible, but the input and output being assessed must remain identifiable. Participants also need not see identical first rows when their samples differ.

An attendee then raises a problem with single-turn examples: a conversation can fail in the middle. Replaying only its opening request may never reproduce the decision that went wrong. DeLucia suggests representing relevant exchanges as separate examples so they can receive their own evaluations and human feedback. Sampling only the beginning of conversations would repeatedly inspect the same kind of behavior while missing later failures.

Another question separates instructions from context. The optimizer should revise the instructions, not alter the recorded facts to make an answer look better. Additional dataset columns can carry the context available to the model. When an experiment regenerates an answer, it needs the information required to answer under those conditions.

That includes retrieved material, tool results, and external API responses flowing into the prompt being optimized. The unit of replay is one decision, but that decision should not be stripped of its inputs. Conversely, unrelated information from another stage does not automatically belong in the example. Preserve the context that influenced this step, including relevant conversation history, rather than choosing between no history and an indiscriminate dump of the entire trajectory.

The initial system prompt is deliberately spare: it assigns expertise in JSON webpage creation and supplies the task input. That is enough to begin generating outputs, while leaving considerable room for more precise instructions.

The notebook uses two LLM judges, though a code-based check can replace a judge where it measures the required property:

• The overall evaluator assesses the generated webpage against the input query and evaluation rules. It returns correct or incorrect, with a detailed explanation.
• The rule checker examines compliance rule by rule, identifying more specific violations.

Both contribute feedback to prompt revision. The overall label summarizes an outcome; the explanation and rule analysis help locate a repair.

The evaluator templates are read from files in the repository. The classification setup combines a provider, a template, and the permitted output labels. This makes the evaluator's instructions an inspectable part of the experiment rather than an unexplained scoring function.

Labels can optionally map to numerical scores, making results easier to aggregate. A different task might need multiple classes and a different mapping. The allowed choices constrain the shape of the judge's answer; they do not establish that its reasoning is correct. The template, labels, and returned feedback all matter when deciding whether its signal is useful enough to guide optimization.

The generation function is the application under test. In this notebook, the generation model is GPT-4.1, configured for JSON responses and zero temperature. It accepts a dataset and a system prompt, produces outputs for the rows, and returns them for evaluation. Every new prompt candidate passes through the generation function so the experiment can assess the behavior its instructions produce.

The JSON response format helps constrain output form. It does not establish that the webpage satisfies the request or follows every rule; that remains the evaluators' job. The function receives the dataset, current prompt, output model, and concurrency settings, then returns the generated results. Keeping these concerns separate makes it possible to replace the application logic without rewriting the entire learning loop.

An audience question asks whether zero temperature sacrifices useful creativity. DeLucia makes the choice task-dependent. Structured webpage generation prioritizes consistency, while another agent may benefit from different sampling behavior. Here, temperature=0 is an experimental setting chosen for the task, not a universal recommendation or a promise that every run will be identical.

Numerical summaries help track the run, but they do not replace explanatory feedback. For a nonempty set of evaluated outputs, a binary correctness label supports a straightforward judged pass rate:

This summarizes the evaluator's decisions, not independently established correctness. The walkthrough also mentions accuracy, precision, recall, and F1. Interpreting those as classification metrics requires an explicit reference comparison; merely turning the judge's labels into numbers does not supply one.

The optimization cell connects the components in three stages:

1. Generate outputs with the current prompt on the comparison dataset and evaluate their correctness.
2. If the result is unsatisfactory, generate and evaluate training outputs, then use their feedback to revise the prompt.
3. Repeat until the target score is reached or the iteration budget is exhausted.

The walkthrough initially allows five optimization rounds. A stopping threshold can end the experiment earlier.

The notebook calls its repeatedly consulted comparison split the test set. Because the scores influence when optimization stops, this split functions as validation or selection data. An untouched final test serves a different purpose: estimate performance after the prompt has been chosen and fixed. This follows the standard distinction between tuning and final testing. A separate untouched final split is not part of the demonstrated loop.

The loop tracks results across iterations and returns prompt versions, raw outputs, and scores for both splits. Its parameters include the stopping target, optimization budget, and rule configuration. The target need not be tied to one particular metric, but its meaning must remain consistent with what the evaluator actually measures.

At the start, an initial evaluation establishes the baseline and checks whether it already meets the threshold. An application with existing production outputs and feedback can use that evidence instead of generating everything from scratch. The notebook begins with a fresh evaluation to make the process visible, rather than requiring that exact starting procedure for every deployment.

Inside a revision round, generation first populates the training outputs. Evaluation then attaches correctness, explanations, and rule violations. The prompt-learning optimizer receives the prompt and this feedback-bearing training data, and produces revised instructions. A context-size limit can constrain how much material enters the optimization request. The explanations are therefore operational inputs to revision, not just fields retained for a report.

The returned prompt is still a candidate. The experiment generates outputs under that prompt, evaluates them on the training and comparison splits, records the score, and checks its stopping conditions. If neither the target nor the maximum loop count has been reached, the process continues. Producing a plausible revision is only one step; its resulting behavior must still be assessed.

The saving helpers preserve the experiment in CSV and JSON rather than leaving it in notebook memory. Records include iteration numbers, rule counts, scores for both splits, prompt versions, and the outputs and evaluation results needed to inspect the candidates. Returning to the internal-link teaching example, a revision should add a new candidate without erasing the failed output that motivated it. The illustration below preserves that distinction: adding P1 does not fabricate a new response, a score, or a winner.

Execution takes longer than explaining the control flow, and intermediate loop outputs become useful while participants wait. DeLucia estimates that the workshop's notebook run will take roughly twenty to thirty minutes. That is a planning estimate for the demonstrated run, not a general latency benchmark for prompt learning.

Evaluation errors send participants back to the dependency adjustment from setup. DeLucia reiterates the ==2.2 pin used for the live session. This is a concrete reminder that notebook compatibility is part of getting the experiment running, before any optimization result can be interpreted.

A question about the generic evaluator leads back to the repository's prompts directory. Its rubric is loaded from a file, so the evaluation criteria can be opened and examined. The score is not the only thing worth debugging; the instructions that produced it are part of the experiment too.

The dependency detour prompts a participant's joke about why people use Docker. The discussion also mentions nest_asyncio for notebook event-loop integration. That compatibility concern is separate from overlapping independent requests through asynchronous execution; an event-loop patch does not itself make model inference faster.

One participant reports that a single-loop notebook run took about six minutes after reducing the optimization budget from five loops to one. The purpose is to get through an inspectable workshop run, not to establish convergence or a controlled speed comparison. A small run lets participants see outputs, feedback, a revised prompt, and saved results before committing to more iterations.

The final notebook step extracts a prompt using the saved comparison results. Its stated objective is the best test accuracy, and it displays the original and optimized instructions. The last candidate is not automatically the strongest one: selection follows the recorded performance. For a Python list of evaluated candidate records with prompt and selection_score fields, that selection rule can be expressed as:

def select_prompt(records):
    if not records:
        raise ValueError("No evaluated prompt candidates")
    best = max(records, key=lambda row: row["selection_score"])
    return best["prompt"]

The selected instructions can then be brought into application code. Generating and selecting a prompt does not automatically adopt it in the application.

DeLucia then presents the managed alternative to maintaining the notebook loop. In the Arize workflow shown, prompts live in a prompt hub, while datasets carry human annotations or evaluations. Those datasets can come from collected traces or be ingested separately. The platform supplies a place to manage the artifacts that the notebook has been passing between functions.

A prompt-optimization task specifies its name, training dataset, output location, and feedback columns. Parameters control the experiment. Starting the task produces an optimized prompt in the hub, where the instructions can be retained as a new version alongside their results.

The presentation describes further integration of evaluations into these tasks as ongoing work. The managed offering's practical role is to provide task infrastructure and version storage for teams that do not want to maintain those loops themselves. The endpoint is an application-consumable prompt version with the experiment results available to explain its selection.