Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

Talk transcript

1. 0:01

   Hi, everyone. My name is Soheil Feizi. I'm founder and CSO at RELAI. I'm also an associate professor in the computer science department at University of Maryland. Today, I'm going to talk about continual learning for AI agents, how we can go from failures to durable improvements.

2. 0:19

   And if you're interested in any of the tools that I'll be talking in this presentation, you can visit at our web-- at our website, relai.ai. Let's get started. Humans learn mainly from experience by interacting with the world and getting feedback.

3. 0:37

   The goal of continual learning is to imitate the same for agents, so they can also learn from experience by acting, getting feedback, and improving without forgetting.

4. 0:50

   All right. So here's basically a bigger picture of how continual learning for agents look like. So here is an agent that interact with the world, with diverse users, with complex tools, uh, with various data policies.

5. 1:07

   And as I mentioned, the goal is to continuously improve the agent from its experience without forgetting. And this learning can happen in different layers of the agent. It can happen in the model layer, where potentially we can change weights of LLMs or other models used in the agent, or use different types of models in the agent.

6. 1:27

   It can happen in the harness layer, where it brings the context, proper context to the LLM, uh, with components like prompts, skills, tools, code, workflow. And it can also happen in the memory, either in session memory or persistent memory of the agent.

7. 1:48

   But there are two, I would say, fundamental challenges in continual learning for agents. The first challenge is how to get feedback. How do we know if the agent did well?

8. 1:58

   And if not, what should it have done instead? That's basically the first part, getting the feedback. And the second part is how we can act upon that feedback, how we can optimize and improve agent and learn from that feedback.

9. 2:13

   Which layer, which component do we need to change, and also how? I'll be talking about these two challenges, current, uh, approaches in order to deal with them, and also provide some perspective of how we think about these two problems.

10. 2:28

    So let's get started with the first problem. Where does the feedback come from?

11. 2:34

    So the easy case is when we have a benchmark and some evaluators on that benchmark, so the agent can run tasks from the benchmark. Now we have the evaluators in order to score, and we can get grades like pass, fail, or reward, as well as potentially some feedback on the agent, uh, behavior and agent performance.

12. 2:55

    This is usually what is happening during the development time, that, uh, different teams, they curate benchmarks in order to understand the performance of the agent in, uh, certain applications.

13. 3:08

    But in production, we don't have such benchmark. We have logs. Here's an example of a session log where a user is interacting with the agent. Maybe the user is not very happy with the way, way the agent is behaving, but we don't have any explicit feedback.

14. 3:23

    So there are two ways of getting such feedback on such session logs. One is automatic, using some other models or LLMs or code in order to analyze the log and provide feedback.

15. 3:38

    In some cases, even the a-agent itself can look at its log and provide some critics of it. It is automatic, and it is scalable. And the second approach is where we have human experts to look at some handful of these logs and provide some domain expert, expert feedback on those, uh, agent outputs.

16. 3:59

    Uh, this is lower in the volume, but it is critical because it provides expert, uh, knowledge on, uh, the behavior of the agent, and it is alignment with the way that we want agent to behave in those applications.

17. 4:12

    Either way, now we have session log plus some feedback on those logs. Is it enough? The answer is no, because it is still not testable. Here we have log and feedback, but what we really need is a replayable learning environment, a simulation that we can rerun with defined grading on what success looks like, not one

18. 4:37

    instance of what happened and the feedback on top of it.

19. 4:43

    So what is a learning environment? Here we are inferring a distribution from one observation that replays what happened and what success means. Uh, the input is what we have, some session logs and feedback.

20. 4:58

    This is basically one observation of what happened. And now we wanna create a simulation and evaluation environment from that information. That involves, for example, understanding how tools in the agent, uh, behavior in the agent log should behave.

21. 5:15

    Should we use real tools? Should we use mock tools? And if so, uh, what kind of data should be brought to, uh, the mocking process of those, those tools?

22. 5:25

    If the agent is interacting with some users, how we can infer synthetic users from that data, and also how success looks like in that learning environment. What are the evaluators that needs to be inferred?

23. 5:40

    So there are lots of technical challenges in any of these, uh, components. But if we could do this successfully, the good news is the output is executable. We can now run different candidates of the agents against such learning environments, understand the behavior and the performance of the agent in those scenarios and in those patterns, and we can

24. 6:02

    fix the issues, um, based on the feedback, based on the information that we observe, because now everything becomes testable and verifiable.

25. 6:15

    All right. So the second problem is now we have this feedback, how we can act upon it? How we can optimize the agent?

26. 6:25

    And from high level point of view, there are three layers that we can improve the agent. Uh, there's a model layer where we can change the weights of the model, and there are methods like SFT, so, uh, supervised fine-tuning, uh, RL-based post-training in order to make those changes.

27. 6:42

    And these are usually expensive because that requires, uh, more intensive compute in terms of changing the model weights. The second layer is the harness layer, harness engineering, context engineering, where we can potentially rewrite prompts, maybe learn some skills, uh, change tools or add tools, maybe change code around, around the LLM.

28. 7:05

    And there are different methods like GEPA, trace-the-harness, uh, which provides a lot of flexibility in terms of learning from that feedback. And the last layer is the memory layer, where we store facts and learn skills, uh, in order to not repeat those issues, uh, and failures in, in the future.

29. 7:23

    But the good learning is not going to be focusing on any of these components exclusively. A good learning engine should ask for the smallest durable change at the right layer of the agent.

30. 7:42

    All right, so let me, uh, dig a little bit deeper into each of these layers. So first, in terms of updating the model weights, there are various approaches, uh, in order to do that, uh, including SFT, supervised fine-tuning, where we imitate correct trajectories.

31. 7:58

    Uh, we often need labeled samples in order to, um, uh, fit those samples, uh, fit the model to those samples. Other approaches are based on RL, um, reinforcement learning post-training, like DPO, GRPO, RLVR, where we sample the score against the reward or preference signals and reinforce what wins.

32. 8:19

    And there are some categories based on LoRA, low-rank adaptation, that limits the set of parameters that can potentially change. It makes this, uh, the learning in this layer cheaper and also safer, uh, in terms of the updates.

33. 8:33

    But these methods, they usually need benchmarks and explicit evaluators. They cannot be directly applied on, let's say, if you have a log and feedback, unless we turn those into, uh, replayable learning environments.

34. 8:48

    Right. In terms of l- uh, updating the harness, uh, I would highlight the two, uh, categories in, in this domain, in this layer. One is trace-the-harness approaches. Let's say you observe a log, you have some feedback on top of it, you can effectively ask a coding agent in order to analyze the log and improve the agent.

35. 9:09

    So this works on, uh, the case where we have log and feedback, but it is vibe-based. We don't know if even for that particular sample, if the change is effective because it is not testable, and we don't know what is the impact of it on other samples and other scenarios.

36. 9:30

    What, uh, might have been working previously, but with these changes might not work properly and create some hidden regressions. The other, uh, category that I wanna highlight is methods like GEPA and prompt search, where they mutate prompts, uh, they score different candidates, and they keep, uh, the winners using some search algorithms like evolutionary algorithms.

37. 9:52

    And these are-- these methods are testable, but they need benchmarks and explicit evaluators in order to have those, um, have those scorings.

38. 10:04

    And in the memory layer, uh, we effectively write down facts, and we distill skills so the agent doesn't rediscover them. Uh, it can happen in the information, um, memory layer, methods like LETR and MemZero, where they can effectively store a fact or correction.

39. 10:25

    And we have also methods, uh, through skill distillation that compress a successful trajectory into reusable how-to-do skill for the, for the agent. So this layer, in terms of the update, is cheapest and fastest.

40. 10:41

    It works directly on the cases where you only have log and feedback, but usually it is unverified because you don't, uh, have a way in order to test whether or not, uh, writing in a memory will resolve the issues that you have dealt with and whether or not it can potentially, um, create some regressions on some other

41. 11:01

    cases. With that, let me introduce a new subcategory of continual learning called verifiable continual learning.

42. 11:10

    In a verifiable continual learning, the goal is to improve an agent from its own experience, where every fix is proven to help and proven to break nothing that already worked.

43. 11:22

    And usually it involve-- it involves three steps, where we need to have an executable test where the failure becomes a task you can replay and grade. Then we need to have a measured delta where the update is scored on the test before and after.

44. 11:36

    And then we have a regression test. So prior tests still pass even after we make such changes to the agent.

45. 11:47

    So let's think about what are the principles of a practical verifiable continual learning first.

46. 11:54

    And I will argue there are four important principles that we need to keep in mind. So the first principle is replayability. We need to turn a one-off failure into a test that we can rerun.

47. 12:07

    Here, as I had mentioned previously, many cases we have log and feedback, but that is not testable. We need to lift it in a learning environment to simulate and evaluate the agent on a similar pattern, on a similar scenario, so everything becomes testable based on that simulation and evaluation environment.

48. 12:31

    So that's basically the first principle that we need to have. The second principle is holisticness. One failure may have several causes and several possible repairs. You know, let me give you an example.

49. 12:43

    Let's say you have an agent that cites a stale policy and skips the required escalation. The issue might come from the memory where you have some stale fact. It can come from, uh, not optimized prompt.

50. 12:55

    It might come from a tool that doesn't normalize the policy. It might come from the workflow that we need to add escalation gate before refund. It might come from the model.

51. 13:05

    Maybe the model is not a good model in order to have a strong reasoning. So here we need to route the fix to the right layers that explains the failure with the smallest durable change to the agent.

52. 13:20

    And that is basically the principle of holisticness in verifiable continual learning. The third principle is lifelongness. A new fix must improve the new case without breaking the past. Let's consider this setup where we already optimized the agent on K past learning environments, and a new failure comes, and we turn that into a learning environment, EK plus one.

53. 13:44

    What do you want to change? So the first approach is, okay, just focus on this new learning environment, but that can create regression on the past behavior, on the past learning environments that the agent was successful.

54. 13:59

    A better approach is a regression-aware learning, where the regression is not be treated as a post hoc approach, but as a mechanism within the optimization itself. So here we are fixing the recent failures subject to having no regression on the past, uh, learning environments.

55. 14:19

    And obviously, this needs to be done in a efficient manner, so it doesn't scale even linearly with K because K can grow, and the complexity of this approach can, uh, can, uh, can be very, can be very high.

56. 14:38

    And the last but not least principle is the efficiency. This continual learning loop needs to run frequently, and we need to have efficiency in, uh, different layers in updates to the agent.

57. 14:54

    So sometimes the change can be cheap, like for example, writing something in the memory can be medium in terms of the complexity by changing the prompt or harness, and sometimes it can be very expensive by

58. 15:11

    changing the weights of the model. Also, efficiency should be in the optimization loop itself, especially when we have regression-aware optimization, and regression is treated within the loop, uh, not as a post hoc approach.

59. 15:31

    To sum up, these are the four principles of a practical verifiable continual learning: replayability, holisticness, lifelongness, and efficiency.

60. 15:44

    And this is what we have been working on at RELAI to create a verifiable continual learning engine for AI agents based on these four principles.

61. 15:56

    In particular, here's how RELAI's learning loop, uh, runs. You can start with some signals to, uh, to this loop. It can be logs, feedback, or even instructions and prompts.

62. 16:11

    We lift those in-- those signals to replayable learning environments. So that's based on the replayability principle. This makes everything that follows testable and verifiable. Then we do root cause analysis and route the fixes to the right layer of the agent.

63. 16:31

    It can be memory, it can be model, or it can be harness. That touches the holisticness principle that I described. We have regression-aware optimization. Regression is not being treated as a post hoc approach, so that touches the lifelongness principle that I mentioned.

64. 16:51

    And obviously, this loop should run efficiently. That touches the efficiency principle. So the output of this is a reviewable version update to the agent explaining what changes in the agent during this loop and why, uh, those changes are, um, are improving the agent without creating regression.

65. 17:17

    The beauty of it is you can add a VCL, verifiable continual learning, to your current agent in just two commands. So the first one, uh, is a one-time setup.

66. 17:30

    Uh, you create a learning harness in your agent. You can use your own LLM, and, uh, your, uh, agent can be, uh, built on top of any of available major agent frameworks.

67. 17:44

    And then after that, you need two commands in order to activate this learning loop. You can create learning environments using various type of signals that you can have, either a log, logged feedback, or some instructions, and then you can call RELAI optimize in order to use holistic lifelong optimizer to improve the agent.

68. 18:05

    And the output is a optimized version, a pull request that you can review, and you can use it in order to improve your agent.

69. 18:16

    So let's look at, uh, how it actually works in practice. Uh, we built a Continual learning benchmark on a fictional support agent case, uh, where we have reproducible test beds for, uh, continual learning in a tool-using support agent.

70. 18:36

    So we have a single source of truth, and the policies are interacting for, uh, this agent to be handling. So we have deterministic evaluators, and we also build this benchmark in a way that it has some regression traps.

71. 18:54

    So if the optimizer focuses on overfitting on the latest fix, it can potentially break, uh, whether, uh, what the agent was previously successful on other tasks.

72. 19:08

    So let's say we have an agent. Um, we don't even have logs or anything, and we wanna just see how the agent is behaving, uh, let's say when a caller is rude and adversarial.

73. 19:21

    So simply, we can create a learning environment using such instruction. And what it will do, it will create a learning environment to simulate and evaluate the agent. Uh, the simulator will include personas, intent, mock, uh, or real tools, and also the learning environment contain evaluators in order to define success metrics.

74. 19:45

    All of these are produced from just one interactive command.

75. 19:51

    And after that, we can just simulate the agent using this learning environment, see how it behaves. Okay, the score is, uh, not too, uh, high. It is seventy-eight percent, and, uh, in particular, there are two, um, evaluators that, uh, basically show very low scores, uh, of agent in this environment.

76. 20:13

    So these are some of the failures that we observe. How to improve such failures? We can do that by calling RELAI optimize with certain number of rollouts. And as you can see, the average improvement, uh, can be, um, uh, quite high.

77. 20:28

    Uh, it is ten percent improvement on average just with one loop, and the score increases to ninety-seven percent from eighty-seven percent.

78. 20:37

    Okay. Now let's consider the case that now the agent is in production. Now you have a log, you have an agent session that isn't, um, not desired, and you have a feedback.

79. 20:48

    For example, you can say, "Keep fast eligible refunds, but do not generalize generosity beyond refund thresholds." So that's a feedback on the agent behavior. Again, the flow is the same.

80. 20:59

    So we lift it into a replayable learning environment, and we call RELAI optimize in order to mitigate this, uh, issue. Use this feedback without creating regression of the agent behavior in past environments.

81. 21:14

    And this is lifelong, so you can keep doing that, uh, to improve the agent without breaking what already works, and it is compounding. This is verifiable continual learning in practice, where each update is tested, every gain is measured, and nothing that already works breaks during this optimization.

82. 21:35

    So that's it for, uh, today and for this talk. So there are three, uh, key takeaways that I wanna highlight. The first one is agent continual learning is not necessarily model fine-tuning.

83. 21:48

    The updates and many useful updates can happen in the harness and memory layer. So the second takeaway is production logs are not learning environments. We need to transform them into replayable learning environments to simulate and evaluate the agent on the same patterns and scenarios.

84. 22:04

    And the third takeaway is that the frontier is regression-aware continual improvement, where when fixing the new failure, we verify that we don't forget the old ones. We don't create regression.

85. 22:18

    So that's verifiable continual learning built on four principles: replayability, holisticness, lifelongness, and efficiency. And if you wanna try, um, our VCL and apply it to your agent, you can, uh, use it today at relai.ai.

86. 22:33

    Thank you.