How to Improve Your Agents: Academic Lit Review

Talk transcript

1. 0:07

   Hello, everybody. Today I'm gonna talk about AI agents beyond ChatGPT. This talk is tailored for a more research-oriented audi-audience. My name is Jo. I'm from Columbia University. Uh, I'm also the founder of Arklex AI.

2. 0:23

   So many people are talking about agents. What are these? Bill Gates is very bullish on it, talking about the biggest revolution in computing. Andrew Ng is talking about this is the massive AI progress.

3. 0:36

   Sam Altman from OpenAI is talking about, uh, twenty twenty-five as the year of agent. We also see, hear, hear a lot of negative voice about, "Oh, these are just some wrapper of large language model."

4. 0:49

   Uh, they, they really can't plan, then we can't really have agents. And we're also talking about, oh, Auto-GPT is not great, and they can never solve practical solutions. But before we hear all these kind of questions, uh, can we talk about what exactly is AI agents?

5. 1:06

   So let's go back to the base, also in north. So agents is not new. It's really coming down to the large language model have powered agents to become more powerful these days.

6. 1:18

   First, we have the step we call perception. So like humans, agent, AI agent needs to understand the world, the environment, through sensing informations from text, image, audio, video, touch, and so on.

7. 1:33

   And then once we get the information through these sensors, we gotta have this go through this reasoning process. Sometimes people also call it inner monologue. To understand how do we process this information, how do we complete tasks, break down tasks into individual steps, and utilize these inputs of the environment to help us to better think what are

8. 1:55

   different tools, what are different actions to take. So this inner planning process, sometimes we also call them chain of thoughts reasoning, uh, that because it most of the time are powered by large language model.

9. 2:08

   And then we also have some meta-reasoning steps that we can perform, kind of reflect. Basically, whenever you execute a certain action, you can ask yourself, "So far, did I make the right choice?

10. 2:20

    If not, can I go back?" Finally, we talk about what considers to be actions of AI agents. Anything that you perform that may be talking to human, maybe moving from point A to point B are also ta-talking about...

11. 2:37

    are all actions. So in general, we're really talking about agents are interacting with the environment, uh, through actuations of actions. So in general, this is like the total process of agents.

12. 2:54

    Though it is difficult to understand how, how easy it is to, for us to deploy agents, we're just gonna use a nice analog of different levels of autonomy in self-driving cars.

13. 3:09

    So you can think about the very first base w-was a, was a chatbot. It started with twenty seventeen. Then you can just retrieve information and that's it. And level two is more about we call agent assist.

14. 3:22

    You have a customer service agent. They're using large language model to generate suggested responses, and people still has to be the person to approve the sending message. Level three is what we have been talking more right now.

15. 3:37

    It's called agent as a service. So we use large language model to automate the AI workflows that we have, and then usually using it as a service. For example, meeting bookings, um, writing a job descriptions.

16. 3:51

    And level four, it's really coming down to one person is not only doing one task at a time, it can delegate the AI to do multiple tasks. And these tasks have inter, intersections about, uh, sharing components, sharing knowledges and resources as well.

17. 4:08

    So this we call autonomy agents that can behave, uh, and perform ta- multiple tasks together. And then level five is one level above, is what we really think about the Jarvis in Iron Man.

18. 4:21

    So we trust the agents a hundred percent. We delegate the agents with all our security measures, like our keys, and then agents would perform on behalf of us. So self-driving car is, uh, having this similar five levels, right?

19. 4:37

    And but self-driving car, you can consider it as one example of agent because it's a agent, but doing the perception, doing the actions of reasoning about planning and trajectory of executing in terms of driving.

20. 4:52

    But self-driving is very high risk, which means that you have to make sure that there is no errors happening. Otherwise, we are talking about lives and de-- life and death.

21. 5:03

    Um, so but AI agents, what we talk about it, you always like can separate them into low-risk tasks to high-risk tasks. So when you talk about low-risk tasks, maybe there are some back-office tasks, like for example, um, filing reimbursements, um, which you can have human supervision and over time, trusted over time to be able to automate it.

22. 5:26

    Um, and then customer-facing tasks are usually considered to be more high-risk task. And over time, we can see people going from back office into front office over time to deploy these agents.

23. 5:40

    Today, we're gonna talk a little bit more about how do we improve large language models to l- to make it, to be able to reason better, to reflect better.

24. 5:51

    And then we're gonna talk about how do we elicit better behaviors from these existing large language models that optimize towards AI agent task. Finally, how do we learn from the examples or traces we searched over time to feed this information back to the large language model so that we can have a large language model that is optimized

25. 6:14

    towards AI agent tasks? So let's start with the first work, which is a NAACL Best Pa-- Outstanding Paper on self-improvements. So let's start with an example reasoning task, which is a mathematical reasoning task.

26. 6:31

    So here is an ex-- simple example. There are usually two ways that we can actually solve it through using large language model. One is on the left side, we call it few-shot prompting.

27. 6:43

    We generate examples of the same mathematical... similar mathematical problems and its r-- corresponding answers, and use them as a context, or sometimes we call it prompts, to a large language model, for example, GPT-4, and then get the right answer.

28. 6:59

    And the other methods we call chain of thoughts, which is instead of giving it example, we give it an instruction saying, "Let's think step by step." Um, so the models actually are following this step-by-step, uh, chain-of-thought process.

29. 7:14

    Sometimes we also call it reasoning over the tokens and be able to reach the correct answer.

30. 7:21

    Uh, more recently, uh, there is a prompting method that combined the two methods together, um, that actually improved the outcome. So instead of asking the model to just simply s-think step by step, we actually give some information about these data in the prompting in order to improve its performance.

31. 7:41

    So for example, we f-- give the same question, we ask the models to solve it step by step, and then we take the question and the initial answer and then prompt the large language model to generate feedback.

32. 7:57

    So based on the question and answer, what do you think about... Do you think the answer is correct or not? And then we have these code-generated things like in step two, the part blah, blah, blah is incorrect.

33. 8:09

    This is because blah, blah. So what w-we call this process is, uh, reflection or self-improvement. And then we combine the reflection or feedback together with the original question-answer, and then prompt the models again.

34. 8:24

    And then finally, the model is able to update the answer and the internal processes to get the, uh, final feedback correct. So what we call this process self-refine or self-improvement.

35. 8:39

    So we do find that this kind of, uh, prompt feedback could improve, and you can do it multiple times if you couldn't reach the correct answer. And then you can stop it when you feel a hundred percent that you are reaching the correct answer.

36. 8:53

    But in reality, we do find this, uh, interesting self-improvement process could be having a problem with smaller large language model. So here, previously, we were all talking about large language model that is beyond seven billion or thirteen billion.

37. 9:09

    But if you're using a smaller model that is cost efficient, like a Llama, uh, seven billion, we do find that the feedback that it generates sometimes contains what we call noise.

38. 9:21

    And these noise can propagate down to the correction steps as well. So this is what we usually call the blind is leading the blind. So we actually, instead of improving the results at getting even less results over time.

39. 9:38

    And then people may say, "Oh, these feedbacks generated by these smaller models are not great. Can we change it into a bigger model?" So you can think about these kind of feedbacks.

40. 9:47

    Also, nowadays, people call them verifiers. Uh, one of the interesting things that we saw is because these large language models demonstration or internal logics are not compatible with the smaller models.

41. 10:01

    For example, the steps are completely different. If we feed this kind of internal logic back to the smaller model, there is incompatibility, so the feedback is not useful at all.

42. 10:13

    So you can really think about this in human learning as well. If you find your kid is making a mistake on using pronouns, you can't just say that you, you should use a second person pronoun instead of a first person pronoun in this situation.

43. 10:27

    Your kid would never understand you. You should simply just say, "At this point, you should... I want it," instead of saying, "You want it." Um, so we have to dumb it down in terms of the feedback to cater for these smaller models' internal logic.

44. 10:43

    So this is where we come from in terms of how do we help the smaller models to also acquire the self-improvement process, uh, by distilling information from large language models.

45. 10:56

    So first, we're gonna reformulate the problem in terms of, uh, we wanted to retrain the model with attempts of, uh, solving the problem as an input, and then we're gonna generate feedback and updates.

46. 11:10

    And then we wanted to consider not o-- using smaller models to generate the feedback step. Instead, we're actually using large language model to edit the smaller model's feedback so that it's more tailored towards the smaller model.

47. 11:27

    Um, you can also use Python scripts for this particular math reasoning task because which gives you more correct feedback. So now you can think about our proposed method, TriPass.

48. 11:39

    Uh, we have the same question. We use the small model to prompt it to get the answer, and then we prompt the smaller model to generate the feedback. And then we ask the large model to edit this feedback and then use this cor-- uh, corrected feedback as input to further generate the updated answer.

49. 12:00

    And this process of, uh, correction and getting the feedback could be iterated multiple times until the problem is solved correctly. Because, uh, it's mathematical problems, you ha-- we have the ground truth in some of the benchmarks.

50. 12:18

    So we can actually utilize this, um, like you're just... For every problem, we'd be able to generate traces like that until we reach the correct answer. And then we can filter the trajectories of these kind of trial and error for a balanced set, and then use it to train our existing smaller models to be able to do

51. 12:39

    self-improvement with the guidance of larger model or other toolings like, uh, Python scripts.

52. 12:47

    Uh, we find like if we are using a weighted, um, self, uh, weighted supervised training proc- policy, and specifically we're talking about we have to train on the policy data.

53. 12:59

    We have to generate these, uh, feedback in real time because your models over time is gonna change. So the model has to be updated in real time. Um, we're gonna evaluate this on a big bench hard mathematical problems, uh, for arithmetic, word sorting, data understanding, logical deductions, and so on.

54. 13:20

    And then we... Through all these kind of different tasks we collected, we do find like after three iterations of using our Tripod. So when we say three iterations, like you co-- you provide feedback, you, uh, you edit the answer, you provide feedback, and you edit the answer.

55. 13:35

    You have these kind of a loop. By three times, you're able to reach forty-eight percent, which is, uh, even better than using some supervised trained data to do it.

56. 13:47

    And then the question is really coming down to, we improve the performance, which means that these kind of on-policy self supervision is useful. But if the model fine-tune with the data that we, we talk about self-improvement, it really learns the self-improvement in itself.

57. 14:04

    Uh, so we do find in this, all the four tasks, we w- looked at, um, the first time correction, which is the pink one, and then if the second time, later ones, it was correct after self-improvement, the dark pink, it actually was, um, obviously happening all these kind of different fun tasks.

58. 14:22

    And, uh, the range is pretty high in terms of seven to eighteen percent.

59. 14:28

    So this question is really coming down to, um, this is what definitely we're ha- getting extra answers correctly. But where does these kind of delta really come from? As if, can, can it just obtain from fine-tuning on gold answers, um, instead of, uh, giving them the internal logic and, uh, and policies?

60. 14:47

    So we do find that if you give on-policy data, we do much better. And then you just use simple SFT data, you just, like the improvement is cutting in half.

61. 14:58

    So the takeaway is very simple, that we can actually improve these, uh, models' reflection or self-learning abilities without explicitly u- human and supervision data. We could use these sy- synthetic ways to generate data policy to help the models to improve.

62. 15:17

    And of course, we have the limitation that is that what we call about the verifiers or the editors, which are large language model. If these large language model cannot edit it correctly, then we always have the ceiling effect as smaller models wouldn't be able to distill these information from these large language model.

63. 15:35

    So we're still hoping there will be a big language model that's pushing the boundaries of all the reasoning tasks. And then we can actually use these kind of, um, uh, syncing in by step or syncing in, in scale, uh, test time scale in order to be able to use smaller models to catch up to large models' ability.

64. 15:55

    Now, we talked about a meta-reasoning type, uh, we call a reflection. Now, we go into, can we actually elicit better, uh, results or planning results from large language model itself?

65. 16:09

    We all know in large language model area, there are three factors that would impact the train, the training of, uh, the training results of a large language model in the train- pre-training process.

66. 16:21

    Um, the number of compute, uh, the, the data size, and the parameter size. And we definitely see it's cor-- it's like linear correlated with how the performance look like.

67. 16:32

    So this is what we call, I guess, the scaling law. We still believe that currently, uh, we're not reaching towards the ceiling of scaling law, but we are close.

68. 16:41

    Um, we, we're relying on more GPUs, but unfortunately, data size is just... We have one internet, as Ilya talked about in Europe. How do we generate more data? How do we get better data, uh, is a million-dollar question.

69. 16:55

    And then parameters-wise, we're already reaching like billion parameters. If we increase it even more without the support of abundant diversified data, uh, the models are harder to improve as well.

70. 17:08

    But, um, pre-training, it's difficult. You could... Y- y- you can do it with, uh, if you're now Google, if you're now OpenAI. For smaller companies or academia, what kind of things could we do?

71. 17:20

    So there is a couple of new work related to GPT-4.01, um, is g- getting us a new direction, which is called test-time compute scaling. So instead of pre-training a large language model, you take the model as it is, but give it more steps or budgets in terms of inference.

72. 17:38

    Um, so specifically like what we talked about, I said to sync by step by step, I said to do reflection, and so on. You can actually reach better results in the end of the task, right?

73. 17:51

    So given the budget, we see it's like a linearly increased in terms of, um, task success. So can we actually further, further elicit better behaviors with existing pre-trained model?

74. 18:04

    And today is what we're gonna talk about, how do we elicit stronger model behavior through chain-of-thoughts like processes, uh, that is even more on the tree search side that is more complex, but it gives you better results.

75. 18:17

    And finally, how do we further improve the base large language model by using the data we generated from these incentive processes?

76. 18:27

    So let's start with a, um, very concrete task. Um, I have been working on dialogues for many years, and then this is one of the example. Um, but many dialogue tasks, you can think about it as a sequential decision-making.

77. 18:41

    You're at a certain point, talking to another person, you need to make a decision given the conversational context. What should I say next? So this process is you also have a look ahead, right?

78. 18:53

    Uh, in terms of, oh, this is my plan. How do I strategize my conversational strategies to pro-- to do some of the tasks? For example, we have the, the donation persuasion task.

79. 19:04

    The boy on the left is trying to persuade the girl on the right to donate to a charity called Save the Children. So they started with a conversation saying, "How are you doing?"

80. 19:13

    "I'm good." And then we talk about some of the strategies people use for persuasion. Uh, we call it inquiries. "Great. Have you ever donated to charities?" Right? And then given the user's response, "I'm in the right place at the right time, then I would do it," um, this boy needs to think about and ahead of time what

81. 19:32

    kind of strategies to apply at this point in order to convert this persuadee.

82. 19:38

    So let's take a step back and think about this, uh, sequential decision-making in a more toy problem, for example, like chess. It's really coming down to, uh, you have to think about w-w-what's your next move?

83. 19:50

    And you can think about what's your move in a more efficient way. It can simulate if you do this, what your compon-- opponent is gonna do, right? So you kinda have these kind of what we usually call planned out or like a, a simulate and evaluate process.

84. 20:06

    And we do find that in chess games, grandmasters usually have many steps ahead of time in order to see what is your best practice at this particular moment. Um, so this is very similar to AlphaGo and all these kind of algorithms that we're talking about that uses what we call tree search.

85. 20:26

    So the idea is very simple. You propose a certain move, and then you simulate, um, what are the, what are the v-values or like changes after you make this move.

86. 20:36

    And then you eval in terms of what are the outcomes of this particular move. And then you do this multiple times until you can s-- you obtain a stronger move by si-simulation.

87. 20:50

    So in our, uh, EMLP twenty twenty thirteen work, we basically brought, brought the similar ideas of chess into a conversational setting, which has a very clear goal. Uh, we don't really need any training data.

88. 21:05

    We hopefully could tr-- uh, be able to train this model with simulation. So we start with, uh, so multicolor tree search. We call it, uh, zero training. How do we design this multicolor tree search model is we're gonna prompt large language models such as ChatGPT.

89. 21:23

    So first, we're gonna have the search potential, uh, which is, uh, promising actions. You basically prompt a large language model to act as the policy, and then you simulate action outcomes.

90. 21:35

    So basically, if you say these p-persuasion strategy, what would be the outcome? We also do prompt a large language model, uh, evaluate. Then finally, we're g-gonna do evaluate action quality.

91. 21:46

    So once you perform this action, what are the quality of the actions after interacting with the environment? We also prompt a large language model. Um, finally, we basically update each action quality over time.

92. 21:59

    Uh, you will be able to have this process. And then when, when we talk about self-plays, right? You also have to simulate what your opponent's behavior. So here we used another large language model to simulate the user.

93. 22:13

    Again, we also use prompting large language model. So basically, given the conversational history, I want the models to generate the simulation.

94. 22:23

    So, uh, we want to see what are the prompts here particularly. We have, for example, prompting of negative reactions. That is one of the process. And then we want to see, given the prompt, what are the, um, evaluation of the success.

95. 22:38

    For example, is it a point two, point three, or point five? This is all very important in terms of how to evaluate the performance of a particular action. Traditionally, for these kind of MCTS with zero training, you would have a closed loop of MCTS.

96. 22:55

    Basically, it's given the history, what are your actions? But in reality, especially in our conversational task, we have to account for a lot of variance within humans' response. People are gonna try different things, um, so you can consider this process a stochastic, uh, process.

97. 23:12

    Instead of using closed-loop MCTS, we're using open-loop MCTS. So basically, given the conversational history, we're gonna stochastically, randomly sample one of the one possible simulated tasks. So basically, you can simulate different conversational strategies to persuade your opponent.

98. 23:32

    And then the idea of this particular task is that we have a very, um, objective way of evaluating it, which is whether the person donated to the charity or not.

99. 23:44

    But in itself, if you ac-can't actually run with hu- real humans, then it's really hard to have a good policy because it's very subjective, it's very hard to train.

100. 23:54

     So here we, particularly as I wanted to have GDP zero, which is our proposed method, to be able to com-compare it to without training, be able to generate a more stronger competitive results.

101. 24:08

     So we performed evaluations. First is we asked another large language model, for example, ChatGPT, to look at the conversations and look at the general outputs. One is using our planning algorithm, one is not using our planning algorithm.

102. 24:23

     Which one is more likely to, to be persuasive can lead to better outcomes of the task? And then, uh, on the other side, we have to run the human studies.

103. 24:34

     So basically, we release the model that is actually basically, uh, performing at the task of a persuader. We ask the mechanical turkers online to interact with this chatbot, uh, to see whether they wanted to donate or not.

104. 24:49

     So the models that we deploy with our planning algorithm actually definitely gets better donations. Um, and also we find that people find it's more convincing, more natural. We get more coherent as well because the process is optimized for the goal.

105. 25:06

     We also find through the analysis, we can learn that these, uh, models can self-discover a lot of task information. For example, we do find that, uh, they wouldn't do the task like, "Would you like to donate?"

106. 25:20

     in the very early on of the conversation because that usually, according to the literature, is also not good. We wanted to pave the way into the big ask. And we also do find it learns how to diversify its strategies as well.

107. 25:33

     It's not using one strategy, uh, only for in the entire conversation. It learns to use different persuasion strategies, emotional appeal, logical appeal to diversify its, uh, potential of convincing various different people.

108. 25:49

     So in general, this work really talks about, um, for dialog tasks, you can use multi-colored trees, uh, to simulate what are the possible behaviors, and then use these kind of policies to drive in real world these decision-makings.

109. 26:02

     Although this is not as clear as the chessboard that has very, uh, detailed and limited action space. But if you use these kind of quantifiable search spaces within dialog tasks, we can still gain better performances than not playing.

110. 26:19

     So of course, this is just a conversation task. Can we extend it into even larger AI agent space that not only perform actions of talking, but it performs actions of other tool use and, uh, um, manipulations?

111. 26:34

     Can we actually transfer the policy that we learned into other tasks?

112. 26:39

     So this comes to our recent work in ICLR, uh, uh, with the name of AEXACT. So for large language models to be able to do various AI agent tasks, we gotta teach it how to perceive the world.

113. 26:54

     And then we're gonna start with a visual, visual large language model. We call it visual language, large language model. Because this is... At least we wanna have to be able to process image as well as text.

114. 27:06

     So the traditional, uh, visual large language model, or we call, like, uh, VLM, are trained with tasks such as, uh, visual question answer. Like, what is he doing? He's performing a sta-- uh, skateboard trick.

115. 27:19

     But in reality, for AI agent task, we want the computer, uh, to do things, which, for example, the input is a image of your, uh, computer screenshots. And then the actions ways are, "Can you help me clear my shopping cart?"

116. 27:35

     And then you have to perform actions like click button of the shopping cart, uh, click delete, and so on. So the entire training process or entire usage of the current AI agent that we need, uh, for a large language model that is a multimodal, is very different from the previous VQA task.

117. 27:53

     So we really wanted to see if there is a way that we can adapt the traditional VQA, uh, visual language model into a more action-based visual language model, right?

118. 28:02

     If you look at, uh, humans can do these kinda benchmark visual web arena, like a, uh, clean my shopping cart, find me a green pair of shoes and things like that, uh, eighty-eight percent.

119. 28:14

     Well, if you just use a simple GPT-4V without anything in terms of plannings or whatever smart things, you get sixteen percent. So the absence of these kind of agent environment interacting with data and training, uh, is very lacking in this one, in these agentic tasks.

120. 28:31

     So here we want to see if we don't have the budget to retrain GPT-4o or do anything with it, is there a better way that we can actually elicit better performance through test-time compute?

121. 28:46

     So we introduced a new, uh, algorithm, uh, called RMCTS, which is short for Reflect Multi-colored Tree Search. Um, very like multi-colored tree search, it is search algorithm that can explore the vast action space on the fly, and also improves decision-making by incrementally constructing a search tree.

122. 29:06

     Unlike with simple multi-colored tree search, we extend it to incorporate a contrastive reflection. So we basically allows agents to learn from past interactions and dynamically improve their search efficiency.

123. 29:19

     Um, and also we're using multi-agent debate to get reliable state evaluation instead of using single, uh, prompting. Uh, so here we have two, uh, um, modifications that we did.

124. 29:33

     So the idea is that on top of performing a multi-colored tree search and improve its decision using a task, we can equip the entire system with a memory module, such that it can improve its behavior across different tasks.

125. 29:46

     Uh, practically, the idea is that after completing a task, we use reflective, uh, use contrastive reflection to help the model internalize, uh, the success or the error it has made and save this experience to a vector database.

126. 30:01

     So we basically cache the, uh, knowledge that we just learned. Then during any future task, the agent will also retrieve reflections similar to the current task, um, or the current, uh, computer state.

127. 30:13

     And to enhance its decision, we're based on making this as a new environment.

128. 30:21

     Finally, the RMCTS, we also introduced the method to improve the success val-- or the value estimate used during the... Uh, specifically, instead of directly prompting the m-language model to evaluate the progress, we find this evaluation process can be much more robust if we use in a form of a debate.

129. 30:41

     So basically, we're asking the models what it, uh, if it's a good action Why it is a good action. If it's a bad action, why it is a bad action.

130. 30:50

     So that you can have these kind of more holistic overview to counterbalance biases from one model to prod. Um, so basically through these two modifications of caching information and using more robust verifier, we're able to create this multi-colored tree search with a, a reflection.

131. 31:08

     Uh, during each task, the RMCTS builds a search tree on the fly by interaction with the env-- by interacting with the environment. And then our RMCTS use multi-agent debate to provide a more reliable state estimate and better guide the search process.

132. 31:25

     Then after each tar-- task, we-- RMCTS is now originally performs this contrastive self-reflection at the end of the task in order to improve its future execution. So basically, cache the information that we learned through reflection.

133. 31:41

     Um, to eval-- uh, to evaluate our method, we use two benchmark, um, Visual Web Arena and OSWorld, which are the most popular world-- uh, most popular AI agent benchmark.

134. 31:53

     So the Visual Web Arena is a benchmark that evaluates AI agent's performance on browsing tasks. So you have a browser that you actually do various things like you're doing a Reddit or doing shoppings and so on.

135. 32:05

     While the OSWorld is a benchmark that consists of computer tasks such as navigating the file system, using apps such as the VS Code, Excel and so on in a Linux, uh, computer environment.

136. 32:17

     So we first wanted to look at the Visual Web Arena. We find that our search or our algorithm outperforms other existing search algorithms such as breadth-first search, um, deep, uh, depth-first search A* and then vanilla MCTS.

137. 32:32

     We also compared our res-- our method to non-search methods like a simple ReAct or as well as like, um, other search methods that are from various different ways. We find that augmenting LLMs or the visual large language models with search algorithm can basically improve the performance without additional human supervision.

138. 32:53

     Um, so through these kind of cachings, through better verifications of the states, we're able to get be-better performance. Uh, we then basically also compare the, uh, another task, uh, which is O-OSWorld.

139. 33:07

     We can-- We're actually num-number one in the Visual Web or, uh, Vis-Visual Web Arena leaderboard. Uh, we also have the best non-trained methods in the OSWorld, which means that we only do test time inference instead of fine-tuning original models.

140. 33:26

     So given what we see, like autonomous center can actually improve agents without human supervision, can we actually consider transfer these additional knowledge we, we... that we obtained through the search into the training process of the base large language model?

141. 33:41

     And we basically, uh, also invented the algorithm exploratory learning. So compared to imitation learning, which is a direct transfer, train the model using the best action found in the tree, uh, we propose to use exploratory learning, which tree-- which treats the tree search process as single trajectory.

142. 34:01

     So the tree, basically the models will learn how to actually linearize the search tree traversal to motivate the model to learn how to explore, backtrack, and evaluate. So through this process of learning, teaching the model to do the exploration, instead of g-giving a, uh, the final good answer, we, we actually make the models to be able to

143. 34:24

     improve its decision processes by itself. So if we look at the, um, on the weight, if we let the model to do exploratory learning instead of imitation learning, which is, uh, the darker color, through test-time compute, basically, if you give it a, a, a succe-- a lim-- um, a constrained budget can improve the performance over time.

144. 34:50

     Um, here we basically provide an example, a trajectory after exploratory learning. For example, like this, given the task such as find the most relevant recent coffee maker with a touch screen, then comment with great item.

145. 35:03

     So this process, the agent basically learns how to perform the action, release it, uh, did not lead to desired state, then go back and try other actions. So this evaluate and backtracking process happen through the process.

146. 35:19

     So the model actually learn this process, which we find is very surprising and great.

147. 35:24

     So in summary, uh, or the reflective MCTS models that we introduced just simply relies on test-time compute scaling, is able to help the models to get better performance. And through exploratory learning, by training on the examples of research, we're able to actually further improve the large language model.

148. 35:45

     So there are many more things we can try that don't rely on human supervised information, which we can provide in an academic or smaller company setting. So for example, how do we get better RL methods to reduce reliance on the search tree?

149. 36:00

     How to do model predictive control methods to reduce this expe-- really expensive environment setup and interaction?

150. 36:09

     So some of the... I al-- so, so far we basically talked about how do we elicit better large language models ability through test-time compute, through retraining from refined, refined or backtracked exploratory traces.

151. 36:24

     So some of the recent things and the ongoing work that we have done is more focused on how do we improve controllability, and also at the same time, the autonomous exploration together for the agent orchestration layer.

152. 36:39

     So we just released our, Arklex open source, um, agent framework that has more features and more con-- compared to many other existing frameworks such as LangTrain. Uh, we have various interesting components that we have.

153. 36:53

     For example, continuous learning, task decomposition, which could give you, uh, developers more, uh, more flexibility in selecting different things. One of the things that we have been doing we think is combining not only the machine learning engi- machine learning expertise, but combining the system expertise, the HCI expertise, and also security expertise together, uh, to form a

154. 37:18

     group to learn how do we advance the system in a more deeper and practical way. So currently, all benchmarks I talked about is one single agent, uh, performing one single task that human assigns to you, right?

155. 37:33

     What if, like, you're, um... this is super inefficient to run this, like, in a separate th- instance. What if I have one human wants to-- the model to perform multiple tasks together on the same computer?

156. 37:46

     Then we will co- encounter all these kind of system-level problems about scheduling. How do we interact with database to avoid side effects? How do we have better securities in terms of knowing when to do human handover, when to act- actually request human supervision?

157. 38:02

     And then, you know, what's even more is if we have think about the entire process as a community. You will have multiple humans interacting with multiple different agents and assign different works.

158. 38:15

     The multi-users, multi-agent planning is even more complicated, can even lead to a lot of these kind of adversarial settings that we want to quantify. So nowadays, we're working together to establish more, uh, realistic benchmarks that have system integrations and also algorithm that will build on top to not only consider task completion, but also consider efficiencies, consider

159. 38:40

     securities, and so on, which will provide the, for the basis for all these kind of applications in the future. Okay, thank you so much for listening. Um, uh, if you're interested to know more about our, uh, framework that integrates all the research that we just talked about, please join us at, at the ArcLex AI