Keynote: Why people think "agent" is a buzzword but it isn't

Talk transcript

1. 0:00

   Hello, my name is Chip. I started an AI infrastructure startup a few years ago, and after selling it last year, I have been happily unemployed. Before that, I was with NVIDIA, Snorkel AI, and also taught a, a couple of courses at Stanford.

2. 0:16

   I have, uh, my, uh... for today, I want to talk about the challenges in building agents or why people think agent is a buzzword and why I think that it's not.

3. 0:26

   I have been wanting to come to, like, the AI Engineering Summit for a long time, but Swyx, like, never invited me until very recently. I shared the session of agents, uh, from my book, AI Engineering.

4. 0:38

   It's a very long session. It's, like, 8,000 words, and people seemed to like it, so Swyx invited me here. And, um,

5. 0:45

   I actually, like, prepared, like, another talk for, uh, for, for the summit. But then after watching a lot of talks yesterday, I realized that people have covered a lot of ground, so I create a new talk hoping to cover, like, newer, like, more exciting topics.

6. 1:02

   So this is a new talk created especially for this conference, and I hope you like it.

7. 1:09

   I heard that if you are to give an agent talk today, you're obligated to define what an agent is. Um, I know that a lot of people think, uh, oh, there are a lot of talk about, like, agents is just hype, but I don't think so.

8. 1:22

   I think there's a lot of, like, exciting use cases for agent, but I guess I'm, like, preaching to the choir. So agent is not a new term. Uh, when I was working on my book, uh, I decided to, like, look at a lot of books from, AI books from the '80s and the '90s, and try to understand,

9. 1:39

   like, how people define agent back then. And a definition that's, like, really, uh, resonate with me, uh, is from the book by, um, by Stuart Russell and Peter Norvig.

10. 1:49

    So they defined an agent as anything that can perceive the environment and that acts on the environment. So let's say that you have an agent that play chess, that the chess board is its environment and its actions are the chess moves.

11. 2:04

    Um, ChatGPT, right? ChatGPT can interact with the internet, so the internet is its environment, and it can do actions like web browsings. It can also use calculator. It can also, like, generate text and images.

12. 2:18

    So one of the most popular use cases of agent nowadays is, like, coding agent, and here is from the paper, like, SWE- um, SWE-agent paper. And as you can see here, the environment for SWE-agent is a computer with terminal and file system, and the list of actions it can perform includes navigate repo, search files, view files, edit

13. 2:40

    files. So the environments, like, determines the kind of actions that the model can perform. So if you're in a game, if an agent is in a game, it can only perform the action that the game allows.

14. 2:53

    At the same time, giving the model more actions can also help expand its environment. So if you give the model the ability to browse the web, then now the internet becomes its environment.

15. 3:05

    There are many reasons why, like, we would want to give a model access to actions. So first, actions can help address a model limitations. So all the models have the cutoff date, and that make it, like, pretty hard to, like, answer questions that require new informations.

16. 3:22

    By giving the models access to, like, newer APIs, su- such as, um, news or weather, web browsers, the model now can get, like, relevant recent information to answer questions.

17. 3:34

    A very common limitations that people discover very early on with AI is that AI is pretty bad with math. So instead of trying to train a model to, like, be really, really good with numbers, you can simply, like, give the model access to a calculator.

18. 3:49

    Another thing is a very exciting use case that it can turn a text-only or image-only models into a multi-model model by giving it access, uh, to tools, um, or actions.

19. 4:00

    So for example, um, given a language model, right? A language model can only process text and output text. So if you want a model to, like, also be able to process image, you can...

20. 4:11

    I can give you access to, like, say, an image captioning model. So, like, given an image, it can, like, use this tool to generate captions, and then use a caption to generate a response.

21. 4:20

    Now the model can process both text and image. It's very cool. But something that's, like, even more cool, so I think it's, like, why agents are so exciting, is that, like, actions allow you to embed models into the workflow.

22. 4:34

    So now, for example, you can give the model access to your inbox, uh, your Slack, your calendars, or the code editors so that the models... so that you can use a model into digital workflow instead of having to open, say, like, a, a web browsers so that you can use AI.

23. 4:51

    So when I'm talking about, like, agent, uh, people will always ask me, like, "Okay. If agents are so cool, so why aren't everyone... why, why isn't everyone using it?

24. 5:01

    Like, tell me." Like, everyone asking me, like, what, what would be, like... give me one good use cases of, like, agent. So why isn't everyone using it? It's because, like, doing agents is, like, really, really hard.

25. 5:13

    So for the rest of the talk, I will cover, like, a few reason why, like, doing agents is so hard. So we're gonna start with the curse of complexity.

26. 5:23

    So we know that, like, task failure rate increases as the task complexity increases. This is true not, not just for AI, but also for human as well. Like, if you're given more complex task, we will be more likely to fail.

27. 5:36

    So let's say that you're building an application for your company, and you're okay with a failure rate of, like, say, um, 1 or 2%, right? And the model makes, like, makes mistake, like, 2% of the time for one step.

28. 5:50

    So over 10 steps, the mistakes, the model can make mistake about, like, 80%, 18% of the time, and that is a lot. That might be unacceptable. And, like, if you increase your number of steps to, like, 100 steps, the model become, like, almost, like, worthless.

29. 6:04

    Like, it could make mistake most of the time.

30. 6:08

    For a lot of agent use cases, um, a lot of u- a- agent use cases are pretty complex and might require mu- multiple steps to solve, uh, to solve them.

31. 6:18

    So it's not that you don't want to use agents for, like, simple tasks, but, like, simple, simple tasks just don't need, don't usually need agent to do. And also simple tasks might have, like, lower economic value, so, like, they are less exciting for agent to solve.

32. 6:37

    Uh, let's, let's go through, like, a very, very simple example. Like, let's say that you want to ask the agent to, like, how many people bought product from, like, Company X last week?

33. 6:47

    So this is a very, very simple query, but the agent might need to break it down in, like, in several, in several steps. Like, it might first get the product list of the Company X, and then for each product in this list, it will want to get the number of, like, order for, like, last week.

34. 7:03

    And then given all this number co- um, order counts, it would have to sum it up. And then given this numbers, it has to generate a response to the users.

35. 7:11

    So even with this very, very simple query, very simple task, there were, like, like, four steps. And the more steps there are, like, the more complex queries, even, uh, the higher number of steps, and the more likely the agent is going to fail.

36. 7:25

    So in, in the, uh, a vast majority of, um, agent use cases I'm seeing right now, um, it's very, very rare to see them, like, consistently being to solve tasks that involve, like, more than five steps.

37. 7:39

    I do believe that enabling agent to handle more complexity would unlock many, many new use cases.

38. 7:46

    Um, so this is tricky questions. Um, how to know what complexity does your agent, your agent can solve? Because you want to give agent, um, the, the task that, like, has a right level of complexity it can solve so that it doesn't fail and, like, cause, like, catastrophic, uh, business failure.

39. 8:06

    So to... So, um, different kind of tasks, different use cases have different definitions of complexity. A very, a very, very common way to define complexity is by the number of steps needed to solve the task.

40. 8:18

    So this is by, um, this is, like, a synthetic planning benchmark that I'm working on, and I'm hoping to, like, publish very soon. Um, so I use, I use synthetic data set, uh, synthetic benchmark because it allows me to, like, control the level of complexity to study a model behavior.

41. 8:36

    So now I can ask the model, like, generate, like, tasks that require, like, five steps to solve. So, so, so with that, um, um, so in, in my benchmark, most models perf- don't perform quite well.

42. 8:49

    Like, most model can only solve, um, tasks, like, that have, um, at most, like, five... That require at most, uh, five steps. And after ten steps, um, most models fail.

43. 9:01

    And this is, like, consistent with another studies that I have seen. Uh, it's an older study now. It was, um, from, like, twenty-twenty one. So it's in, uh, it's in coding.

44. 9:10

    So the results, so the actual pass rates for the task for models must have increased a lot by, by now. However, the, the, the learning, the insight, uh, is still, like, I think still very lev- relevant.

45. 9:25

    So in this, um, in this paper, they try to construct, like, different docstrings, and then they ask the model, the agent, to, uh, to the model to generate code based on the docstring.

46. 9:37

    So they, they count the complexities. They consi- they measure complexity, uh, of the task based on, like, how many steps needed in the docstring. So for example, like, for this task, like, first you want to con- you ask the model to con- write code to convert the string into lowercase, and then you ask the model to write

47. 9:55

    code to remove half of the characters in the string. So, like, this c- are considered, like, two building blocks or, like, two steps in the docstring. And they found out the same result, like, um, as, uh, as I did, is that the success rates, the pass rate, like, decrease rapidly as the number of steps increase.

48. 10:16

    Um, but the good news is that, like, with newer models, um, they are get- actually getting a lot better with planning. So here in the same, uh, results, you can see this, like, here's this three very nice curve that come from, like, DeepSeek R1, Gemini 2.0 Flash Thinking, and o1-preview.

49. 10:35

    I didn't test on, like, o1 and o3 because I didn't have access to this model when I run this test. And you can see this, like, the curves are being pushing upward.

50. 10:45

    Like, the models, the newer models are able to solve, like, tasks with more complexity. And I do believe that this is gonna increase over time, allowing us to, like, using agent for more practical, complex, real-world task.

51. 10:58

    Um, so here's another, uh, result from my benchmark. So as you can see here, it shows the number of tasks that each model was able to solve. And in overall, you can see there's, like, there's a pretty big difference between, like, newer reasoning models such as o1-preview, DeepSeek R1, and Gemini Flash Thinking, and non-reasoning models just like,

52. 11:19

    uh, Sonnet 3.5, Gemini 2.0 Pro, or, like, GPT-4o.

53. 11:27

    Different use cases might define, uh, different, um, the complexity differently. So here is a paper from Zebra Logics. This, like, just came out, like, just last month, in which, uh...

54. 11:38

    It's a, it's a logic task. So they define each problem complexity by its number of, like, Z3 conflicts. So you can see this, like, by the... They also got the same result.

55. 11:49

    Like, the model success rates, like, decrease rapidly as the number of, um, as the number of Z3 conflicts increase.

56. 11:59

    So I think there are several tips to get the agent to handle more complexity. First, we might want to break tasks into sub-tasks that agent can solve. So you don't want to give an agent a task more than it can handle.

57. 12:11

    So let's say that a task, uh, your tasks, like, consistently require something like five or six steps to solve, and the agent can maybe like solve it, or can do at most like three steps, then you might want to break the task into like two sub-task.

58. 12:26

    Another way to like have the model deal with more complexity is do like test time compute scaling. Uh, so I, I... Test time compute, uh, but I think that in the last few years people have been talking a lot about test time compute scaling, so it's one of the very, one of the new or very exciting concepts

59. 12:43

    that gave rise to like reasoning models, and I'm very excited about it. So the idea is that like you can have, you can give the model more compute during inference so, so that, um, so that it can either generate like, uh, using more, um, more thinking tokens, so it can think more.

60. 13:01

    Or it can also, it can, it can also use the compute budget to generate more, more output. So for example, given a math problem, it can maybe like output 10 different samples, 10 different solutions, and then pick the one that like the model, like most of this out- pick the ones that's most common.

61. 13:21

    Like most, um, that's what the model thinks output most of the time. So yeah, so it's test time compute scaling. You can also use stronger models. So using stronger models can also call like, uh, train time, train time compute scaling because now you need to invest more compute into like training bigger model.

62. 13:42

    Okay, so we finished the first challenge, which is like the curse of complexity. The next part we're talking about is challenge related to tool use. So tool use is basically like natural language, uh, to API translations.

63. 13:55

    And what does this mean? So a lot of time for agent, right, we have humans using agent, and the human gives the agent instruction in natural language. So for example, an agent or a human might give the agent a task like, "Hey, given this customer email, create an order."

64. 14:12

    So the agent, um, will need to translate that into like, uh, functions that can perform this task. So it might first need to call a function to extract the customer ID from the e-mail address, and then it might call another function to extract the order ID from the content of the email.

65. 14:28

    And then given this customer ID and the order, you would need to actually create the order. So now you can see that it can trans- it need to translate from this natural language to just a set of API calls.

66. 14:40

    The challenge with this is that the challenge comes from both sides of the, of the translations. So for natural language can be like extremely ambiguous, and at the same time on API side, you can have very bad AB- API and very bad documentations.

67. 14:55

    So let's go into the first example of like ambigu- ambiguous natural language. Consider this, uh, agent with access to very, very simple functions, like fetch top products and fetch product info.

68. 15:07

    So fetch product info can return you like the product price. So let's say like the, say that the fetch top products take in like three argument, like start date, end date, and number of products, right?

69. 15:18

    And then the user has this query, find best-selling products under $10. So now, uh, the, the agent would need, know that it need to call the fetch top products, but what would the start date be?

70. 15:30

    What would the number of product be? Like, how many products should it query? And what start date, what end date should it be? Like, would it be like from like, does the user want best-selling products from like yesterday, from last week, or from last month?

71. 15:43

    So this is very ambiguous. Okay, so now we talk about like very, very bad API or bad documentations. In my coding career, I have been like pretty like fortunate or unfortunate to have seen like really, really, really bad comments.

72. 15:59

    Um, so as an engineer myself, uh, I know that like people don't usually like writing documentations. And if you can't explain the function to the, uh, to, to, to the agent, it's gonna be really, really hard for the agent to know how to use this right.

73. 16:18

    So I do think just like when you, when, when you give an agent, like access you a tool, you will need to provide necessary documentation. As a list, you should need to explain like what the function does, what parameters it take in, like what is the types of parameter, what does the parameter stand for.

74. 16:35

    You also need to show like err- different error codes for the, for the, for the functions, and also like expected like returned values. And the more details, the better.

75. 16:46

    And that's not all. Because like with error code, right, you don't just want like, okay, this model returns this error like step 99. Like it doesn't mean much for the model.

76. 16:56

    You might want to like explain to the agent like, okay, this error is usually caused by this. And if you enter this error, if you encounter this error, maybe this is how you should address this.

77. 17:06

    And one of the, uh, one company told me that like one of the very, very, one of the biggest improvement they got for their agents is after they explained and add to the documentations like how to interpret, uh, returned values of the functions.

78. 17:22

    So let's say that a function like return the value of like one, like what does that mean? So if you help the model interpret the result, the model can, the agent can actually be able to perform, like call the functions like a lot better and then be able to plan a lot better.

79. 17:38

    Um, another very important thing to think about is that like, um, it's very... If tool use for agents can be like counter-intuitive for us, because humans and AIs like have fundamentally different ways of like using tools.

80. 17:52

    So first of all, like, um, humans and AI have, have different like preference. So humans might prefer working with like visual thing, like with GUIs, whereas like AI might work better with like APIs.

81. 18:03

    So like if you ask a human to use Salesforce, they might go to Salesforce website. But if you like assign a task for AI, it will like, it would perform much better, like not having to deal with a lot of visual cue and just like calling the straightforward API instead.

82. 18:17

    And also like humans and AI operate in different way. Like Humans like, at, at least for me, uh, I find impossible to perform multiple tasks at once. So I would perform, like, different step by sequentially, whereas AI can perform tasks in, like, parallel.

83. 18:32

    So first of all, um, if you need agent to perform, like to browse, um, if you need to, like, browse 100 websites, um, it could be, like, very, very boring for humans.

84. 18:44

    Like hu- I did that for my book. Like, I browsed, like, thousand of websites, but it was not fun at all. However, for AI, like browsing 100 of website or, like, 1,000 of websites, extremely easy.

85. 18:55

    You can just send out, like, open like, uh, like query, like this thousand of websites and get back the summaries, and it's pretty straightforward. So that is actually a challenge for, like, training or, um, creating examples for the models to do, to, to do planning or tool use.

86. 19:12

    Because given a task, what the human annotator does might not be optimal for AI. So that's, that's, that's the reason why the reinforcement learning is so exciting because with supervised fine-tooling, like, uh, you are teaching AI to, like, clone human behaviors, which might not be optimal for AI.

87. 19:32

    Whereas with reinforcement learning, like you let the mo- the model figure it out, like with try and error, and it might find ways to do it that is optimal for AI.

88. 19:43

    So there are several tips, like how to make agent better at tool use. So the first is that you should create like very, very good documentations. Like, with everything, not just the function descriptions, parameters, but like error, like return values like we just talked about.

89. 19:59

    You should also, like, give agents, like, very narrow and well-defined functions. So I just caught up with a friend, um, working for a very big company. I, I wouldn't say the name, but if you say the search engine, you probably know what it is.

90. 20:11

    And he was saying that like for, for their use cases, uh, they give their agents like only three or four very narrow and small well-defined tools. For the agents, like they...

91. 20:22

    For their tasks, like the agents just did not work at all with like more than five tools.

92. 20:27

    You should also, like, uh, because of, like, the ambiguity of natural languages, it can help the models understand the task or the user query better by using techniques like query rewriting or using like intent classific- classifier to help, like, classify the user intent.

93. 20:44

    You can also, like, instruct or you should definitely instruct your agent to ask for clarifications when it's unsure of what users want. So for example, like if user asks like five, um, five best-selling products until, like, under $10, it can, like, make some random guess, like to fetch product from yesterday or from last year.

94. 21:04

    Or it can also like ask user like, "Hey, do you want top product, best-selling products from yesterday or from last week?" Um, you can also... One, one, one pretty exciting or interesting, uh, direction I'm seeing is that, like a lot of companies are building specialized action models for specific types of queries and APIs.

95. 21:24

    So we already have like specialized model, action models for coding, right? Now we have the model trained specially for, let's say, like VS Code or like for coding. Um, so why not have like specialized action models for different environment as well?

96. 21:39

    So for example, I've seen like a, a Salesforce might be interested in like building... Maybe I shouldn't say like Salesforce, but say general. Like different companies with very complex, like, ecosystem might want to like train action model for their environment.

97. 21:54

    Okay, two down, because of course the complexity is a tool use issue with natural language and API translations. The last one is context. And it's really funny because we have been talking about context for a long time, like first for RAG and now for agents, we still talk about context.

98. 22:09

    So models, like AI, has always requires a lot of informations. So before agents, like a model has already had to work, like system instructions, which can be pretty long if you, like, really want the model, really want to...

99. 22:24

    want the application to perform well and secure. So you might want to instruct the, the model to like what kind of queries it should respond to, what kind of query it should not respond to, what kind of tone it should carry.

100. 22:35

     And there are also like user instructions and like examples. But with agents, like you see a lot more, um, informations. So first you might need to pass documentations about the tools to the agent.

101. 22:47

     And the more tools there are, the more documentation will be needed. Um, of course, like after you call a tool, there can be tool outputs. That's a model we need to keep track of as well, and this will grow with more like execution steps.

102. 23:02

     And, um, after getting back like a tool outputs, right? The model may need to reason, like, "Okay, now I got this result. What do I do next?" Or like after a model generate a plan, the, the agent might also want to reason like, "Hey, is this plan like reasonable?

103. 23:15

     Should I execute it?" And all these reasoning tokens like take a lot of ti- take a lot of like, um, take a lot of input tokens, and this also grows with more complex tasks.

104. 23:26

     So like the informations that an agent can work with, like can grow very, very, very quickly. Um, and I haven't even mentioned like other kind of like information, such as...

105. 23:35

     such as like table schema for tasks like text to SQL. Let's say that you want to do like a text to SQL task, right? And you're not just... You, you don't have...

106. 23:43

     You, you have not just one table, but like 1,000 of tables. So when, when you translate a SQL query, you might need to figure out like what table to apply the SQL query to.

107. 23:53

     And for the model to be able to pick the right table, you might need to pass in like all the table schemas. And if you have like 1,000 like table schemas, that can be a lot of informations for the model to process.

108. 24:07

     So, so one thing that like I have experienced, like when I was working with agents that I would love to have more research on is like, um, how to make a model that works well with both planning and long context.

109. 24:19

     Because in my experience, like some models that are good with planning- I'm not necessarily the models that work with long context. And the reason is that, like, plannings are like reasoning, usually like, um, require a lot of reasonings.

110. 24:31

     Like, it require a lot of... generate a lot of thinking of reasoning tokens. So this kind of task are like output heavy, whereas for long context it's like input heavy.

111. 24:41

     And I have, in my benchmark, uh, my personal benchmark, I see in this like, um, models that perform well on my long context benchmarks don't perform as well on the planning benchmark and vice versa.

112. 24:53

     Um, so okay, so we've talked about, like, an agent, like, has to deal with a lot of informations, and that information might not fit inside a model's, like, efficient context.

113. 25:02

     So I want to highlight the word, like, efficient here, because a model might have very long context, but then it might not use that context, like, efficiently. So, like, a model might be able to fit in, like, a million tokens, but, like, if you give it anything more than, like, three thousand tokens, it might get really, really,

114. 25:20

     really funky and, like, hallucinate all the time. So at least, like, in my, in my personal experience, um, I have, like... Uh, yeah, so, so I've, like, done a lot, a lot of, uh, a lot of, like, um, benchmark evaluations just to see, like, at what point of the, of my documentation does the model start, like, hallucinate

115. 25:38

     and making up things. Um, so, so, like, if you can't fit all your information into the model context, like efficient context, you might need to realize on, like, other form of, like, information persistence or information, uh, storage.

116. 25:54

     So context, you can think of it as like a short-term memory. Like, you should use this for, like, uh... It should, it, it should be used to store information relevant to the task at hand.

117. 26:03

     And then you can also supplement it with, like, long-term memory. For example, like external databases or storage. Um, and it's very common with use case like RAG, right? Like, so if you connect a, a model to your external databases, then you're connecting it to, like, um, long-term memory.

118. 26:20

     So, um, you can also, like... In the case of agent, right, um, you can, like, store less immediate relevant informations in, like, external, external file. So let's say that your task requires, like, ten steps.

119. 26:35

     So maybe, and, and the output from this, all these ten steps, like, doesn't... don't quite fit into the context. So you might want to store the output of the first few steps into external file, and then we retrieve the output, like, when necessary.

120. 26:50

     And of course, like, there's also, like... So we have short-term memory, uh, long-term memory, and another level of, like, memory system is internal, internal knowledge, which is, like, the knowledge that the model already has.

121. 27:02

     So if you have some informations of models, like, that is essential for, like, the model to perform, like, multiple tasks, you might want to include that in the training data and fine-tune the model on it so that, so that the model can just use this as part of internal knowledge instead of, like, having to waste, like, context

122. 27:18

     tokens. Okay, so that is pretty much for today. Uh, so I think we talk about, like, um, what is a, what is an agent, uh, different challenges to building agents, including, um, like, first including, um, trying to, like, get the model to handle the right, the task of right complexity.

123. 27:39

     And we talk about tips, like how to make the model handle more complexity. We talk about tool use challenges of, like, how to translate between natural language and API.

124. 27:47

     And we talk about, like, how to help get model to, like, handle longer context with, like, a memory system. So thank you so much, everyone. Um, I do have a website, and if you have any questions or if you want to talk about the agent planning benchmark I'm working on, feel free to reach out.

125. 28:04

     Bye.