Why the Best AI Agents Are Built Without Frameworks (Primitives over Frameworks)

Talk transcript

1. 0:00

   Well, hello there. I am Ahmad Awais, and I'm going to vibe code an AI agent built with AI primitives while I deliver this talk, right? So let's start here.

2. 0:09

   One of the most common AI agents that we've seen in production is there's some data, and there's a chatbot as an agent, and you're trying to chat with that data because, well, all of these LLMs are self-attention algorithms anyway, right?

3. 0:24

   So you go to chai.new, and you say something like: "Chat with PDF."

4. 0:29

   And let's see what happens. Chai is now going to vibe code an AI agent for you, and it's going to build it on top of AI primitives instead of an AI framework.

5. 0:40

   And as that is happening, it just takes like a minute. How about I tell you a little bit more about who I am and what are we talking about today, right?

6. 0:51

   So, uh, look at Perplexity, Cursor, v0, uh, Lovable, Bolt, and even now Chai. You will see one common, uh, theme across all these production-ready agents that are, you know, building millions of agents, millions of gener-- you know, millions of people are using them.

7. 1:09

   And this is one, this one really good trend that you can pick up, that all these AI agents in production are actually not built on top of any AI frameworks because, well, frameworks do not really add that much value.

8. 1:23

   They are bloated, they move super slowly, and they, they're filled with these abstraction that nobody really needs. Instead, you should be building on top of AI primitives. This is what my entire talk, uh, is about today.

9. 1:37

   I am Ahmad Awais. I've been around the block for quite a while. If you use WordPress, Next.js, Node.js, React, you're probably using my code because I've contributed to all of these software.

10. 1:48

    I have also built, uh, hundreds of open source packages, mostly automation CLIs with Node.js and created a Shades of Purple code theme, all of which are downloaded like forty, fifty million times, uh, you know, a year.

11. 2:03

    And I've gone up to, you know, as, uh, as technical as you can imagine. I've contributed to NASA Ingenuity helicopter mission. And in Apastro, I've been a VP of developer tools, VP of engineering, Google Developers advisory board member.

12. 2:16

    All I'm trying to say is I'm deeply technical. I've gone through this phase of working with and building frameworks. And now, why do you think I am talking about primitives?

13. 2:27

    I think primitives have this native ability of working really, really well in production. Like Amazon S3 is a, a really good, uh, example here. Amazon S3 is a primitive where you can upload data and download data, and they scale it, [chuckles] uh, massively.

14. 2:43

    They're not building a framework for object storage. It's a very simple thing. It's a low-level primitive you can use to build lots of things, right? And that is what we are, uh, talking about here today.

15. 2:54

    My journey in LLMs actually started with... in 2020 when Greg Brockman himself gave me access to GPT-3. GPT-3 was just like what? Uh, maybe a month-old model. And I had, I had already started building, uh, you know, something like GitHub Copilot, which, uh, launched in 2021, a year later, right?

16. 3:13

    Even now, you know, we've been building it, uh, things and agents for like, I don't know, five years. Building and deploying and scaling AI agent remains to be the biggest pain there is.

17. 3:25

    And I think everybody has a different definition of AI agents, but this is my take at it. I think AI agents are just a new way of writing code.

18. 3:34

    Everything we know of how we used to build code, how you used to build coding projects, uh, or SaaS, all of that is changing because of AI and because of agents, and it's, it's just big enough.

19. 3:46

    It's not... It's, it's just a new way to write code. It's big enough. You know, if you try to put it inside of a framework, that abstraction might not be enough.

20. 3:55

    Instead, how about we build small building blocks that are useful across the stack, like something like threads. You know, every agent needs to store some, uh, sort of context or history of conversation.

21. 4:07

    So threads would be an awesome primitive, and we build that, right? So why not build that? Why I think that is important is because here's my belief. I think most of engineers are going to become AI engineers.

22. 4:20

    This, you can already see with full stack AI engineers, web developers and front-end developers. They're quickly transitioning into this AI engineering role because they are shipping a lot of product with AI, and they are building stuff with, uh, different LLMs or whatnot.

23. 4:35

    They... With vector stores and this and that. Even DevOps engineers and ML engineers are now shipping product. So when everyone is building as an AI engineer, what we are trying to do, uh, here at Langbase is improve the experience.

24. 4:49

    We want to become the fastest possible way for you to build a production-ready AI agent.

25. 4:56

    A lot of people, I think, start with this painful way of building AI agents, where they pick up a framework which is filled with, uh, you know, obscure abstractions, which are really, really hard to debug, and then they have to figure out how to deploy

26. 5:10

    and scale those agents. We think the other way. I think if you are building on top of predefined, really good, highly scalable AI primitives, especially composable primitives that come with a piece of cloud in it, like, uh, you know, memory.

27. 5:25

    Memory is an AI primitive, which is like human-like memory, which has a vector store in it. You can throw in, I don't know, terabytes of data inside of memory, and it will auto-magically scale.

28. 5:36

    So if you build an AI agent with that memory or with that parsing, chunking or threads or tools infrastructure, what you get is a serverless AI agent that automatically can do the heavy lifting for you.

29. 5:50

    But what does that actually look like? Let's take a look at what is happening here. Today, in this talk, I'm going to share... And by the way, like, uh, I'm gonna probably, you know, it's already done.

30. 6:01

    I'm gonna deploy this before, you know, w-we move forward. I am going to go over eight different AI agent architectures that are built purely with AI primitives instead of a framework.

31. 6:13

    Look like, look at this, you know, I said chat with PDF, and it figured out that, well, you need a memory for storing this PDF, which will have a vector store, and you need, uh, an AI agent, an LLM, uh, which will be, uh, used to ask questions from this PDF, right?

32. 6:30

    So it went ahead and did this, you know. We need-- It created a memory, created, uh, a way for you to generate answers, an agent on top of that memory, right?

33. 6:40

    This, uh, the flow of this is very simple. I want to retrieve a P-- some PDF content and then generate the answer, right? And the, you know, it's already done.

34. 6:50

    As you can see these lines, the first step is to build that memory, and we are using a primitive here, langbase.memories. This primitive is where we will put the user, uh, input.

35. 7:02

    The, you know, the question that user is trying to ask is the name of that memory, the PDF, uh, document memory. Uh, if you go here, you will find that memory.

36. 7:12

    And this, all of this can be built, uh, with API as well. What I'm gonna do here is I'm gonna go to, I don't know, uh, amawais.com/about and amawais.com/talks.

37. 7:25

    Uh, this page has, uh, information about me. This one has information about my talks, like the one I'm giving right now, and I'm gonna throw all of these as

38. 7:37

    PDF files in it. So I have already saved them on my desktop, so I'm gonna just quickly grab them

39. 7:44

    and upload here. I've also... I'm gonna also gonna do this API key Langbase.

40. 7:53

    So I'm also gonna throw a PDF version of this particular file into the memory. As you can see, uh, this is the doc about Langbase, uh, how to get API keys from Langbase, uh, my talks and my About page.

41. 8:09

    All of these files are right now being processed, which means, uh, a parser primitive is converting these files, uh, from PDF to text, and then a chunker primitive is going to chunk these into small pieces of context, which are going to be used in a similarity search.

42. 8:27

    If I refresh these, you will see that all of these files are now ready, right? Now,

43. 8:34

    now let's go to this agent code and see what is there, right? So we are going to ask a question from this memory, right? Which will return back some memories, and these memories are in the second step, as you can see here.

44. 8:49

    This is what we are going to use as context, right? This is generate answer step, and in this step, we are basically going to create, uh, an AI agent that is going to use that context to answer the questions we have, right?

45. 9:05

    Very simple. Uh, it has also, uh, Chai has also, you know, built an agent app for you, which makes it really easy to use, but you can also go ahead and use, you know, uh, an API with whatever programming language you prefer.

46. 9:19

    Let's try the agent app. Let's ask it something, some, uh, very simple. So who's the founder and his last three talks? There you go. Right now, this information is in two different files in our memory, which was parsed, chunked, and embedded automatically using those primitives.

47. 9:43

    And you can see the answer is already here. I'm the founder. Here are, you know, the last three talks I did. Uh, and how do I get an API key?

48. 9:54

    This should probably also give me, uh, an answer on how to get that API key from that other file that we had, um, added in there.

49. 10:04

    There you go. How, how you get an API key. There's a bug. You can actually go to the app mode and vibe code [chuckles] uh, a fix for this bug, and this app, uh, will be fixed.

50. 10:14

    You can also, you know, make it public or whatnot. What's happening behind the scene here is most interesting, you know. Uh, I think every agent needs all these primitives that we are building, right?

51. 10:27

    Primitives are like, you know, memory, which is an autonomous RAG engine, a workflow engine that is built, purpose-built for multi-steps agent, threads where you store and manage this context and conversation, a parser to extract the context, and a chunker to split this, right?

52. 10:45

    And using all these primitives, you can build almost any AI agent. We actually did a lot of research on stateofaiagents.com, where you can read a lot about how people are building, uh, agents, what type of primitives they are using, and what type of, uh, LLM is required and doing really well in which type of industry, right?

53. 11:09

    Let's now go over different AI primitives and different agent architectures that use different AI primitives. So the first most common architecture that we see is an augmented LLM. Augmented LLM is basically an agent.

54. 11:24

    It's an agent that is going to get an, uh, you know, some input, and it's going to generate some output, which will have an LLM. It will have some way to automatically call tools, so it can connect to MCPs or call different APIs.

55. 11:39

    It will have access to threads. Threads is another AI primitive which will-- which is where you can store the conversation users are having with this agent or the context of this agent, uh, you know, which might be, you know, completely, completely asynchronous, right?

56. 11:56

    A thread, uh, of context, a scratchpad. Like for example, when you're booking a flight, there's a certain set of information that you kind of keep in your head or on a scratchpad.

57. 12:06

    Like, I'm gonna land here, I'm gonna go do this and that, and that is useful when you are, you know, booking that particular flight. So you can use thread to store that, uh, you know, memory-like context.

58. 12:18

    And then there's memory, the long-term memory of, you know, different events or whatnot. This could be terabytes of data that you want to search when you're using this agent, right?

59. 12:29

    Uh, which obviously requires more context and requires, um, you know, vector store. So in this example, in this augmented LLM, we see tools as primitives, threads as a primitive, and memory as a primitive.

60. 12:43

    And you can basically build almost any type of AI agent with this augmented LLM architecture. As you can see here, Langbase pipes or agents provide you th-with that primitive.

61. 12:56

    Now let's, uh, look at what type of agents and what type of architectures can you actually build, uh, using that augmented LLM that I just showed you. Here's another one.

62. 13:07

    So prompt chaining and composition, where you use multiple agents, as you can see the purple blocks here, are working together. You get an input, an agent creates an output, and based on that output, you basically decide if you wanna go forward, right?

63. 13:22

    Maybe you got an email, uh, and you figured out if that was spam or not, and if it was not spam, then you use another agent to write a draft email in response.

64. 13:33

    And you can, uh, take a look at, you know, what's happening here. There's a summary agent, there's a features agent, there's a marketing copy agent. This code is plain JavaScript or TypeScript, if you will, right?

65. 13:44

    You can see here, um, a summary agent, uh, this is a feature agent, and this is a marketing copy agent. They're all going to work together to generate an output that you need based on an input.

66. 13:57

    A more interesting one is, uh, something like agent router, right? Where an agent, uh, or an LLM router that you build basically decides which other agent is needed to be called next, right?

67. 14:12

    Let's see what this is-- what this architecture looks like in production.

68. 14:18

    None of this, uh, we are going to build all of this with AI primitives. None of this is built with a framework. We are going to basically create three specialized agents for different tasks.

69. 14:29

    One is a summary agent for summarizing text, the other one is a reasoning agent for analyzing and explanations, and then a coding agent for obvious reasons, you know, when you need to write code.

70. 14:41

    And all these are built for different LLMs. So summary is from Gemini, uh, reasoning is with DeepSeek, uh, Llama 70B, and coding is with Claude Sonnet, right? Now let's look at this code.

71. 14:53

    This code right here is only going to use AI primitives, and you will build your own, uh, AI framework on top of it instead of using a bloated AI framework, right?

72. 15:05

    So as you can see, here's a routing agent. Router is being told what the job is. It has access to all these agents, and it is supposed to respond back with valid JSON in picking which of these agents is going to do the job for us, right?

73. 15:21

    And here is the documentation of a very simple set of agents. So summary agent, uh, uh, summary agent, a reasoning agent, uh, right here, and a coding agent, right?

74. 15:32

    Now, all of these are going to run together. All of this, as you can see, is just plain old code. There's nothing new here. You can easily write this.

75. 15:40

    There's, there's no magic, there's nothing to learn. It's basically you're just calling these agents together, and one of these agents are going to respond back with which next agent you need to run, uh, you know, for the task.

76. 15:55

    So the task is very simple. Uh, why days are shorter in winter? Now let's go ahead and run this code.

77. 16:06

    So the main agent, the router decisioning agent, has decided that it needs to run the reasoning, or you need to run the reasoning agent, uh, to-- for this particular task, right? [chuckles]

78. 16:19

    Obviously, you're not writing code here, you're not, uh, writing summary. You can see how the, the scene maker made the right choice. And then you pick up the same, uh, uh, reasoning agent that he had built based on this answer, and then throw that input in there.

79. 16:35

    And here's the, you know, answer of why, answer of why days are actually shorter in winter. Don't get distracted by this answer, [chuckles] by the way, right? So another very common architecture for building agents we see is running, uh, agents in parallel.

80. 16:52

    This is absolutely simple to use. Uh, there's no abstraction needed. For JavaScript, uh, in JavaScript, you can basically build a set of agents. As you can see here, here's a sentiment summary in the scene maker agent, and you can promise.all all of these agents, and they will run in parallel.

81. 17:10

    My favorite, however, is this agent orchestrator worker, where an agent basically decides to orchestrate and build N number of worker agents that are going to solve a problem, which is then synthesized by another agent, right?

82. 17:27

    This is exactly what, uh, a deep research agent architecture looks like. So this one is really good. What we are going to do in this is we are going to create an orchestrator agent that will plan and create subtasks for worker agents, and these worker agents are going to work on those subtasks, right?

83. 17:47

    Let's see how that look-- what that looks like. So this is the orchestrator, and the entire thing it is going to do is give us back this type of response with subtasks, which are going to be, uh, things that, you know, these worker agents are going to do.

84. 18:03

    It's a very simple worker agent. It's going to get a subtask, and it will complete it, right? So let's look at the m-main input here. The input is write a blog post on benefits of remote work.

85. 18:14

    I care about productivity, work-life balance, and environmental impact. Let's see what happens here. So the orchestrator here is going to generate a bunch of subtasks. As you can see, that is what it has done.

86. 18:27

    So write the introduction, write a section on productivity, on work-life balance, on environmental impact, and then write a conclusion. So as you can see here, there are like one, two, three, four, and five subtasks.

87. 18:43

    So this is going to take five workers, worker agents to complete. As you can see here, this is the first worker agent, second,

88. 18:55

    third, fourth, and here's the fifth one that is writing the conclusion, right? And finally, we are going to synthesize all of this into one thing.

89. 19:07

    Look how simple this is. This is like what, like 90 lines of code, and there's no framework here. You're basically building very simple,

90. 19:17

    you know, agents, worker agents in an orchestrator agent, and you have promise.all running all of them,

91. 19:25

    and the data is just flowing through that, right? There's no need to build a complicated abstraction layer on top of this because maybe, you know, in a couple of months, you'll see that agent-- this agentic workflow will get even better with LLMs understanding how to run these worker agents.

92. 19:43

    If you are using any AI framework, an abstraction that you are hel-- you know, kind of stuck with, when these agents and when these LLMs become much more better at these agentic workflows, you'll have a very hard time migrating off of it.

93. 19:57

    So it's much better to build on top of primitives instead of building on top of a pre-built abstraction, right?

94. 20:05

    We're probably also gonna take a look at this evaluator optimizer where an LLM is used, an agent is used to generate a response, generate something, and this something, this is probably, I don't know, let's say a marketing copy.

95. 20:22

    This is going to be evaluated by an LLM as a judge, right? Which will either accept it or reject it with feedback. I'm just pr-- quickly gonna go and run this.

96. 20:32

    You know, here's a generator which is generating,

97. 20:36

    you know, some product description. Here's an evaluator which is going to either say accepted or provide with feedback based on the prompt you write here, right? And what we are doing is we are writing an eco-friendly-- description for an eco-friendly water bottle for conscious millennials.

98. 20:53

    So, the first agent takes a stab at it, and the evaluator actually says no [chuckles]. You are basically missing, you know, the point with this particular type of audience, the eco-conscious millennials, right?

99. 21:08

    And provides very specific feedback on what to do and how to improve this. And this evaluator should be built by probably the best possible LLM you can think of for your space of what you are trying to evaluate.

100. 21:22

     If it is related to health, you're probably calling a memory and, you know, you're doing all that you need to do to make sure this evaluation is really good, right?

101. 21:32

     And the second iteration, as you can see, is very well--

102. 21:38

     is generally well done based on this particular feedback.

103. 21:42

     Finally, obviously you can call tools with this. We can skip that. I think you can probably easily understand that. The most interesting one is memory. Memory is when you upload data, you create a memory agent, you upload the data, and then you retrieve that data and ask questions, which is exactly what we saw here in this code

104. 22:02

     where we created a memory. Let me go to the agent code where we created a memory, uploaded the data manually, which you can also do through an API as you can see here.

105. 22:14

     And then an agent was used to answer questions related to that data. This is also a very common pattern for building agents.

106. 22:24

     Now that you know all of these AI primitive patterns, you can build pretty much 80% of the [chuckles] most complicated AI agents out there. And I've done just that. Like, I've basically asked Chai to build me a deep researcher like Perplexity, where it actually went ahead to build things to analyze the query, do a web search, consolidate the

107. 22:46

     results, and then create a response. It is using Alexa here, and all the code is here. I can actually just probably go ahead and ask it something. I don't know.

108. 22:57

     What's the latest with OpenAI? Something like this, and it's probably gonna go ahead, send this basic query, and start doing that deep research. I've also built things like, for example...

109. 23:11

     Oh, let me go back. I've also built things like, I wanted a receipt checker which would do OCR, and since Chai or Langbase didn't have an OCR primitive, I found one from Mistral.

110. 23:23

     So I asked it to use Mistral OCR and gave it an example of how to use it, and that is what it exactly did here. So it's processing the image with OCR and using GPT 4.1 right now, extracting from the user input whatever is needed, and also using Mistral OCR latest model on top of it to do

111. 23:43

     a double fallback and really improve the OCR, based on top of the image we are going to send it, right? Let's see. You know, I have an example here.

112. 23:54

     I think this is Palo Alto $15 delivery or something. Yeah, there you go. Total paid 15 bucks in City of Palo Alto. It's a parking ticket, I guess. It's analyzing the result right now.

113. 24:10

     And there you go. City of Palo Alto and 15 bucks, right?

114. 24:15

     Similarly, I built an agent where I was like, let me add an image URL, and let me chat with that image, right? So I just added this image URL.

115. 24:26

     What is the expression

116. 24:31

     The person in this image. Okay, let's give it a go and see, you know, what happens.

117. 24:40

     There you go. Uh, eyebrow is raised, uh, quite skeptical and curious, which is what I generally look like in real life, right? So, uh, and the code for this is pretty easy.

118. 24:51

     Analyze the image using GPT-4o, uh, a vision-capable model. Uh, takes the image here, uh, takes the image URL here from here, passes the input, and that is pretty much it, right? [chuckles]

119. 25:04

     It's a pretty simple flow as if you ask me. So, well, this, this is pretty much it. And we've seen like, you know, it's May 30th, and you can see almost every other minute there is a new agent being built.

120. 25:17

     I saw somebody building a dental cost advisor, uh, a business swamp pro, uh, I don't know, a task refiner. All sorts of amazing, uh, agents are being built with Chai, and Chai is building all these on top of AI primitives instead of building it on top of a framework pr...

121. 25:37

     uh, that would have some abstraction that you probably don't even need when building agents, right? So the idea is very simple. All the production agents that we know of, when they are not built on top of frameworks, what good a framework would do to you in a fast-moving space when every other week there is a new paradigm,

122. 25:59

     a new LLM, a new problem being solved already that used to take a lot? Uh, why not build your AI agents on top of really good AI primitives? Uh, and you can either build those AI primitives, or you can use the prebuilt AI primitives that we built or some of our, you know, friends, uh, are building over

123. 26:20

     at different, uh, companies. Uh, and also, uh, if you really enjoy, uh, you know, vibe coding, give Chai, uh, you know, uh, a try. It will try to use the primitives it knows, and it will also try to u-- make it really, really easy for you to quickly build an agent, deploy it, and use it right away

124. 26:44

     instead of building a silly demo that may or may not scale, right? Uh, I am Ahmad Awais, and I am always hanging out on Twitter. I would love, uh, your feedback.

125. 26:54

     I would love to know, you know, what you folks think and what you prompt, ship and ship with our AI primitives or not. Take care. Ciao. Use your code for good.

126. 27:04

     Peace.