Build Your Own Deep Research Agent + Technical Writer

Talk transcript

1. 0:00

   [upbeat music] Hello, everyone.

2. 0:16

   Not too loud, I hope. Uh, yeah, I hope it's fine. Perfect. Uh, so yeah, we will start to kick off the workshop, and we will introduce ourselves shortly. But first, uh, we just wanted to present this slide because that's basically LinkedIn, uh, this year.

3. 0:32

   It's, uh, the type of content that when you ask ChatGPT, that's basically what you get, a very generic response. And there, there are a few things wrong with this

4. 0:43

   actual response or that we don't really like seeing. The obvious ones are the, the slop words, the AI slop, so the-- all the delve intricacies that we all know about.

5. 0:52

   But there are more, uh... And fortunately, there are, there aren't even some em dashes in there.

6. 0:57

   Mm-hmm.

7. 0:57

   But there are some other problems like the...

8. 1:00

   Wow. I will see that.

9. 1:03

   All right. The lines are a bit off, but the, the most that is more general, like most companies miss or most, uh, people miss, it does that often. These are some examples from LinkedIn posts where they all say, "Most teams, most, uh, people," or here, "most AI projects."

10. 1:23

    And I'm actually guilty of that one. So, uh- [chuckles] -we learn as we do. And there are some other problems with hallucinations or here, outdated information. Obviously, this was generated like last week, and GPT-4 is not state-of-the-art.

11. 1:39

    And likewise, um, there are some AI phrases slop with-- that, that we see a lot. The, uh, the obvious rapidly evolving one, but also, uh, the classic, "It's not about something, but something else," that it generates a lot.

12. 1:54

    And the worst of all is that, uh, typically this is really meaningless and shallow. It doesn't provide any value or anything useful. And so what we need here is a proper research and proper writing around these language models.

13. 2:11

    And to give more context of this workshop, uh, we basically at Towards AI create courses and tons of videos, training, and technical content. And to do that, we need to create technical content around AI engineering.

14. 2:25

    So we need technical writers, which means we need AI engineers, writers, editors, uh, that iterate together and a lot of time to create a good, uh, storytelling, a good...

15. 2:37

    Just a good article, good lesson, or good video in general, which in the end costs a lot of money. So what we try to do is to automate this process, which, uh, we did, or at least a part of it, because a-another part is really a, a human thing to make a good story or relate to the

16. 2:56

    others. And so that's what we did. We built a system for replacing this, this whole process of doing a very, uh, thorough research and then technical writing. So what it does is we give a topic like, "What is harness engineering?"

17. 3:12

    And we have our own deep research agent that will search various websites and tools, uh, use tools to do lots of things. And then another system to write an actic- uh, an actual technical article with code images and ideally a non-slop writing in there.

18. 3:30

    And actually, the system, since we build courses, we used it to build a course to teach it to build this system. So it was a really fun project to do in general, and it allowed us to iterate and test a lot and to have, uh, user feedback directly with our students.

19. 3:47

    So it was, uh, it's really nice. And that's what we try to share here in this two-hour workshop, or at least a much more compact version, uh, with a deep rese- a, a smaller, simpler deep research system and a writing agent strictly for shorter content for a LinkedIn type of post.

20. 4:07

    And also, we try to share what we learned building it. And, uh, you can open the... or, or get the GitHub repository. Uh, it's public. Uh, in the first thirty minutes, I will just talk with slides, so you won't be needing it.

21. 4:20

    But after that, uh, my colleagues will jump in with the code, so it might be worth to, to have it. And we will show this, this QR code again later on, so you can get that on time.

22. 4:33

    But basically, I will cover what we learned and some terminologies because we are education-- a, an educational company, and that's what we like to do. And so I will cover like some basics to bring everyone to the same level or at least to our definitions of the-- of things and share what we learned by building it.

23. 4:51

    And then, uh, Samridhi will take over to, uh, talk about the actual deep research agent and Paul with the rest. So who are we? Uh, on my end, I'm the CTO and co-founder of Towards AI.

24. 5:03

    I'm-- My name is Louis-François, and I was a PhD student in AI until, uh, ChatGPT came out, where I basically switched to being an educator full-time and creating Towards AI to, to focus around that.

25. 5:17

    I've been doing videos to explain research paper for seven years now, and, uh, obviously right now I'm more around AI engineering. And I've been, uh, developing in the space since 2020.

26. 5:29

    At a, a first startup I was hired at. And so I will leave Samridhi to, to introduce herself.

27. 5:35

    Hi, everyone. Um, um, I'm Samridhi. I am a machine learning engineer. I'm also a technical writer and a consultant helping Towards AI grow. So nice to meet you all.

28. 5:47

    Yeah.

29. 5:47

    Hello, everyone. I'm Paul Iusztin, and I'm building a software for eight years. And I've been into the teaching AI space for four-plus years. And I'm also the author of the LLM Engineer's Handbook bestseller, and I'm excited to show this, uh, workshop to you guys today.

30. 6:09

    So I'll start with, um, introducing-- We always start with a problem, and here it's, it's pretty general. I start introducing the AI engineering problem space, where basically it's just that all our decisions as AI engineers are governed by constraints that typically are, are, are applied less to software engineers, which is like the cost per task with-which can

31. 6:33

    swing a lot based on the model you use, the architecture you use. We have latency requirements, uh, with reasoning models and other things that we can control. We obviously have some quality to respect and data privacy in this case that we need to be careful of.

32. 6:49

    Fortunately, we have a stack to be-- to, to help us, uh, whether it is from prompt engineering, context engineering, to using tools, orchestration, or building our own evaluation systems.

33. 7:01

    And we like to see that as some sort of autonomy slider that goes from very simple, just prompting a model, to building agentic systems, where the more you add complexity from prompting, from prompting to more advanced workflows to agentic systems, the more autonomy you add, but also the less control you have over your whole system, and obviously

34. 7:25

    the higher the cost. And to us, that's very important to choose, uh, when we build for-- We also build for clients, so that's why we try to teach what we build in all th-this loop.

35. 7:36

    But we, uh, to us, it's really important to build the right system where we don't need, uh, too, too much autonomy where it can add problems and uncertainty. And obviously, most people are interested into building agents, but most of these agents that our clients want are actually, uh, s-somewhat super simple workflows or at least workflows that we

36. 7:59

    can come up with pretty easily. And so I want to try to, to start by defining workflows up to multi-agent systems, finishing with this deep research example that we have, uh, that we have made.

37. 8:13

    So what is a workflow? A workflow is very simple. It's just a language model but augmented. Mostly because language models just take in tokens and text and generate tokens and text, so you cannot do much with text, uh, other than reading and writing emails, which a lot of us, some of us do more than that.

38. 8:32

    So ideally, you want to add stuff to it, like adding data that it may not have access to, whether it's at your company or some other data that needs, uh, to, to best answer the user.

39. 8:46

    You can add, uh, you can give it access to tools to do other things than just generating tokens, and you can add a memory so that it can remember the interaction and be much more useful.

40. 8:57

    But still, this is not an agent. It's just a thing we can build easily that can be reliable. And a workflow can be even more complex than that. You can chain prompts together to reduce the latency and complexity to do many more things based on strict conditions.

41. 9:14

    You can even do that with a router to decide automatically based on conditions on what you want to do, whether it is to use a smaller model or, or just do a different task and then come back.

42. 9:28

    You can do that in parallel, uh, to be more efficient or to improve the results with, uh, majority voting and other techniques. And you can even add loops to automatically improve the results based on a judge feedback.

43. 9:43

    But all this is still not agentic. It's just a workflow that we can build and add all these things. We can even combine them. We can build really advanced workflows.

44. 9:54

    So when, when does that thing become an agent? And here again, I'm using a, an, uh, an illustration from Anthropic. It's, uh, that-- It's a definition we really like.

45. 10:03

    It's just when it needs to take actions and, more importantly, when it can react to what happens in the environment. So very simply, it's, uh, it, it should be able to take autonomous action and reacts to, to what happens, as I said, and be able ideally to plan, to decide which tool to use, and more importantly, which

46. 10:24

    tool not to use, how to respond, and everything. So when should we use this thing, an agentic system, compared to a workflow? Well, typically, we always want to use the simplest solution.

47. 10:38

    If just a prompt works, that's ideal. And when we build things, we always try to start with questions and use our sort of autonomy slider.

48. 10:49

    So just a list of simple questions we can answer would be if the model already knows enough, uh, about the task to be done, whether it is the user asking questions or some other task.

49. 11:01

    Obviously, you can just prompt it and do it and ideally add some examples. So, so few shared examples just because it's, it really helps the model to adapt, somewhat adapt on the fly, which would be just simple prompting.

50. 11:14

    Then if you need external context, if it's not too big and you can paste it in the prom-- in the prompt under somewhat around two hundred thousand tokens, you can just paste it right away and, uh, ideally have it beforehand, before your user ask it and use context caching so that it's more efficient, uh, to just, for

51. 11:35

    example, answer several questions about the same report or the same UR privacy documents or whatever internal documents you may have. Now, if the context is not known until, uh, the user asks a question to your model, this is where you may, uh, try to inject knowledge on the fly based on the question, uh, whether it is because

52. 11:56

    your data is private and you need to, to retrieve it to answer the question or if it's something recent that the models haven't been trained on or domain-specific, you might want to inject knowledge.

53. 12:09

    And if you want to do all this In different ways or depending on conditions, this is where you use a more advanced, quote-unquote, uh, work, uh, workflow with predetermined steps and sequences.

54. 12:23

    And so in practice, an example of a workflow we built for a client was a support ticket handling, where basically you have always the same steps in the same order.

55. 12:34

    Uh, the, the, the system receives the tickets, it class... a ticket, it classifies it, route it to the right team, it drafts a response, it can validate it against the policy of the team, and then send it.

56. 12:46

    And here, the key point is that these steps are always the same. The order never changes. It always needs to do, uh, this, these six steps in the same order.

57. 12:56

    And so building this as an agent would just add overhead without adding anything extra. You don't need to dynamically react on anything, right, uh, here. You just need to execute these steps.

58. 13:09

    So a workflow definitely makes sense in this case. So for an agent,

59. 13:15

    what you want to ask yourself is if your system needs to take actions or to be able to branch dynamically. So whether it is to call different API, uh, depending on what the user needs or write to the database or not, do different-- do reasoning and other things.

60. 13:31

    This is where an agent, uh, can be useful. And an example we had was, uh, a CRM platform in, uh, in, in Canada that wanted to create a chatbot for, for their, for their CRM platform, for their users, their clients, to be able to generate marketing content automatically.

61. 13:52

    And initially, uh, they reached out to us because they were applying to an AI grant, and they wanted a multi-agent system to do, uh, a lot of different things with agents for everything, which, uh, did sound really well for the, the, the grant because it was like AI-focused.

62. 14:11

    But it seemed really overkill to make many agents to do something quite simple as a, as a, a marketing chatbot. So in the end, what, uh, what we always do is to try to really understand what the client needs and wants, which is oftentimes very different.

63. 14:28

    And so we, uh, by talking with them, we figured that the actual workflow was always somewhat the same, always simple. We need-- The agent needed to make a plan to decide what it should do, then retrieve the data specific to the u- the client, generate the content, validate it, and fix it if needed.

64. 14:51

    And so the tasks were always sequential, the same, the same few tasks that we wanted to do. They were super, uh, coupled, always, uh, the same thing about generating marketing content, but just for different clients and for different formats.

65. 15:06

    And the whole sequence is context-dependent, so the-- whether it is the plan, the, the retrieval or generation or fix, it all needs to have the content in mind to be able to generate the, the, the best content as possible.

66. 15:20

    So splitting decisions across different agents as the client wanted would have been, uh, would have caused a lot of issues, whether it is information or just hand-off errors with tool calls and other things that would add, um, that would reduce reliability.

67. 15:35

    So instead, we just built one agent but used tools our, as our capabilities. And this is really great because we, with tools, you can just have, uh, they can have their own system prompt.

68. 15:48

    They can have their own validation logic, even their own LLM or own, own LLM calls. So we can do tons of things with tools. And, uh, for, from our example, we have, we had like a validation tool, one tool specific to all format, the SMS tool or email tool.

69. 16:05

    And the results is that we use tools as specialists, but the global context stays within our only agent, the decision maker and the planner.

70. 16:15

    But we- if we do that, it means that everything stays within our agent. So we have a constraint, which is the context window of our agent, of the model that we are using, which is con-- uh, which is, uh, made of many things.

71. 16:30

    The system prompt, obviously, the, the instructions that we give it, the tool definitions, the different schemas and how to use these tools. Few-shot examples of, uh, the, the task to be done.

72. 16:42

    Uh, it's really great, obviously, to teach by examples. As we've seen with models, they, they learn very quickly, uh, that way for most tasks. You can have retrieve data depending on what you want to do, and even entire conversation history, if it's a chatbot or whatever type of agent that evolves over time.

73. 17:02

    So all this can take up, take, can take up a lot of space,

74. 17:07

    which causes a, a problem because as we go through each of these steps, the context grows and the performance degrades, which, uh, we call context rot. And the problem is that this happens much before the actual context, uh, window limit of like one million token, for example.

75. 17:26

    It worsens quite fast after like two hundred thousand, or it keeps changing, but around two hundred thousand now. And, uh, this is mostly because the lost, the lost in the middle problem, where we basically teach the, these long context models to be able to handle long context by feeding them a large corpus, inserting a random fact in

76. 17:47

    them, and retrieving that fact. So it's basically more retrieval of one specific thing, but it doesn't teach those models to leverage the whole book, the whole context to answer the question.

77. 17:58

    So it's definitely not ideal, but it would be too expensive to build this kind of large dataset. So this, this type, this way of training them caused the, the problem of,

78. 18:08

    uh, of us having to manage this context budget, which means to always keep the context as lean and relevant as possible, both to reduce cost in terms of token amount But also to, uh, improve the, the metrics that we are using.

79. 18:25

    And we can use many techniques for that. We can train the content, we can summarize it, we can retrieve based on, uh, criterias. Or, uh, there are also many, uh, interesting techniques in the cloud code leak, uh, compaction methods that I recommend looking into.

80. 18:42

    Uh, really powerful ones. But the one I'm most interested in for this talk is the delegation that we can do to either tools or sub-agents with their own context, which is what most, um, harness that we use are doing.

81. 18:59

    And so this leads us to the multi-agent systems, where we basically want to use them when we have too much of this context, mostly like over twenty tools or, or just the context becomes way too large, we can delegate to tools or to agents.

82. 19:17

    And, uh, th-there can be other reasons, like if you, if you need to make autonomous decision-making, or you need tool variability, or just security compliance, if you ha- you need to have one agent locally in inside one hospital, for example, because I was in the, the healthcare field, uh, before, and we used to ha- to have everything

83. 19:38

    local. So that can be a, a reason for multi-agents. And why am I talking about all this? Because what we've seen is that AI products are never just you build an agent or you build a multi-agent crew AI thing.

84. 19:54

    They basically combine all of, all of that. They combine tools, workflows. S-some parts are more specific and predefined, some parts have more flexibility.

85. 20:05

    And, uh, we believe that AI engineers need to understand all these to build, to build complex systems. And deep research actually implement that. It's a very good example of a complete system since it's-- since they integrate all of these techniques.

86. 20:22

    And before we show how we built our own and allow you to use it as well, ju- let me just cover quickly what a deep research agent, uh, system is, just in case.

87. 20:33

    It's a reasoning system that will plan what it needs to research about. It has some autonomy to decide what to research about. It has tools, uh, internet or web access.

88. 20:43

    It's reliable. It will cite its sources. It has feedback loops, uh, with itself, but also with the human for human feedback.

89. 20:53

    So this is much more agentic, as we see. It ne- It, it evolves in an environment, which is typically the web or, uh, user-provided sources or just API access.

90. 21:03

    And it's goal-driven. It has one objective. We don't tell it how to do it exactly. We just tell it, "Do a research about something." So it's typically much more than just a chatbot.

91. 21:14

    It basically replaces someone that would do a very thorough research about a specific topic. It plans, it search, it inspects, pivots. It synthesize information and can iterate. So deep resource systems are really, uh, to us, one of the best projects to learn how to build such a complex end-to-end system.

92. 21:36

    So we decided to build our own. And in this case, our own is basically an, an end-to-end technical article production from a topic because we create lessons, course lessons in our case.

93. 21:49

    So it came out of a real utility, which is the, the most important part. So the goal here is to take a topic, research deeply about it, and write a very good technical article with code integration, with...

94. 22:04

    that is actually runnable, with, uh, images that is useful, not just randomly generated images throughout.

95. 22:12

    And, uh, there are a few challenges when doing this. First, you need a really high precision and recall when doing the research because you need as much relevant sources as possible, but you don't need too much of them because of the conte- the limited context problem.

96. 22:27

    You need to reduce hallucinations and AI slop, obviously, and incorporate a lot of human feedback, especially in writing, in the writing aspect.

97. 22:37

    And as we always do with projects, and I think, uh, you also do as well, we started this with asking ourselves questions to better understand how we should do it and if we should do it at all.

98. 22:48

    So the first question is: Is this worthwhile? Uh, is there a solution outside of, uh, of what we could build that already exists? So our analysis is that high-quality c- technical content is expensive to produce.

99. 23:02

    We do that every day and, and we need a lot of people, and it needs a lot of time and review, and it's just really time-consuming, really expensive, especially because these people need to be somewhat senior AI engineers to be able to explain anything in AI engineering.

100. 23:18

     So they need expertise, but to teach someone not to build a successful product. So it's expensive and not that lucrative. So, um, yeah, it would be ideal to automate most of this process.

101. 23:31

     Uh, as I said, the research needs strong recall and precision. And on the other side, the writing needs to be more constrained, needs to avoid hallucinations, be of high quality, follow your tone or the writing tone that you want, the structure that you want.

102. 23:46

     And in terms of deep resource tools, uh, we use all of them pretty much daily, and they are re- way too ex-ex-exhaustive. They, they gather way too much content, and they have a lot of noise.

103. 24:01

     It's, uh, not ideal for us in our case. We do use them, but we decided to build our own just to be, to, to be able to do exactly what we wanted.

104. 24:12

     So the decision is, yes, that this was worthwhile, but, uh, especially as a writer augmentation, not to replace them. Uh, because as we've seen, uh, even if you do as-- uh, if you build the best system possible these days-

105. 24:28

     It's really hard to create a piece of content, whether it is a video or a written lesson, that connects with the reader or the viewer. You really need a human touch to be-- so that it's relatable to someone.

106. 24:41

     You need to make jokes, good jokes, not the same ones all the time, and the ones that are, are appropriate to the, the context. So in-- It was, uh, it's really difficult to automate all this, and we want to keep a human in the loop, uh, there.

107. 24:58

     Um, now the, the first actual question we ask ourselves is, what should be the architecture? Uh, if it should be an agent, multiple agents, a workflow. And the analysis here is that first we have the research part that is really exploratory.

108. 25:13

     It needs to, um, find lots of sources. It needs to explore the web to understand what it needs to find if it misses information. So it needs to be able to iterate, to pivot, to search again.

109. 25:27

     It needs a lot of flexibility. And on the other side, the writing agent is much more deterministic. It needs to, to follow a specific tone, to follow a structure, to, uh, to, um,

110. 25:41

     to... Yeah. It needs to be much more constrained than it needs flexibility. You don't want this AI slop. You don't want specific words, specific sentences. You want specific sentences and specific formats, so it's much more constrained.

111. 25:54

     So we have a conflict here where the research needs flexibility, but the writing needs constraint. And so the decision is to split into two systems: the research agent that is more exploratory, dynamic, more agentic, and the writer agent, the writer system that is more deterministic and consistent.

112. 26:14

     And we have tons of these questions, and we show in our course exactly how to build these two systems. But here it's just two hours, so I will-- we focus on the research agent a bit more, especially for the next few questions.

113. 26:28

     So here, how should it be created? Well, how should the, the research agent communicate with the writer workflow, the writer step?

114. 26:36

     Well, we pivoted a few times after testing it, and that's the most important part, is to actually use your product or a potential user should use your product to give you feedback as early on as possible.

115. 26:47

     And on our end, we used it to build a course to use it, so it was really easy to iterate and to teach how we-- what we learned and what we did.

116. 26:57

     And what we saw is that our users, which were myself and the team,

117. 27:02

     s-saw that we always used either both agent or we didn't use a system. So we didn't-- we weren't just alternating between the research and then the writer. If we were to alternate, it was just with the writer at the end to add new suggestions, new topics, new, new, uh, information to add to the article or to remove

118. 27:22

     to it. But the research was already quite perfect and, and done beforehand, as you will see with, uh, Samridhi.

119. 27:30

     And if a big change was needed, we just would rerun the process again. So we didn't need to have proper orchestration in place.

120. 27:39

     And the two agents work on the same artifacts. The research agent produce a, a research.md file with all its research summarized and send it to the writer. So they need to have ways to communicate together.

121. 27:51

     So the decision that we made i-is to separate the two agents and run them sequentially without orchestration. So just a script, just a very basic script. And our, and our two agents are in the same project so that it's easier to maintain, but also they w-- they share dedicated files, so it's easy to have them communicate with

122. 28:14

     each other, but also have them communicate with us through these files. And having them together in the same project is useful for AI-assisted coding with Cloud Code or whatever system you are using.

123. 28:26

     The next question for the research agent is, uh, how should it behave? So we really need-- We really wanted to understand what would be the process that we would want out of this research assistant before building it.

124. 28:39

     And again, we refined this after testing. And in the end, we really needed a good human-written guideline, which would basically be just what is the topic about anything I'm interested in covering in this lesson and, uh, useful links that we think are relevant.

125. 28:58

     Uh, sometimes I send a AI news, newsletter on some specific topics. Sometimes it's my own videos or other topics that we covered.

126. 29:07

     And, uh, we saw that, uh, obviously there's a, like, creator bias here, but we saw that the more we define the agent's goal in the guideline, so the less freedom it had, the better the results were.

127. 29:19

     Then if we give links in that, uh, guideline we send to the agent, the agent needs to scrape web pages. They needs to be able to digest YouTube videos, GitHub repositories, even if they are private, uh, to the user, to myself,

128. 29:35

     to, uh, gather all the, the necessary context. And then after that, it needs to do its actual deep research thing. So search the web for anything that is missing and needs understanding, and then revise all this context and write that into a very nice final artifact that will be sent to the writer agent.

129. 29:59

     So how did we do all that? How did we scrape web content that we give the links to and do web searches? Well, we, as always, simplify the problem and don't reinvent the wheel.

130. 30:09

     We use existing things, ex-existing systems and APIs and divide the problem into simpler problems. So first, scraping solution, we use, uh, Fireclaw and agentic scraping. And this is for many reasons, uh, that, uh, well, Samridhi will talk a bit more about the research agent, so I won't cover it too much.

131. 30:29

     But It's really useful to have this kind of, of scraping for more, um, dynamic websites and other types of websites that can block scraping. And for web searches, we used, we used to use Perplexity, now Gemini with grounding for, uh, get, to get precise answers with sources directly, which we can then scrape if we need to.

132. 30:52

     But otherwise, we can already prompt it g- uh, well enough to get the information we want from this grounding queries.

133. 31:00

     Now, for YouTube videos, because there are a lot of great information on YouTube, we want to ideally pass just the text to the agent because the video is, is quite heavy.

134. 31:13

     And we tried, uh, a lot of thing, but typic- a lot of things, but typically you need to download the video and extract the transcript. But fortunately, Gemini now handles this directly with a YouTube URL.

135. 31:26

     You just give the YouTube URL and ask questions about it, or even ask to provide the, the transcript, so it's very useful. We decided to use that, obviously.

136. 31:36

     And, uh, now for the GitHub content, we just used the GitHub ingest library to get, uh, a GitHub-specific content into markdown, and we can provide it with a, a token to access our private li- liber- uh, private repositories.

137. 31:53

     So it's really simple, and you will, uh, see that in the, the repository.

138. 32:00

     For framework, uh, we used MCP with FaastMCP

139. 32:05

     so that, uh, we can have tools for the different processes, and we can have an MCP prompt to the-- to basically give a recipe on what to do and how to use the tools to, uh, the agent, which is, uh, very useful.

140. 32:20

     We will talk in depth about that in the, in the next, uh, hour and a half.

141. 32:26

     Then for the research, uh, the, the setup, we basically just used Python, UV for managing the dependencies, and GitHub obviously. And for the models, we, uh, recently changed for everything to be Gemini, mostly because, uh, of the YouTube thing that I mentioned.

142. 32:43

     But also we saw that first it works really well, and we can alternate between Pro and Flash, the, the cheaper one and, and more expensive, depending on the complexity of the task.

143. 32:53

     So we use them alternatively in, in the, the, the code. And, uh, the, the one thing that we really like is also that it has a free tier for our students, so it's, uh, really convenient for us.

144. 33:06

     But obviously we, we also compared to the other models. We were using Opus before and o- and other cloud models and even OpenAI's, but we've seen that Gemini is much more interesting, especially from OpenAI these days.

145. 33:20

     So, uh, we are using Gemini, but you need to compare with different, uh, models to be sure that you are using the right ones. And finally, for the more theory part, uh, the-- how the, does the research agent communicate with the user?

146. 33:38

     It's inside the MCP prompt. We have specific steps to stop and, uh, interact with the users. And all these interactions are handled through files, uh, whether it is the guideline or the research file or other files that Paul will talk about in the writer part.

147. 33:55

     And by the way, all-- Everything I mentioned and much more is in, uh, some sort of cheat sheet that we also linked in the README of the repository. So you can definitely access it, uh, for free.

148. 34:08

     Obviously, it's on GitHub. And, uh, now is the part for where we talk about the deep research, and you can... So you can scan the code if you haven't to get the repository,

149. 34:19

     and Samridhi will take over.

150. 34:21

     You wanna come down? Okay. Sorry, everyone. Just give me one minute.

151. 34:41

     Okay. Awesome. So I'll be talking about the deep research agent. And so what is a deep research agent? As Louis already explained, um, that, you know, it can search any topic by searching the web.

152. 34:58

     It can analyze YouTube videos. And finally, it'll give you a cited report. Um, how we've built it. We've used, um, MCP for this, which is an open standard for giving agents, um, you know, access to tools and data.

153. 35:13

     Um, the key idea, like the design choice here, is that the agent is the brain, um, that is doing all of the reasoning. As like Louis highlighted, that, uh, any gaps that are there in the research part is being done by the agent.

154. 35:27

     It thinks if, you know, there's anything missing in the research, if it needs to run multiple researches or it has to search something else, that is the part that is being done by the agent.

155. 35:36

     Whereas the server is, uh, the MCP is handling all of the capabilities. So it's gonna be exposing all of the tools. Um, it's gonna be ex- uh, you know, exposing the resources and the prompts that are there.

156. 35:48

     Um, we've used something called FaastMCP, which is a library that, you know, really, um, makes, um, creation of like agents very easy. So, you know, it hides all of the complexities, and you do not have to write any sort of protocols or any sort of communications between agents.

157. 36:04

     All of that is handled by FaastMCP. And we're using Claude Code as our agent harness. Um, so you know. But this code is independent of that. If you're more inclined towards using Cursor or GitHub Copilot, you can, you know, swap, um, Claude Code with any of the ca- um, agent harness that you like.

158. 36:22

     Um, so I'll, I'll be repetitively talking about, uh, you know, these three things. So, you know, an MCP server exposes tools, prompts, and resources. So what are tools? Tools are basically all of the actions an agent can take.

159. 36:37

     So, you know, like deep research or like, you know, going and doing Google search or like analyzing a video and giving us a transcript or compiling a report. So all of these action-oriented things are done by tools.

160. 36:50

     Whereas, uh, the second thing that the MCPs exposes is prompt, which is basically instruction. You know, when you're talking to ChatGPT, you give it instructions. So this, in case, is, uh, prompts that the agent can follow.

161. 37:02

     And, um, in our case, we have a detailed prompt that I'll show you in a minute with the code. And finally, the third thing is resources, which is the data the agent can read.

162. 37:12

     So, you know, this is like static data, which is the model names we're using, those sort of versions that we have. If we have any sort of feature flags, those are all part of the resources.

163. 37:21

     So these are something that the agent can read. There is no action being done here. Um, in terms of tools, we have three tools. So first is a deep research tool.

164. 37:31

     Um, the deep research tool basically does, you know, Google search and give us, uh, sources for everything that we have. So it provides us with answers for any sort of query we had, uh, we have, and also it provides us all of the sources of that.

165. 37:45

     So, um, this is... W- we're using Gemini API for this. Um, the second tool that we have is the analyze YouTube video tool. Um, for this as well, we're using the Gemini API.

166. 37:56

     So you can, um, take any YouTube URL and, you know, paste it, and it's gonna give you a transcript here. Um, and finally, we have the compile research tool.

167. 38:05

     So what the-- It g- basically compiles the output of the previous two tools. So, uh, we're gonna be talking more about, um, the .memory file and the details of that when I show you the code.

168. 38:17

     Um, so, um, I just talked about, um, you know, the, the parts of the tool, uh, the agent that we have, but, you know, this is the overall architecture.

169. 38:27

     Um, so we are using Claude Code as MCP client. Just to highlight here, um, Claude Code for us is both, um, you know, MCP client and an LLM. So, um, you know, it is like the brain which is being used.

170. 38:40

     Also, it is, um, MCP client which is talking to the MCP server that we have. Um, the first MCP server that we have has tools, resources, and prompts. Um, these are the details about e- uh, each of the tools that we have.

171. 38:54

     Um, so all of these, um, you know, tools, the, the first two tools write to a folder called .memory. And, uh, basically we wanted something, uh, we wanted to like preserve all of this because it is easier for logging.

172. 39:07

     It is easier for us to verify all of the results that we're getting. So, you know, we are writing down all of our results to a folder. And the third tool reads from this folder and creates a file for us.

173. 39:18

     So a research, a markdown file called research.md. Um, the first two tools do API calls, um, to Gemini 3. And, uh, finally, everything is like, um, you know, read and written to this particular folder.

174. 39:34

     Um, so now I'm gonna show you the code.

175. 39:40

     Okay. So this is how the, uh... Let me just zoom a bit.

176. 39:47

     Sorry. Okay, perfect. So, you know, this is how we have structured our code. Um, you know, you don't have to get intimidated by this, so we're gonna simplify it a lot.

177. 39:56

     So, you know, these are the main two folders, the research folder and the writing folder. Paul is gonna be talking about the writing part. I'll just go, uh, and talk about the research part.

178. 40:06

     Uh, so if you remember when I was ta-- when I started my presentation, the first thing I talked about was the MCP server. So, you know, this is the MCP server file that I have.

179. 40:17

     And then I, uh, so, you know, this is how you set up the MCP server. So we have, you know, some settings that we are loading up here. So these are basically the name of the models that we have.

180. 40:28

     I can show that to you in a minute. And, you know, this is how you set up the server. So you give it a name and the version, and then you register three things.

181. 40:37

     So, you know, this is something that you should remember. We're registering tools, resources, and prompts. Um, and then we're just, uh, setting up basic logging here. And we're using something called OPIC, uh, for observability, and Paul is gonna be talking, uh, more in detail about that.

182. 40:55

     And then we're finally creating the MCP server, you know. And now let's see how we have registered the tools. So if we go to this fold, um, this file called tools, um, the Python tools file, um, this is how we have registered, um, the tools.

183. 41:13

     So for registering, um, the tools with, um, you know, um, FastMCP... Sorry, for registering tools, you need three things. Um, the first thing is, of course, the name of the tool.

184. 41:24

     So here, uh, the name of the tool is deep research. The second thing is arguments. So we're passing it two arguments. The first one is the working directory, um, and the second is the question or the query that we're gonna be asking it.

185. 41:37

     Then you need, um, a definition for what your tool is gonna be, uh, doing. So, you know, you have to be as precise as possible when you're writing this definition.

186. 41:46

     And finally, we also have the definition for the arguments that we have. So, um, you know, um, so this is the way you write the tool. Then we just have, um, the code for the observability.

187. 41:57

     And then finally, we have the implementation. Um, we're returning the implementation for the tool. So this is it. So, you know, this is how you define a tool and, um, um, there's no complications here.

188. 42:08

     But let's go into a little bit more detail about the tool itself. So, um, how this tool is working. So for the deep research tool, we are basically in, in the first step, we're just validating all of our parts.

189. 42:23

     So, you know, we're checking if we are in the current directory or not. Um, and secondly, we are ensuring that we, the memory file that we're writing to, basically it's a folder where we're writing all our results to, if it exists or not.

190. 42:35

     And, you know, we do all of this validation here. And then, uh, you know, we run this grounded search on the query that we have. Um, once we get the result, uh, of this query, you know, we return the status.

191. 42:52

     If it's successful or not, what was the query that we sent? What is the answer that we get? So, you know, we get like a detailed response and, you know, this has, like, been very helpful for our team to, like, debug, um, and like, you know, go through, um, all of the data that we get so that,

192. 43:07

     you know, we can refine the research results we have. Going into more detail about how we are, um, you know, running the, the research, right? So in-- if you see this, uh, the run grounded search function that we have, we have a prompt that we've already written.

193. 43:25

     So if you go here, you can see the research prompt that we have. So we're basically saying that if you have any questions, you have to provide a detailed, comprehensive answer to the question that is there.

194. 43:35

     Uh, focus on only official authoritative references and, you know, make sure, um, you're including, um, all the relevant details as possible, and make sure to cite your sources clearly.

195. 43:46

     Um, so, you know, this is the prompt that we're using for the deep, um, research tool that we have. And then, uh, we're just calling the Gemini API here, and we're getting the, um, answer text and the sources.

196. 44:00

     Um, then, um, in this part of the code, we're just, um... So I mean, every time you get an, uh, um, sources from Gemini API, it's not in a structured form.

197. 44:10

     So here you're-- we're just like trying to structure it in a better way. I'll show that to you when I resu-- Sorry, when I run it, you'll see that, you know, we get the output in this structure where we get the URL, we get the title, and we get the snippets and, you know, finally we are returning

198. 44:24

     the research results. So, you know, this is the-- this is all the details about our first tool. Uh, but going on to our second tool, which is the YouTube video analysis.

199. 44:35

     So this has also got the same structure as, um, the, uh, the deep research tool that I showed you. So three things. The first is, um, the name of the tool.

200. 44:44

     Um, the second is, uh, the arguments that it is using. So even in this case, it is using the working directory and the YouTube URL, and then the details about the tool, um, you know, what it is doing, and the details about the arguments that we have.

201. 45:00

     And then we're just returning, um, the execution of the tool. So going into more details about what the analyze YouTube video tool looks like. Um, so here is the implementation for that.

202. 45:13

     Similar to what we did for the deep research tool, we're, like, validating all of the parts here.

203. 45:19

     Um, and then, you know, we're taking out the v-- uh, YouTube video ID using this get video ID, ID function. If you do not get the, you know, video ID for the YouTube URL that we have provided, uh, we get, um, you know, we, like, try to sanitize the result and get the video ID.

204. 45:36

     Um, and then, you know, we're calling... This is the line where we're calling the Gemini API and, you know, similar to the deep research tool, we're returning all of these different details so that, you know, we can keep track of the things.

205. 45:50

     We can see what is the output that we have got. And let's go into the analyze YouTube video, um, implementation.

206. 46:00

     So, um, you know, similar to the deep research tool, we have a prompt for the YouTube transcription, and here is the prompt that we've written. So, um, so, you know, we're just saying that you have to...

207. 46:12

     This is basically guiding, uh, the Gemini API to give us the transcript in a certain format and, uh, you know, give us all of the instruction and the details that we want in our output.

208. 46:23

     So, um, for this part, um, so here we are, uh, you know, dividing our requests into two parts. Um, the request that we're sending the Gemini API. In the first part, we are sending the file...

209. 46:36

     Sorry. In the first part, we are sending the URL that-- the YouTube URL that we have as a file URI, and in the second part, we are sending the prompt that I showed you.

210. 46:45

     And, you know, this is, like, a very cool thing. So, um, if you, um... So Gemini is a multimodal model, and it actually-- Once you send a YouTube URL as a file URI, it actually sees the video.

211. 47:00

     So it goes through part by part. It doesn't access any sort of transcript. And that's why when we'll be running the, you know, um, video, YouTube video analyzer tool, you'll see that it takes, like, two to three minutes because, you know, it is actually going through your entire video.

212. 47:14

     So it is, like, really multimodal. Um, I mean, in here we are, uh, you know, like, sending the, um, request to Gemini and once we get the output, we try to, like, clean the, um, output that we are getting in a certain format and then, you know, um, outputting it and, like, saving it in the .memory folder

213. 47:34

     that we get. Finally, coming back to, um, the final tool that we have, which is the compile research tool. So what it basically does here is, um, it returns and compiles all of the results that we get from the previous two tools.

214. 47:50

     If you go here and see the implementation for the compile research tool, you'll see that what it is trying to do is it is trying to write to the, um, output research MD file, and then it gives us, like, a success status or a re-- uh, or a return message.

215. 48:05

     Um, so, you know, we have seen, um, how we have registered the tool in the MCP server. Similarly, we've got the, you know, code for registering the resources. So, um, you know, here is all the, all the things that we are sharing with the, um, agent as resources.

216. 48:23

     So basically, what is the name of the server that we have, the version that we are using, what sort of Gemini model are we using, what is our YouTube transcription model?

217. 48:32

     And all of these different details are being provided, um, to the agent using resources and Then, um, the MCP prompt. So, you know, um, this is how we have registered the MCP prompt.

218. 48:46

     So, um, basically we're returning something called workflow instructions, and this is the detailed prompt that we have written. So basically, the-- in this prompt, we are telling the agent that these are the tools that are available to you, and, you know, you can use this workflow to use these tools and, you know, some additional details about like

219. 49:05

     the working directory and where all of the results should be stored. So, you know, this is our detailed prompt, um, that we're giving to the in, um, agent. Awesome.

220. 49:15

     So going back, um, you know, this is like the... Oh, sorry. One more thing, uh, I wanted to highlight here is that this is our env, uh, file. So, you know, this is all being accessed as resources, but this is where you'd have to add your Google API key.

221. 49:31

     You can add the Opec API key if you want any sort of observability. Um, but this is the, um, env file that we have. Additionally, uh, apart from that, you know, we have the scripts here, which are like, um, you know, helping us with the code throughout and, you know, these are like scripts, um, we have written

222. 49:49

     additionally, um, to manage, uh, some of the utils.

223. 49:54

     Um, so... Sorry, yeah. Going back, um, so now we have the entire architecture there. And so how do we exactly connect it with, um, our agent harness? In our case, this is Claude Code.

224. 50:08

     Um, so this is a very, um, you know, straightforward way. For Claude, you just need to create, um, an mcp.json file in the project root and, you know, give it a command like this.

225. 50:21

     Um, and then for our case, we are running the MCP, uh, locally. You know, we're not hosting it anywhere. It's being run locally, and it's using standard in, standard out.

226. 50:32

     So for communication, so, you know, this works. But in case, you know, your MCP is hosted elsewhere, you would need an URL like, you know... Um, Notion is a very popular example.

227. 50:42

     Notion has its own hosted MCP. You can a- have access to it by getting the URL. So this part of the code would be changed slightly. Instead of putting the uv command, you would be, um, you know...

228. 50:53

     Instead of putting the uv command, you would have that URL here. Um, and you can connect with the, you know, any MCP that is available, Snowflake, Notion, anything that you want.

229. 51:03

     Uh, one more thing that I would like to highlight here is that, um, as I said at the beginning of my talk, we're using Claude Code as an agent harness.

230. 51:10

     You can use anything that you want. You know, uh, if you like Copilot, you can use it for this. Um, you know, any, um, sort of agent harness you wanna use, you can use it with that, but this is, um, for Claude Code.

231. 51:22

     Okay. So let's see where that file is. So it's in the... So you have to create that mcp.json file in the root directory. And this is, um... I can explain it to you.

232. 51:31

     It's a very simple thing. So here we have, um, you know, the name of the MCP server we have. So, you know, it's deep research. And this is the command that we have.

233. 51:42

     So we are using uv because it simplifies everything. So, you know, any sort of, um, a virtual environment that you're creating, any sort of things, packages that you're downlin- downloading, it's all being handled by uv, so you don't have to do anything.

234. 51:55

     And, you know, then we have, um, this command, which is basically running the server file that I showed you couple of minutes ago. So it is just doing that.

235. 52:03

     So Claude runs the server file as a subprocess locally. So our server is running locally. And this, this, um, entire code is, um... Sorry, this entire command is only doing that.

236. 52:16

     It says run fast MCP, run and, uh, path to the server file that we have. And this is the environment file that I told you that you're supposed to create.

237. 52:26

     Okay. So, um, you know, now, um, it's time to see, um, how the code works.

238. 52:34

     Um, okay, cool. Um, so I would go to Claude.

239. 52:45

     Um. Uh, sorry. Okay. So if you go to Claude and you do MCP, uh, you can see that I have two servers here. So, you know, because of the mcp.json file that I have, um, you know, it was automatically able to, um, detect both the agents.

240. 53:03

     Uh, so we'll just be seeing the deep research agent right now. And, um, you know, this is all the details. You can see the command. You can see the arguments that we have.

241. 53:12

     You can see all of the capabilities that the server is exposing. So, you know, you can just do view tools and, you know, you can see all of the details about the tools.

242. 53:20

     So, you know, the tool name, the description that I showed you, all of the arguments, the parameters that we have. You know, similarly for all of the other tools, you can see all of the details here.

243. 53:31

     Um, now, uh, you know, I'm gonna try to, um...

244. 53:36

     Sorry. I'm gonna try to, you know, run one of the tools and see, um, how Claude does it. So I've already written out this question. This was, uh, from a previous AI talk about, um, AI agents.

245. 53:52

     Um, so I'm gonna just tell it to analyze this video, uh, for me, and it should automatically be able to detect, um, the tool that we have, uh, that is being exposed by the MCP server.

246. 54:06

     So you can see here that, uh, you know, it has automatically picked up the tool that we are exposing and, uh, and it is, um, you know, it is, uh, trying to use that.

247. 54:17

     And on the side, um, you know, you can see that there's a .memory folder that I told you, you know, where we are tracking all of our results. It is being created, and it's gonna write, um, the transcript to the...

248. 54:29

     Um, sorry, write the transcript to this, uh, file. So as I told you,

249. 54:35

     um, Gemini is watching or like analyzing this video in real time. Um, that's why it's taking-- it's gonna take like a couple of minutes, uh, for it to like produce the transcript.

250. 54:46

     Um, so Yeah. Uh, so basically every time you get this question, um, you know, Claude Code tries to see what it can do, what, um, tools are being available, and, you know, then it sends a request to the MCP server can...

251. 55:03

     that can you please execute this tool? And once, um, the MCP is, like, MCP executes the tool, it sends the results back to Claude Code, and then it further analyzes that result.

252. 55:16

     Um, so we can wait a minute for that.

253. 55:24

     Sorry. Yeah. So it has created this markdown file. So it says that this video is from, um, you know, the AI Engineer Code Summit. It has provided me all of the details and, um, you know, it, it is, like, pretty good detailed transcript that we can see here.

254. 55:45

     And, you know, it also gives, um, uh, you know, gives its input at the end. Plus, I also, like, really like this thing that, you know, we are storing all of the transcript, but it gives you, like, a really neat summary at the bottom where it says, you know, what were the key ideas that were shared in

255. 56:00

     the video and, uh, you know, so... And it says that, okay, the transcript is here. So, um, you know, this is, like, a short example of how we could, um, you know, run and see one of the tool if it's, uh, if Claude is able to s- um, run that or not.

256. 56:14

     And now, uh, I will try to, um, you know, run the entire pipeline again.

257. 56:22

     So, sorry. Uh. Okay. So I've already written a question for that.

258. 56:39

     So I'm telling it to do an end-to-end research and...

259. 56:54

     So, um, now I'm trying to use the deep research tool that I have and the YouTube video tool that I have, and trying to see, um, you know, how it runs both of them together.

260. 57:05

     So, um, it says that, um, it is trying to, um,

261. 57:10

     it is trying to do the analyze YouTube video part first.

262. 57:21

     So, um, when you have like a, you know, like a big question like this, the entire workflow, how it works is Claude breaks down the question into, like, different parts and then sees which tool it needs.

263. 57:33

     So, you know, it has like an entire reasoning going on that, okay, you know, if these are the sort of tools that I have available, I need, um, the YouTube, um, analyze YouTube video tool for analyzing the YouTube video.

264. 57:44

     I need the deep search tool, uh, de-deep research tool for the first part of the question. So it's gonna divide it into two parts and then try to call, like, both the tools.

265. 57:53

     Um, okay, let's see. Yeah. Sorry, it usually takes some time for it to...

266. 58:11

     Can I ask a question worth filling the gap?

267. 58:13

     Um, sure.

268. 58:14

     Um, uh, very clear with the design of the old MCPs. Have you guys thinks about the difference between MCP and skill and whether this helps any from actually-

269. 58:24

     Yeah, I'm actually gonna talk... That's, that's the next part. I was, uh, trying to remove the skill initially when I, uh, moved it to the temporary folder so that it doesn't pick up the skill, 'cause it, it tends to do that as well.

270. 58:35

     Oh. Uh, my, my question was more like in the whole MCP, the deep research-

271. 58:39

     Yeah.

272. 58:40

     ... have you thought about actually building the whole deep research as the skill rather than MCP?

273. 58:45

     Not about that. Just the prompt part though.

274. 58:48

     Okay. You haven't thought about actually-

275. 58:49

     No.

276. 58:50

     ... whether the skill can actually do all of these things.

277. 58:52

     Not... So I mean, for us, we're using, um, the skill and replacing the prompt part. So, you know, trying to use that entire thing. I told you, like, just in a minute.

278. 59:02

     I, I think that might clarify your question. Um, so I mean, here it gives me, um, you know, it is like, um, analyze the YouTube video again. It didn't give me the entire answer for the first part.

279. 59:15

     Uh, but yeah, I mean, I th- I think it just, uh, uh... Just give me one second.

280. 59:27

     Wait. I have to clear the context of Claude 'cause it tends to, um,

281. 59:35

     use whatever I asked it previously, and I also had to like, uh, delete the memory folder.

282. 59:43

     Um, so till the time, you know, this is, um, you know, running and, um, generating research for us. Um, sorry. Um, till the time it's like running and generating research, I could talk about the next slide that we have,

283. 59:56

     um, which is the agent skill. So, um, I'm sure all of you must have heard a lot about, um, you know, agent skill. It is very popular right now.

284. 1:00:05

     So, um, this is, uh, basically a compact, concise way of, you know, telling agent about capabilities and workflows and, um, you know. So basically, tools do the work, um, and, um, you know, the skills provide you the way to do it.

285. 1:00:24

     So, um, you know, we talked about the entire prompt that I showed you, uh, you know, the workflow prompt that we have. Um, in that, you know, we were, um, telling, um...

286. 1:00:36

     Sorry. We were telling, um, our agent that, okay, you know, these are the tools that are available and this is how you should execute them. But instead of, you know, writing it in the prompt, we can instead create a skill out of that And what is the benefit of doing that?

287. 1:00:51

     You know, when one thing is already solving the problem, why should we create another one? The reason for that is, um, that, um, skill have something called progressive disclosure.

288. 1:01:01

     So when you, uh, you know, when, um, like I did, um, you know, f- when I showed you... Uh, I, I'm gonna show it to you in a minute.

289. 1:01:09

     When you load the skill, so, you know, when Claude just like shows you the skill, um, it doesn't load the entire information. It is only, uh, load the name and the description of the skill.

290. 1:01:19

     Once you run a query, it's gonna load the entire thing, and once that query is executed, it is gonna wipe off everything. So, you know, it is not gonna be there in the context.

291. 1:01:28

     Whereas when you're using prompt, um, you know, everything is gonna be there in the context. It's gonna pull the context and so, you know, you'd wanna avoid that. So, you know, um, skills are a very clean way of, you know, doing that, and it is also very shareable 'cause, you know, we internally write a lot of skills

292. 1:01:45

     for our team and like share it with each other. So it's a very, um, you know, um... It's a concise, shareable way, and it's more maintainable 'cause, you know, you can check it into GitHub.

293. 1:01:54

     You can... It's like your go-to place, and you can, um, do the entire thing there. So, um, um,

294. 1:02:03

     okay. So we have the results for this, but before that, I would just like to show you how we have written the skills.

295. 1:02:23

     Okay. So I mean, we've written the research skills, so this is replacing the prompt that we have. Um, so here we have to write the name, um, of the skill that you want.

296. 1:02:36

     You need to write the description. So why this description is necessary? Because when, um, your agent is trying to, you know, match a query to, um, a skill, it is gonna look at this part of, um, you know, this part of the skill.

297. 1:02:49

     So you need to have like a good description here. And, you know, then this is called the front matter, which is loaded, and then, um, this is like the rest of the part of the skill.

298. 1:03:00

     So in our case, you know, we are reusing the research workflow, uh, prompt that we had instead of like writing all of the details in here. We... You can do that, but it's a design choice for us, and, you know, this is like the rest of the body for the skill.

299. 1:03:15

     Okay. So now how could you see all of those skills, and what do I mean that it only loads the front matter? So if you go to Claude again, and if you do...

300. 1:03:25

     Oh, sorry. Uh, clear. If you do skills, you'll be able to see. So these are some of the skills that I've already downloaded, and these are the project skills that we have created, so particularly the research skill that I was talking about.

301. 1:03:42

     And, um, y- and, um, so every time you do this and you see this, only the front matter is loaded, not the entire description, right? And you can actually, if you wanna use this, you can just do something like this, or Claude can like automatically pick up, um, the skill when you ask it a question.

302. 1:04:02

     Um, now I'm gonna try to ask it the same question I was trying to ask, um, the MCP, um, with...

303. 1:04:14

     Okay. So I'm gonna ask it that it researches what an AI agent skills are and then also analyzes the YouTube video for me.

304. 1:04:28

     Yeah, so it is loading the research workflow prompt, and it is able to do that because our server is running locally, and that's why it's able to read the prompt because they're all in the same folder.

305. 1:04:40

     And now it has started to do a deep research. You can see here. Oh, sorry. Um...

306. 1:04:50

     Okay. So, so it has created, um... So it is running, uh, you know, the deep research here, and then it has run like... First- firstly, it ra- ran the analyze YouTube video tool, then it is running the deep research tool.

307. 1:05:07

     Um, and I think one amazing thing here that I would like to highlight is that the initial query that I had was, um, you know, can you, uh, give me details about what are agent skills?

308. 1:05:19

     And every time, you know, our agent harness identifies the gap in the output that it gets. So in my initial question, it is gonna identify all of the gap that exists, and then it's gonna ask a different question every time to fill that gap.

309. 1:05:33

     So, you know, it is doing all of the reasoning behind the scene and making sure, um, you know, all of those gaps are filled in the, um, you know, research that we're trying to do.

310. 1:05:43

     Um, so yeah. I mean, it's-- it usually takes around two to three minutes for me to like run this entire output. Um,

311. 1:05:57

     s- let's see.

312. 1:05:59

     Maybe a follow-up question.

313. 1:06:01

     Sure.

314. 1:06:02

     Um, why both skills and MCPs then? 'Cause you could wrap all of this into the skill.

315. 1:06:08

     Yeah. Um, so I mean, um, we built it originally with, you know, the MCP portion of it, and now, you know, we are like transitioning, but, you know, we wanna like keep our original code and, um-

316. 1:06:22

     So your view is it can be replaced, I guess.

317. 1:06:25

     I mean, not the server part, but like the prompt essentially that we have, we can replace it with skills.

318. 1:06:32

     But why not? Like, I, I, I think the whole MCP kind of workflow can be embedded in skill so that the whole code itself with the harness can literally just run it as you inspect it.

319. 1:06:43

     Yeah, but I mean, we have a lot of detailing and different components-

320. 1:06:47

     Yeah

321. 1:06:47

     ... within the servers. That's the reason we wanted to like maintain that.

322. 1:06:52

     So it's more like comp-

323. 1:06:52

     Yeah. Yeah. It's, it's, it's, it's more of the complexity in that case. That's why we wanna, like, maintain that. Um, so here we can see, um, you know, we've got the output of the video, and now we're, we're just gonna wait for it to, like, produce the output for the MCP file.

324. 1:07:26

     Perfect. So, you know, here you can see we've got, um... This was my initial question that I asked it, and, you know, it has given me the answer for that.

325. 1:07:35

     And this is, like, the first run that we have had, and now it's like, you know, doing like more research on that because it, um, you know, gave us an answer, but it identifies some of the gaps in that.

326. 1:07:48

     So, um, this is the second query it ran, and this is like a different question you can see here, right? So it ran two queries, and it is like, "Okay, you know, this is good enough."

327. 1:07:59

     And then it says, um, "All three research tasks are complete- completed. Let me run one more targeted query and then complete everything." So, you know, that's why we wanted, um, you know, all of the reasoning to be done by agents 'cause, you know, this is something a human would do.

328. 1:08:15

     'Cause if you're reading an article or if you're doing any sort of research, you try to identify the gaps and, like, try to fill it with that. So, you know, that's what it is trying to do, and that's why we needed all of the brain, um, and, you know, are trying to utilize that.

329. 1:08:27

     So, you know, this is like the third query that is, it is running for me.

330. 1:08:32

     And then finally, we expect it to, like, create like a compiled report, uh, which is gonna be, um, any minute.

331. 1:08:43

     Mm.

332. 1:08:47

     Can I ask a question?

333. 1:08:48

     Sure.

334. 1:08:49

     Um, it seems like the identity of Quokka comes from 2025. Do you have an idea of why the pipeline... I mean, Quokka knows the date. So why is it asking queries about the latest skills architecture in 2025?

335. 1:09:07

     Um, I think, um, I mean, that is like intern-- I mean, we would have to, like, see the observability for that and, like, check and identify why it is not, like, seeing the latest, um, code.

336. 1:09:17

     But we also have given it a video from 2025, so I think, uh, maybe it's because of that. So the video that I sh- um, asked it to, like, transcribe, it is regarding that.

337. 1:09:30

     But I think, um, to probably answer that better, um, I would, like, go into all of the observability, uh, part of the code that we have and, like, see what is going on there.

338. 1:09:42

     Um, so, you know, finally, it's asking me if it could create a markdown file for me.

339. 1:09:48

     And so here is the research MD file that our final tool creates. And, uh, you know, it is, it is really detailed. It has got all of the comparison tables, and it has got all, like, all of the details.

340. 1:10:04

     So, you know, I think, um, this is like a really nice way to compile, um, everything that we have gotten so far. And, um, it also, like, compiles and adds the YouTube video link that we have, uh, sorry, video transcript that we have at the bottom.

341. 1:10:18

     Um, so yeah. Um, thank you so much, everyone. Um, now I'll hand it over to Paul. [audience applauding]

342. 1:10:28

     Awesome. Thank you, Luis and Samridhi, for those

343. 1:10:41

     great slides, and just give me a second to set up everything over here.

344. 1:10:57

     Okay. So now we are moving to part three of this whole system, which is the Linkin- LinkedIn writing workflow, which basically transforms this research from deep research agent to polished posts that pass slop detectors, so we can transform all of you into LinkedIn influencers.

345. 1:11:17

     Okay. So just like a high-level overview, this is our full system, uh, architecture, right? So we did... We already implemented the research agent which outputs this research MD file, which will be as input to the writing workflow, which on top of that has as input the guideline.md file, which basically is a fancy way to put,

346. 1:11:41

     uh... It's a fancy way of the user input, and this is how we will model the user input to guide the generation process, and the final output will be, well, the, the LinkedIn post, right?

347. 1:11:54

     Okay. Now let's actually zoom in into the, the architecture, which is split into three big pieces. So the first one is where we build up the context, and ultimately we, we end up with the big system prompt.

348. 1:12:07

     We-- 'Cause remember, this is... Ultimately, this is not an agent. This is just a workflow. Because in reality, like to write a piece of content, regardless of what it is, a LinkedIn post, a video, or an article, you don't really need agents, right?

349. 1:12:24

     Because as Luis said in the beginning, this process can be... It's very static. You always kinda go through always the same steps, so an agent will just make it everything a lot complicated.

350. 1:12:36

     Of course, you can put agents in some parts of it, but we won't show that into this, um, into the, this workshop. So the first part is to load the context, which is, as I said, the guideline MD file, which is the user input, the research, and then we have some static files, the profiles, and a few

351. 1:12:55

     short examples which I will dig into a More detail a bit later. So we take all of these, build the system prompt, and then pass everything to the LLM.

352. 1:13:06

     So this is phase two. So, so far nothing fancy. We just basically create this huge system prompt and call the LLM to create the fir-first draft of the post.

353. 1:13:17

     And then the last phase is to apply the evaluator optimizer loop, which is probably the most interesting part of this, where we let the LLM to create in a loop reviews and edit the post a, a couple of times, right?

354. 1:13:33

     So this is super important because in reality, you can apply this not only to LinkedIn posts but to like any type of content, like starting from video transcripts to reports, financial reports, medical reports, uh, even articles or like book chapters or very detailed article lessons as we did in our course where we needed to follow like text

355. 1:13:57

     snippets, images, code snippets, references, and all these little details that LLMs usually sometimes get right, but most often don't get right. And even if they get right 80% of the time, it's annoying to manually go and fix that, right?

356. 1:14:13

     That, that's the whole point. So all of this is automated end-to-end. So we'll dig again into this process in more depth later on. So this is like just like one of our examples.

357. 1:14:26

     So th- this is the beginning of a LinkedIn post that we asked to generate. And

358. 1:14:33

     just for fun, I just want to copy this post, so it's the exact same post,

359. 1:14:38

     and put it into this like pretty popular slop score detector.

360. 1:14:44

     So you can trust me that this actually works [chuckles] and hit Analyze. And yay! No, not, not slop.

361. 1:14:54

     And this is actually lower and less sloppy than a human. [laughs] As you can see, it's over there on the bottom. Of course, not all the posts are this, uh, not sloppish, but most of them are, are really good and that they sound, uh, human.

362. 1:15:11

     But remember, this is LinkedIn, uh, language, so we need to sound a bit like LinkedIn, right? But ultimately, as you can see, it doesn't have any em dashes, any weird, uh,

363. 1:15:24

     adverbs, verbs and things like this. It can be read very, very nicely. The, the structure i-is well... It's schemable as we would like for LinkedIn and things, things like that.

364. 1:15:36

     You, you can check the full, uh, post, uh, at this link in the GitHub repository and also try, try the slop test yourself if you want using that link.

365. 1:15:46

     Okay. So now let's start to dig deeper into the first phase of this, this, this workflow, wh-which-- where our core focus is understanding how we can actually control the generation, right?

366. 1:15:58

     Because as Louis said in the beginning, we don't want just to write, "Hey, I want a LinkedIn post on topic X." We actually want to dump our ideas, values and thoughts into that post so it actually sounds like us. [clears throat]

367. 1:16:11

     So the first trick is actually to structure this guideline, which is the user input, uh, and structure it more than just write me a post on X, right? So this, this actually guides what we write, what we want to write.

368. 1:16:26

     So usually what we did, we created a template for this where we need to fill in such as

369. 1:16:32

     what topic we want to address, what angle we want to address, some key points we want to address, the narrative flow and, and, and things like this. And this is the only piece that changes from post to post.

370. 1:16:44

     So basically from the whole system, this is one we need to write ourself as a human as input to write a different post. And yeah, as I said, this is dynamic and changes.

371. 1:16:56

     Next, the... And again, you, you can see an example here into this example from the repo along many others, but I will, uh, show them a bit later.

372. 1:17:07

     So the second trick is to add these writing profiles, which basically tell the LLM how to write this post, right? Because i-in reality, you don't really want the LLM to just do its own thing.

373. 1:17:19

     You want to guide it. And these are static because, as I said, with the guideline you, you, you define what you want to write and this, this is how you define how you want to write it.

374. 1:17:29

     It's basically the styling layer on top of it.

375. 1:17:34

     And we, we mainly created three profiles with-- which by theirselves are mark-markdown files, which are the structure profile, which bas-- where basically we define things such as how many characters on average a LinkedIn post has, like the, the core structure of a LinkedIn post.

376. 1:17:51

     We want the hook, like the body, the call to action. Also the terminology... For the terminology, we define things such as the active voice and the AI slop words and expressions that we want to ban.

377. 1:18:04

     And un-unfortunately, [chuckles] this is all you can do. You just need to keep track of a huge list of delve, tapestry, vibrant, and all those verbs that we love, uh, and just kindly ask the LLM to not use them until you start using them as a human as, as well. [laughs]

378. 1:18:21

     And the last one is the, the character profile, which is kinda static, but you can also configure it. For example, in our writing workflow, we actually configure it under my, uh, Paul Iusztin biography, right?

379. 1:18:35

     So it knows a bit of... about me, like how I want to write, how I, how I like to write, my style and, and things like this. So this is how you can add a bit of personality to, to it.

380. 1:18:47

     So again, we put all of this in-into the system prompt plus the guideline that we defined before. And you can see these are static. So as I said, we define them just once and you can access them under Writing Profiles and look around the...

381. 1:19:02

     There are, I know, markdown files of a few hundreds of lines of, of, of words, right?

382. 1:19:10

     So the last trick is actually to add few-shot examples. So this is nothing fancy, but in reality, the hard part is actually to get them, right? Basically, this is, uh, under the data collection part of your system.

383. 1:19:23

     And in our particular use case, we added kinda three LinkedIn posts from, from, from my writing. And the, the key idea is to use high-quality and representative, uh, few-shot examples and make them as varied as possible in things such as topic, length, structure, and so on and so forth.

384. 1:19:44

     And one question I, I think many people are curious about is why three LinkedIn posts? I know three is a magic number, but in reality, I just guessed it.

385. 1:19:56

     And the thing is that with few-shot examples, because you always pass them in your system prompt, you want the lower number possible that gets the job done. So usually people tell like three, five, 10, 20 few-shot examples, but usually you want to start with a bigger number and trim down that as much as possible to...

386. 1:20:16

     until it works. And when it stops working, you put, put that back in and, uh, because you want to keep the, the idea is that you want to keep your system prompt with the few-shot examples as small as possible because,

387. 1:20:30

     well, you always pass that to LLM, which translates to more cost, more latency, and even degradant, de-degradance in performance because the, the content grows.

388. 1:20:43

     Okay. So for, for that, we-- I made available a dataset in the GitHub repository where I extracted like, uh, 20 random, um, LinkedIn posts from, from my po- profile that are more, uh, that got more, more traction.

389. 1:20:59

     So I use that as a dataset for, uh, this writing workflow and further down the line to configure oth- other things. And one thing to, to, to highlight is that probably the only way to reduce this load on the system prompt with few-shot example is through fine-tuning, but that, that is often overkill and adds a lot of

390. 1:21:23

     friction into your, your, uh, tech stack. So the last part of the system is the evaluator/optimizer pattern, which probably is the most, uh, uh, fan-fancy one from, from, from all of this.

391. 1:21:37

     And basically it contains two LLM calls, and this is really important. It con-contains two different context windows, right? We have the, the writer and the reviewer, where basically the writer first writes the draft and the reviewer with a complete different context window takes the draft and, and reviews it.

392. 1:21:57

     And this is super important to like avoid bias, uh, because LLMs usually tend to be biased in liking what they already written. So by just putting that into a completely new context window can, can, can remove that bias.

393. 1:22:12

     So basically how, how this works, the reviewer checks adherence of, of the draft against to the guideline, which is the user input, against the research, basically to remove hallucinations, and against the profiles, those structure, uh, terminology, and character profiles, so we ensure that it adheres to it.

394. 1:22:33

     And then the editor, which is often the writer itself, applies the reviews, and then we run this loop in our examples three, four times to basically write the post, review it, edit it, optimize it, and so on and so forth, similar to like a fine-tuning optimization loop.

395. 1:22:51

     And a few tricks here that we absorb over time is that

396. 1:22:55

     just by keeping in our, uh, working directory all those versions, it helps because writing is subjective, and often if you apply this, uh, reviewer too aggressively, it, it, it, it might make it...

397. 1:23:09

     You might n- just not like it, you know? So you want to look around the last versions and pick, pick the one that you like the most. And again, because writing is subjective,

398. 1:23:20

     usually the evaluator/optimizer patterns work with a score. So basically the reviewer gives a score to the input, and you loop until you reach upon a specific threshold. But because creative work is subjective, that threshold is very hard to, to, to quantify, right?

399. 1:23:39

     And this loop becomes very noisy and not reliable, and we, we realize that it's just easier to just put a fixed number of iterations and let the user run these iterations again manually if he just wants to.

400. 1:23:54

     And now let, let's dig a bit deeper into how the reviewer works. So basically as input, it has the current state of the post.

401. 1:24:03

     Next, as context, it gets the guideline, the research, and the profiles, which basically it, it, it's what the writer needs to look around and they are basically the rules the writer needs to, to follow.

402. 1:24:18

     And the outputs is actually a set of pydantic objects, right? And I think this is the most interesting part here is that we actually constrained LLM to output a list of structured objects, and this is powerful because if you give a pydantic object to an LLM, pydantic objects have this property where you, you can put a field

403. 1:24:39

     under each attribute and actually explain what that field means. So basically this is a prompt engineering technique whi- which guides the LLM a lot better into actually understanding what each attribute of the return object actually needs to contain.

404. 1:24:55

     So for example, we-- our review model has this profile location and comment attribute, and an example is, uh, an example output is, for example, it

405. 1:25:07

     We violated the terminology profile on paragraph two, where the LLM used it-- used the leverage band term. And like this, we-- the editor, when it gets these reviews, will understand exactly what, what it violated.

406. 1:25:25

     And we-- From our test, we realized that it's a lot easier to use this structured, uh... Actually, it's a lot more performant to use the, these structured outputs than just letting the LLM, I don't know, spit out whatever it wants.

407. 1:25:42

     Again, you, you can check this code, uh, uh, within this Python file. The code itself i-is pretty simple, so unfortunately, I won't have time to dig too, too deep into it.

408. 1:25:55

     And now on, on the editor part, basically it gets this list of pydantic objects and the current state of the post, and also it gets all the context that the writer initially it has, because the editor is actually the writer itself, right?

409. 1:26:11

     This is similar to how a team of writer and editors will work. I write something, I give it to a, a team of editors, I get the reviews, and I, I apply them.

410. 1:26:21

     And another important thing is that reviews are not created equals, right? Especially people like to, to, to, to give reviews on, on everything usually, and this also is true for the LLMs.

411. 1:26:36

     And because we loop for a couple of times, we realize that it's super important to put a priority on them. So usually we always want to prioritize the guideline first, which is the user input, then the research, and then the profile.

412. 1:26:52

     And this is super powerful when, uh, those, those reviews clash on the same, uh, paragraphs or on the same sentences and things like this. They let me know what to pick up and what to apply.

413. 1:27:07

     Okay. Again, you, you can check the system, uh, the prompt, uh, uh, within this Python file, within the repo.

414. 1:27:16

     And here is a concrete examples where we can see the post V zero, which is basically what the LLM spits out, uh, before the evaluator optimizer loop. And this is how it looks like after four, uh, reviewing iterations.

415. 1:27:31

     So we, we can see that the text is nicely formatted for LinkedIn. Al-also, like the first sentence, which is the most important, is a lot-- is punchier and, and things like this.

416. 1:27:44

     This, this is valid also for like the first part of the post and also for, for the second part, part of the post. Again, it looked at... It modified the structure, the wording, everything to look more on,

417. 1:27:56

     uh, to look more, uh, about something that you would expect on, on LinkedIn.

418. 1:28:03

     And again, you, you can check the whole post and all the versions from zero to four, uh, within the examples directory.

419. 1:28:13

     And ultimately, let's-- let me actually show you [chuckles] how this directory looks like. So to test this code, you actually have three levels. So I created a simple make command just to check that everything works.

420. 1:28:26

     So no-- nothing crazy here. If this works end-to-end, it means your code works locally, or you can pass that to cloud code to, to, to make your code work.

421. 1:28:35

     Uh, so we serve similar to how the research, uh, agent works everything to an MCP server prompt. And then we coupled-- we connected this MCP with, with a write post skill.

422. 1:28:51

     And the thing is that this takes like three, four minutes to run, so I already,

423. 1:28:59

     I already ran it to avoid wa-wasting your time. But here, here is how you can run it. So basically, you take this write post skill

424. 1:29:13

     and just as the LLM, um, we take, for example, the same example that, that, that we took. We cop the-- copy the relative path, and we ask it to use the guideline from this dir.

425. 1:29:30

     And then basically, it will, it will know to pick up the guideline and the research associated with it and, and, and write a post. And the output is

426. 1:29:41

     here. As you can see, we have the four, the four versions of the post... Actually five versions of the post, and also the guideline, which I think is the most interesting one.

427. 1:29:56

     So basically, as you can see, instead of just like kindly asking LLM to write about something, you need to be very explicit about what you want from it. So you need to put an angle, the target audience that you want, uh, key points that you actually want to cover, the tone, and things like this.

428. 1:30:13

     And al- actually, a constraint of characters that overrides like the, the, the other profiles that are static. So encodit-encoding in the system prompt that within this guideline, you actually can override like the default profiles if you want something different.

429. 1:30:28

     And basically here you can go wild and input a-anything you want. But I just wanted to highlight, like you actually need to put in some effort to, to think it through, you know?

430. 1:30:37

     You cannot just fully automate, uh, everything if you don't want to sound like through AI slop.

431. 1:30:46

     Okay. And on top of this, we actually have like a small image generator. We haven't put a lot of effort into this, but th-this prompt actually generates the post using this writing workflow and then asks another tool to automatically generate an image for, for, for that post.

432. 1:31:07

     Okay. So now to move on So just to wrap up on what we build on the writing workflow, we, we build this writing workflow, uh, pipeline that generates reviews and edits the, the post, which again, you can apply to any type of content.

433. 1:31:28

     We tested it with almost e-every type of content, and it works really well.

434. 1:31:35

     You just need to adapt basically the, the profiles, the examples, a-a-and things like this.

435. 1:31:43

     And then we learn, like, how to control the generation, the guideline, the profiles, and the few-shot examples. Again, for different, uh, content types, you will need to adapt these.

436. 1:31:54

     And then how to serve everything as an MCP server and skills. And yes, for this local example, you could do everything through skills. You don't really need MCP servers to make it work.

437. 1:32:06

     But from my experience, MPP-- MCP servers help you distribute logic, uh, a lot easier because skills are like your local setup, your, your local hacked setup that works great for you.

438. 1:32:19

     But if you want to distribute business logic,

439. 1:32:23

     you don't really want to ask someone, "Hey, download those skills. Uh, in-install these UV dependencies over there. Plug in those CLI tools. Oh, no, you also need those credentials."

440. 1:32:35

     And it quickly becomes a mess to distribute this at scale. And MCP servers so-solve that problem, and skills can help you personalize how you use that. Or basically, if you want a skill to be more than a prompt and actually run that code, it works, but it can become a, a quick mess to set it up if

441. 1:32:53

     it's too complicated, right? Uh, so now let's move on to part four of this system, which is observability, more exactly monitoring and ev-evals.

442. 1:33:06

     So we will use the writing workflow as an examples for evals, and we did monitoring for, for both.

443. 1:33:14

     Uh, so for monitoring, we will go really quick over this because, like, theory-wise, there's not much to say.

444. 1:33:22

     So the idea is that the core problem behind monitoring, which i-in my opinion, like exploded when we started to use agents and workflow, is that debugging workflows and agents purely through the logs is hard.

445. 1:33:35

     Like, I personally can't really understand what, what's going on inside the terminal, especially when you see those thinking of the agent w-which hides a lot of stuff. So it quickly becomes painful to debug what's going on just using the, the logs.

446. 1:33:51

     So basically, you need some tool to monitor this nicely that captures all your traces, such as all the LLM and tool calls, your input/output, your metadata. It captures everything about your run, and also latency and cost tracking, right?

447. 1:34:08

     So and as a bonus, it also stores your traces to build AI-- an e- AI eval layer on, on top of everything,

448. 1:34:16

     which I will dig la-- uh, which I'll dig more into it a bit later.

449. 1:34:23

     So now let's actually look, uh, into our mon-monitoring log. So we used Opq to, to, to monitor both our agents, the research and the writing workflow. And yeah, just let's look into it because it's, it's a lot easier to understand like this.

450. 1:34:40

     So for monitoring, usually you have like three big concepts. You have those threads, which is basically in our use case is like the whole workflow of writing an end-to-end post, like starting with the post itself plus the image.

451. 1:34:56

     Uh, for example, as you can see, this

452. 1:34:59

     workflow has forty-four messages which basically captures all the bouncing around between the user and the, the LLM. You can more intuit-intuitively see it as a conversation thread, which that's why it's called a thread, you know?

453. 1:35:16

     And then you have the traces, and here you can actually dig deeper into what's going on. So a thread contains multiple traces, right? And within a thread, you can act-- Within a trace, you can actually dig into what, what's going on.

454. 1:35:30

     So for example, for a generate post trace, here we have the high level overview generate post call, where we can see how long it took to run, how many tokens it consumed, how much it cost to run.

455. 1:35:46

     And here we can see all the tool calls and LLM calls that happen under the hood, right? So we can see the models used, the cost per model, the latency per model, and all of that.

456. 1:35:57

     And on the right, we can also see like the,

457. 1:36:01

     the high level input and output of the system, some metadata, the token usage, and basically everything that happened within that run, which is s- which helps you a lot to quickly understand what's, what's happening, right?

458. 1:36:17

     And we have something similar for the research agent as well. Uh, here the thread is a lot easier to, to, to understand because, for example, we have this latest thread which, which captures all those tool calls.

459. 1:36:29

     Basically, all the deep research, uh, tool calls, the, the, the tool call that creates the final report and all the bouncing around, and then the threads-- the traces which have just single tool calls, that they capture just a single tool call where, where as you can see, uh, we have just one LLM call per, per, per deep

460. 1:36:51

     research tool call. Okay. So now let's move on to,

461. 1:37:00

     to, to, to, to AI evals. So I want to start with why evals matter because it's not necessarily a cool topic, but it can do a lot on, on, on your system So basically, let, let's take our writing workflow example.

462. 1:37:18

     Let's say that we want to, to, to check how well it works, right? We all start with vibe-- with vibes. We, we, with vibe checking. When we do... And when we generate just one post, we read it, we, we say that, "Hey, it's cool, awesome.

463. 1:37:32

     It works." Well, then we have 10 posts. Uh, we read them maybe, maybe not. Maybe we read two, maybe we read thr-three and call it a day and say, "It works for, for the others as well."

464. 1:37:46

     And then when we want to check how well it works in 100 posts, it quickly becomes impossible. And just imagine that you do this at the beginning, but as you evolve your, your, your system, you start to plug in more features, right?

465. 1:38:00

     And every feature can just break everything. It... And yeah, this is standard practice in, in software engineering, but here you work with prompts, and just one word somewhere randomly can just break all your features if you're not, not, not, not careful, you know?

466. 1:38:17

     And you need this, this, this layer on top of it.

467. 1:38:22

     So basically, evals fit in three big, big layers. So we have optimization, [clears throat] which is very similar, like, to training a model. So basically, what we have this AI evals layer which lets us quantify how well our writing workflow writes LinkedIn posts, right?

468. 1:38:40

     Then when we want to improve it, we have a score which tells us that, hey, do we move in the right directions or we don't move? So on every change, we run this AI evals layer, and we know if we did better or worse.

469. 1:38:56

     The next layer is regression testing, which is basically similar to testing classic software engineering, where, uh, whenever we start working on a new feature, we just ensure that we don't break current functionality.

470. 1:39:10

     And the layer three will be to actually run this in production on live traces to actually get warnings and errors and alarms and all of that, uh, when users actually use your system.

471. 1:39:25

     And in reality, how this is different for-- from, from like nor-normal unit integration and regression test is that everything starts from the eval dataset, right? So this is a data problem.

472. 1:39:36

     We actually build a model here. So here is how we build it for our LinkedIn posts, where we were lucky enough to actually already have the data. So I extracted 20 real posts for-- from, from my LinkedIn.

473. 1:39:50

     Why 20? I, I said that is a big enough number for a workshop, but in reality we'll probably need to go at least to 100.

474. 1:40:01

     Uh, then I reverse engineered the guideline and the research. Basically, I took the, the guideline from, from the post, and then I ran the deep research agent on top of the guideline to find, uh, whatever I need to support that post.

475. 1:40:18

     And then I generated the output. So basically, I put the guideline and the research as inputs and generated new posts. And this is super important because you actually want to see results from your real system.

476. 1:40:30

     And when you want to generate, like, synthetic data of some sort, never ask the LLM to directly generate the output. Always ask it to help you generate the input, but never the output.

477. 1:40:41

     You want the output from your real system because that, that's what you're testing ultimately.

478. 1:40:47

     And then I looked around and labeled each output with binary pass and fail, uh, labels and two, three sentences critique on why exactly I gave it a pass or fail, uh, label to it.

479. 1:41:02

     And usually I, I stop when I find the first error. It's just easier, uh, for me while labeling and also for the LLM to understand that whenever I see the first fail, I stop at it, I write why it failed in two, three sentences and move on.

480. 1:41:18

     And ultimately, I split all of this into a train dev test split because ultimately, uh, remember that here we're building a machine learning AI model, so we need to tre-treat it as such.

481. 1:41:30

     So we need a classic split. And only when we have this dataset and, uh, this split, we can actually build the evaluator and evaluate the evaluator,

482. 1:41:44

     which most often, for some reason, when they build LLM judges, they think they can, they can just skip this, uh, last step, which is probably the most important one.

483. 1:41:56

     And again, what's the most important and interesting here is the dataset that we created because actually creating a LLM judge out of this if you have the dataset is just very easy.

484. 1:42:12

     Okay. So now let's see how this, uh, LLM judge would work for, for our current scenario. So as input, it will take the generated post that it, it needs like to, to label, but also the profiles, research, and guideline.

485. 1:42:25

     And this is super important because the LLM judge actually needs to understand the context used to generate this post.

486. 1:42:35

     And ultimately, it outputs the pass, fail, uh, label plus the critique. Basically exactly the same labels as our dataset. And then we pass as few short examples our train split from the dataset.

487. 1:42:50

     And this is the most important one because, like, the system prompt of the LLM judge can be extremely simple if it has the right few short examples in place that tells the LLM judge, uh, what decisions to take, right?

488. 1:43:06

     So the, the few short examples look like this.

489. 1:43:10

     It-- They have the writer input, which is the guidance and research, right? So the input to our writing workflow system. Then the writer output, which is basically the generated post of the workflow and the labels, right?

490. 1:43:24

     So in the few-shot examples, we also have the labels, the pass and fail label, and the two, three sentences critique about the label.

491. 1:43:33

     And this is like you, you can check out our system prompt, uh, within this file, but th-there, there is nothing fancy. The most important part from this is building the dataset.

492. 1:43:47

     And our last step is to actually measure the judge re-reliability. Because ultimately what we did here, the LLM judge is just a binary classifier, right? We, we, we output pass and fail labels.

493. 1:44:00

     So yeah, we used an LLM judge because the reality is that it's easy, but we could have just used any, any, any other model that gets the job done and can output these pass and fail labels.

494. 1:44:15

     And ultimately, when we measure judge reliability, our final goal is to align the LLM judge with the domain expert. And how we will do that,

495. 1:44:26

     we will do that by testing it against the dev test, uh, dataset splits.

496. 1:44:33

     So this is a process very similar to training any other binary classifier. There's, there's nothing new here, just the word LLM judge i-is fancier. So what we do basically is test the LLM judge against the dev test dataset splits, and then use F1 score, the F1 score, which is a combination of precision and recall, to actually measure

497. 1:44:55

     how well the LLM judge perform against these splits. And the process look, looks like this.

498. 1:45:02

     So we first run the LLM judge on the dev split, compute the F1 score, and adjust the LLM judge and prompt examples to maximize this F1 score. Because most probably when we first start this process, the F1 score will be low, and we basically want to

499. 1:45:19

     get a score as high as possible. And we repeat until we converge. Usually, you need to do this a couple of times. And when you think you're done and the LLM judge is ready to go, you run it on the test split as the final validation step, right?

500. 1:45:37

     Basically training a machine learning model. And now it's ready to run on new data. So after we go through these steps, we have the LLM judge, which is ready to run on new samples of data, which will just output the binary, uh, pass or fail and the critique.

501. 1:45:59

     Now, like to wire everything together that we had in this, in this section

502. 1:46:05

     is that we first build the dataset, then based on that, we build the LLM judge, then we calibrated the judge, and only after that we can run it on real data.

503. 1:46:17

     And usually all these steps are managed by some observability platform. We used here Opic, but

504. 1:46:25

     you can use any other platform or just build something in-house. It doesn't really matter. But the idea is everything should be, uh, managed by a cohesive platform.

505. 1:46:37

     Okay. So now as a quick demo, let's emulate as much as possible this calibration step, because the LLM judge is already calibrated, so I will just show you

506. 1:46:52

     how the LLM judge performs on the, on the eval split.

507. 1:47:07

     So under the hood with Opic, we, we built like a very simple evaluation harness where it, it allows us to run these calibration steps and then run it on, on production traces.

508. 1:47:21

     So in reality, the code i-is, for our use case is, is pretty straightforward. So just to show you around

509. 1:47:30

     a bit until it, it runs. Okay. So actually, what, what it does now, it, it, it, it takes all the generated posts, right?

510. 1:47:48

     It pass, passes the LLM judge on top all, all those generated posts. And then what I wanted to show you is this F1 score, where basically we compute an F1 score based on the outputs from the LLM judge against our labels from the, uh, dataset, which I think is more interesting to see how our dataset looks like

511. 1:48:12

     really quickly to get a sense of it. So

512. 1:48:19

     basically here we have like a single sample

513. 1:48:25

     within our dataset where what's more interesting is like we have a link to the media of the post, to the guideline, to the generated post. Basically links to everything that we need as input and output, both for the deep research agent and the writing workflow.

514. 1:48:41

     And then just putting this scope, we plug them in as a few-shot examples e-either for the post generator or for the evaluator or, for example, for the dev split and things like this.

515. 1:48:55

     So this is how we are careful that we do the splits the right way between the LLM judge and also the, the, the generator. And for simplicity, we have all those files linked together in this YAML file here, uh, locally within GitHub, so you can check everything and play it, play it

516. 1:49:18

     And run everything very easily with, with minimal configuration.

517. 1:49:25

     Okay, so if we go back into the terminal, we saw that it had like a perfect score. Yay. So now let's actually run this on the,

518. 1:49:41

     on the test split. And the goal here is to see that our LLM judge did not overfit it, right? So we ran it on the dev split, we get the perfect score.

519. 1:49:51

     Usually this is a high signal that it overfitted. So unless the F1 score on the test split is not one or around one, it means that our LLM judge is overfitted.

520. 1:50:04

     So again, similar to the process before, it's basically the s- the exact same, uh, steps but now ran on the test split.

521. 1:50:16

     So let's see the final F1 score, which is again one,

522. 1:50:23

     which is good. And the scores are so good because we do this just on five samples per split. But i- if you like expand your split, which you should have at least like twenty, thirty samples, uh, the score probably will not be perfect.

523. 1:50:39

     Okay, and let me actually show how these experiments look inside, inside Opic.

524. 1:50:48

     Can I just quickly ask a question?

525. 1:50:49

     Yeah, sure.

526. 1:50:50

     In terms of your decision of deploy the judge, what sort of the range F1 score you would say is good, good practice between overfitting versus reality, et cetera?

527. 1:51:00

     Yeah, I, I think that the most important is like the dynamics between the F1 score on your

528. 1:51:06

     dev split and the dynamics on your test split. Because the over, the overfit... When the model is overfitted, it is actually

529. 1:51:16

     like how it's compared between the two. For example, if it... as I said, if it has like a very big score on the dev split-

530. 1:51:23

     Yeah

531. 1:51:23

     ... and a low score on the test split, it means it overfitted. But the score itself usually is very, um, correlated with your own data. So for example, you're pleased with, uh-

532. 1:51:36

     Would you say what you're looking for is more like the relativity of the both score, like if they are more in the same range?

533. 1:51:43

     Yeah, yeah. Exactly, exactly.

534. 1:51:45

     So it's less to do with the absolute value-

535. 1:51:46

     Yeah, exactly

536. 1:51:47

     ... of F1 score.

537. 1:51:48

     Yeah, exactly. The absolute value usually is very correlated with your data.

538. 1:51:51

     Yeah.

539. 1:51:52

     Usually on the test split you say, "Okay, this F1 score is good enough for me"

540. 1:51:57

     And then you run on the test split to see that you have the same F1 score on the test split as well.

541. 1:52:05

     Okay. So again, usually we- with these observability platforms, you have this experiments tab where you can actually run like,

542. 1:52:15

     uh, experiments similar to fine-tuning experiment trackers from, from, from, from the old school. And for example, here we have the dev and test experiments. So let's open the, the, the test experiment that we just ran, and here we can actually see the output from the binary LLM judge and our label, and we can see that there's a,

543. 1:52:38

     a perfect match between the two. And if we hover over this, we can also see like the, the critique of, of the, of the judge. But in reality here, we compute the score only between the, the labels and this critique is more useful for us because ultimately our goal is us as humans to look over these results

544. 1:53:00

     and understand what's going wrong with, with the system.

545. 1:53:05

     And then as a final step, we... I

546. 1:53:09

     also prepared like a online, uh, simulation where I simulated some online, uh, traces

547. 1:53:21

     where the LLM judge actually runs on top of them and

548. 1:53:26

     you get the binary labels and the critique. But the thing is that this takes like five minutes or even more to run because it actually needs to gen-- for real, it needs to generate all the posts and run the LLM judge on top of it.

549. 1:53:38

     So I'll just show you a result over here,

550. 1:53:43

     which, uh, I, I bundle it together as an experiment.

551. 1:53:48

     So it's this online test evaluation suite where, again, for example, just let us dig into, into what further. You can see, for example, the generated post here on, on the right and then the results from, from, from the binary, from the judge where we, we will find the score and the critique itself.

552. 1:54:13

     And of course, this system is not yet perfect. I probably need to refine it more because here on, on live traces, as you can see, it kind of failed.

553. 1:54:23

     Uh, to, to be honest, it just passed, uh, an F1 score. Uh, it just give us a pass on all the traces, while in reality I also have the labels, uh, for these simulated scenarios.

554. 1:54:35

     We can see that it failed. So probably I, I need to go back to the dev split, expand the dev split with new, uh, samples. For example, I could just take these samples and put them in the dev split and start refining it again and again and again until, uh, it actually works.

555. 1:54:55

     And I'm kind of running out of time, but

556. 1:55:00

     you also have this skill to actually run a, a demo end-to-end, right? Research and writing. You can, for example, write a guideline about s- about something that you want to write a post about and give that as input to this skill, which is just a file, and it'll know to pick up to do the research and then

557. 1:55:18

     continue to writing the post. And you can also observe that i- in Opic with all the, the threads, traces and all of that. And yeah, I guess that the final step, if you haven't done so yet, is to actually to open up the GitHub repository, run everything yourself and read the code.

558. 1:55:39

     Because without reading the code, most probably we, we won't really understand what, what's going inside. So you can do that by accessing this, this link or scanning the QR code.

559. 1:55:51

     And whenever you're ready and you want to go deeper into building this type of multi-agent systems, we have this agentic engineering course, which basically was inspiration for, for this workshop.

560. 1:56:06

     But instead our goal was actually to design and build production-ready AI agents and not just some s- small local systems, uh, within thirty-four lessons, three end-to-end pro- portfolio projects, a certificate and a Discord community with access to us.

561. 1:56:24

     And so far it's rated five by five by three hundred plus stu- students. And if you don't believe us, the first six lessons are, are free to try out.

562. 1:56:36

     And you can access it using this link or scanning the QR code. Yeah, uh, and that's it for, for the workshop today. [audience applauding] [upbeat music]