AI Engineer World's Fair 2026
Turn 10,994 Notes Into Your Agents' Memory
Read the talk
Turn a Second Brain into an Agent’s Research Memory
A file-based research system connects saved notes to current work, then preserves sources, summaries, and new questions in a project wiki that can keep growing.
From a talk by Paul Iusztin and Louis-François Bouchard
Before you start: Familiarity with coding agents, Markdown files, and basic retrieval-augmented generation will help you follow the architecture and demonstrations.
Finding the research you already saved
Paul Iusztin spent 18 months turning his second brain into a research memory. He reports more than 5,000 notes in Obsidian, another 5,000 in Readwise, material scattered across Notion and Google Drive, and roughly 250 new files each month. Accumulation was working; bringing the right information into an article, project, codebase, or feature was the harder problem.
The missing piece sits between the archive and tools such as Codex, Claude, or NotebookLM. Saving an interesting X post, article, YouTube video, or GitHub repository does little good if, when work starts, you cannot remember that it exists. Searching manually can recover it, but finding the useful subset takes time. The goal is to retrieve high-signal research for the work in front of you, while preserving the personal notes, values, and taste that made the collection worth building.
That is the purpose of AI Research OS, the system Iusztin and Louis-François Bouchard introduce with accompanying code. Iusztin, founder and CEO of Decoding AI and co-author of The LLM Engineer’s Handbook, uses it in his daily work. Bouchard introduces himself as Towards AI’s co-founder and CTO, the creator of What’s AI, author of Building LLMs for Production, and a former PhD student. His research now feeds courses, videos, company training, and lessons drawn from client implementations. Their workflows differ, so the repository is designed to be adapted to an existing research practice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When persistent research earns its setup cost
A research operating system is unnecessary for many questions. If you need a quick answer—the kind of thing you would otherwise Google—use search or a chat assistant. The extra structure becomes useful when the work requires a long sequence of questions, a growing source collection, and decisions you will revisit.
For a small repository change or a one-off article, Bouchard recommends an agent such as Codex or Claude Code. Persistent research becomes more valuable when the next session must continue improving the same work, refer to previous sources, or digest them more deeply. His video workflow supplies a concrete requirement: a new video should take earlier videos into account, avoid duplicating them, and refer back to useful prior content.
| Approach | Good fit | Tradeoff in this workflow |
|---|---|---|
| NotebookLM | Source-based research and repeated exploration | Less ownership and customization |
| Vector-database RAG | Production retrieval at scale | Infrastructure and inspection tooling |
| Personal file-based wiki | Inspectable, reusable project knowledge | More initial setup |
Bouchard finds NotebookLM powerful for digesting research, but describes its browser-based workflow as a poor fit for the agent integration and coding work he wants. A vector-database retrieval pipeline is attractive for production products, yet checking or editing its knowledge by hand requires surrounding infrastructure. For personal use, the desired result is closer to an editable Wikipedia: sources, comparisons, implementation knowledge, and new topics that remain available to other projects and agents. Building that system gives more control, at the cost of more setup than simply opening Claude Code.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move memory out of the conversation
Consider the repeated work around a research session. You give an agent links, PDFs, and background material; it creates temporary structure and runs scripts to make sense of them. In the next session, you paste the material again or explicitly ask it to reuse a growing collection of skills. The bottleneck is no longer supplying information. It is preserving enough organization to use that information later.
Without durable storage, the context window is asked to serve as database, filesystem, memory, and reasoning space at once. Ending the conversation breaks that continuity. The proposed remedy is memory and context management built around plain files, mostly Markdown: humans can inspect them, agents can read them, and neither needs the original conversation to remain open.
Bouchard began with a few hundred notes, hundreds of previous videos, and knowledge from team and client meetings. He consolidated Claude Code skills, Granola meeting recaps, Apple Notes, and saved browser tabs into local files viewed through Obsidian. Codex helped configure automatic saving from Granola. Obsidian provides the interface he likes across phone and desktop; the underlying requirement is local files, not that particular application.
The AI Research OS workshop repository packages this workflow as skills and plugins for agent harnesses. Its research tool can begin with a new video topic and draw on local Obsidian notes, Readwise, NotebookLM, GitHub repositories, YouTube links, web pages, and documents. Users install the plugin and tune the sources and behavior to their needs. Bouchard describes adding YouTube transcript support with a single prompt in a few seconds. Beneath those integrations are three durable layers: raw content, an index, and synthesized wiki pages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
V1: turn a topic and golden links into a report
The first version had a narrow job: prepare research for lessons in an agent engineering course. Its input was a topic plus manually selected golden links; its output was one research.md file. Scraping the trusted links first gave the system enough context to frame better questions than it could generate from the topic alone.
The research loop then worked in stages:
- An orchestrator generated questions from the topic and scraped seed context.
- Separate agents investigated those questions using Gemini grounded in Google, collecting links and executive summaries.
- The orchestrator aggregated the results into a compact representation before another round of questions.
- The system ranked discovered sources against the original topic, fully scraped the top
K, and retained summaries for the remaining links. - It compiled the selected material into a flat
research.mdreport.
Paul reports that three rounds of six queries produced approximately 40–50 links. Ranking mattered because discovery generated enough material to introduce substantial noise; selective full scraping kept that noise from becoming an equally large reading burden.
Paul reports quickly generating 35 lessons with this approach. It served the course workflow, but its useful output was still a static file, and the initial high-quality sources still had to be selected by hand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
V2: let the second brain supply the seeds
The next version aimed the same research loop at the personal archive. A saved collection already reflects choices about what is interesting, trustworthy, or relevant. Searching it lets those choices supply golden links organically, instead of requiring a fresh list for every task. The input can therefore be just a topic, with explicit seed links still available when needed.
The topic seeds query generation, and those queries reach both the second brain and the public web. The described sources include Obsidian, Readwise, NotebookLM, and GitHub; Gemini Deep Research is another suggested extension. Ranking and selective full scraping still follow discovery, and the output is still research.md.
This solves source selection but exposes a different limitation. Research rarely ends when a report is written: another question appears, information becomes stale, or the project needs a different comparison. A flat report does not preserve a convenient update process. Repeating the whole discovery and compilation loop costs tokens and time, so the third version adds a wiki above the research process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
V3: preserve sources and build a reference index
V3 combines deep research with an LLM knowledge base. Sources go in and a wiki comes out. Custom URLs can also enter the pipeline; Paul mentions Bright Data as a way to parse web content, including single-page applications. Instead of folding everything into one report, the system stores source files individually, creates an index, and generates wiki pages above them. Queries can then use both the index and the wiki.
For this lightweight personal system, Paul deliberately avoids making vector databases, knowledge graphs, or dedicated search infrastructure prerequisites. Files and references supply the retrieval structure. The entry point is index.yaml, a catalog containing source summaries and metadata. His displayed catalog contains 10 sources and 38 derivative wiki pages. An entry can identify the original file, origin, title, authors, publication date, and summary; its references lead to raw material and generated pages.
A minimal catalog entry for a sandboxing source illustrates the file-reference mechanism:
yaml
sources:
- id: sandboxing-note
title: Sandboxing research note
origin: local-note
raw_path: raw/sandboxing-note.md
summary: Notes on sandboxing and its role in an agent harness.
source_page: wiki/sources/sandboxing-note.md
derivatives:
- wiki/concepts/sandboxing.md
The summary helps an agent decide whether the source is relevant. The paths tell it where to read more without embedding the complete source in every prompt. The metadata can grow with the workflow while the basic operation remains following references.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read progressively, from summaries to raw evidence
| Layer | Contents | Role |
|---|---|---|
raw/ | Original source material | Immutable evidence |
wiki/ | LLM-generated pages | Reusable synthesis |
index.yaml | Summaries, metadata, references | Agent entry point |
The wiki can contain comparisons, entities, concepts, notes, repository analyses, and open questions the agent could not answer. A repository need not collapse into one summary: it can produce several pages about different architectural concerns. Obsidian can display the connections created by these references. Concept pages might cover a tool registry, context compaction, or sandboxing; entity pages might describe OpenCode, Claude Code, or MCP.
Retrieval follows a hierarchy:
- Read
index.yamlto select promising sources from their summaries and metadata. - Open the source wiki page, an expanded executive summary of the underlying material.
- Follow relevant references into concept, entity, note, or comparison pages if more synthesis is needed.
- Read the full raw article, paper, or video source when those layers do not contain the required detail.
For the sandboxing entry above, a broad question may be answered by wiki/sources/sandboxing-note.md; a more specific one may require wiki/concepts/sandboxing.md or the raw note. Paul describes this progressive reading as token efficient: the agent can stop once it has enough information instead of repeatedly loading full sources. The talk supplies no measured token reduction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let project knowledge change while protecting personal notes
In the workshop’s design, questions leave a durable trace. A question is logged and can lead to a new concept, note, or comparison page, so the wiki develops through use as well as ingestion. Custom links and additional deep-research rounds supply two other ways to extend it. The resulting collection can reflect both what you have researched and what you have struggled to understand.
This evolving wiki does not sit over the entire personal archive as an unrestricted editing layer. Paul organizes his second brain with Tiago Forte’s PARA method: Projects, Areas, Resources, and Archives. In his implementation, saved notes, PDFs, and articles enter a flat resources collection and are referenced from projects and areas. The global Obsidian collection remains his own material; the LLM does not edit his manually written notes.
A new project draws a relevant subset from that global second brain through research or direct ingestion. Skills inside the agent harness then work with the project’s scoped wiki. The project might be an article, video, slide deck, book, course, or codebase; Paul used the technique to prepare this presentation. The boundary is practical: the second brain holds accumulated research, while the project is where that research becomes work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Launch research from an article brief
The workshop packages the required skills as a Claude Code plugin, with adaptation to other harnesses possible. The repository README covers installation and dependencies, including the service-specific CLIs and authentication needed for personal sources such as Readwise and NotebookLM. The demonstration moves past setup into a prepared article-research example.
The first input is a brain dump for an agentic-engineering article, accompanied by references Paul definitely wants included. He opens a Claude session, invokes the research skill, and points it at the brief. The skill infers the intended topic—agentic harness engineering—and scrapes the supplied references before generating research questions. Paul then enables auto mode, explaining that he trusts this workflow after hundreds of uses; that confidence is specific to his setup, rather than a general safety guarantee for unattended agents.
The next decision is research depth. The skill offers light, deep, and fast settings that govern the number of rounds and questions. Paul generally recommends light or fast because broad research consumes substantial tokens.
| Preset in the recording | Query plan |
|---|---|
| Light | One round of three queries |
| Fast | Two rounds of three queries |
| Deep | More extensive research; exact counts not given here |
Paul selects light and estimates 10–20 minutes to search his Obsidian, Readwise, and NotebookLM sources. When following along with a newer checkout, use its research skill specification: the version inspected in August 2026 instead defines fast as one round of three queries and light as two rounds of three then two queries. It also defaults existing-wiki queries to read-only, with optional save-back, rather than making the workshop’s question-driven updates automatic.
To inspect the result, Paul opens an already generated wiki in Obsidian. It has raw copies, an index, and a graph of linked wiki pages. The source pages contain executive summaries computed during ingestion, so later queries can reuse that work. The derivatives include comparisons such as agentic RAG versus filesystems and compaction versus recursive language models. Concept pages extract ideas from the source collection; the agent-loop page combines explanatory text and graphs. The output is therefore both a retrieval surface for an agent and something a person can browse to learn the subject.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Study harness architecture directly from repositories
The second example skips discovery. Paul wants to learn how harnesses work by ingesting OpenCode, Pi, and Hermes. The repository-ingestion prompt supplies the selected repositories and explicitly asks for no deep research. Its requested topics are general architecture, agent architecture, subagents, memory systems, and permission flows.
The workflow clones the repositories and investigates those topics in each codebase. First it creates repository-specific Markdown analyses. Higher-level wiki derivatives can then compare the architectures or synthesize recurring patterns across them. This ordering matters: a cross-repository comparison is built above individual repository analysis, so the reader can move between a broad claim and the implementation it describes.
Paul again opens previously generated results. The familiar raw/index/wiki structure remains, with per-repository pages about topics such as OpenCode’s permission flow and memory system. Comparisons expose architectural differences, while concept pages collect the decisions and mechanisms worth understanding. This makes the same workflow useful for learning from existing code before designing a harness of your own.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with only a few links
The third example reduces the input to three custom links. Paul starts a fresh session and pastes a prompt asking the system to ingest them. Neither this example nor the repository example needs an Obsidian, Readwise, or other personal-note integration. As presented, they use the plugin with Git and curl; Obsidian is a convenient viewer for the resulting files, not a required storage service.
The generated wiki still has an index, an executive summary for each source, extracted concepts, and entities. The displayed index is titled “Agentic AI systems: agent memory, agentic coding, and GraphRAG.” This is the smallest entry point into the workflow: ingest a bounded source set, inspect what it produced, and then ask questions that make the collection more useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return to the wiki with a new question
The final demonstration returns to the wiki built from GitHub repositories. The presenter invokes the research skill, points it at that existing wiki, and asks how remote sandboxing works and how it plugs into a harness. The source set is already available, so the new task begins with querying accumulated knowledge rather than repeating repository discovery.
The intended next step is to answer from the wiki and, where useful, create notes, comparisons, or newly identified entities. The recording introduces that query and describes the possible additions; it does not walk through a completed sandbox implementation or a finished answer. The mechanism is the important distinction from a static report: a follow-up can become durable project knowledge instead of remaining only in a conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The remaining work is maintaining trustworthy memory
The project remains deliberately incomplete in places. Google Drive, Notion, and Slack are proposed connectors, not implemented integrations in the workshop account, despite appearing earlier among possible source destinations. The authors had not needed them in their own workflows. The repository’s purpose is personal usefulness and teaching memory and context management, with users adding the connectors their work requires.
A deeper limitation is deciding which sources have become outdated and which are weak or strong. File visibility makes inspection possible, but it does not automatically establish source quality. The system also remains a terminal-oriented builder workflow through Claude Code or Codex, rather than a polished consumer product. That is an intentional scope choice.
The planned improvements address the maintenance burden:
- Stronger linting: improve the checks applied to the growing knowledge collection.
- Better memory compaction: manage accumulated information as the wiki develops.
- Better source provenance: make sources easier to trust, rank, reuse, and assess for relevance.
These are the problems that remain after retrieval works: keeping a living memory coherent and making its evidence understandable to the person using it.
The educational continuation is Agent Engineering: Building Multi-Agent Systems at Towards AI Academy. Bouchard describes a related system with research and writing agents, incorporating further improvements beyond the workshop project. He estimates around 60 hours to complete the course, whose final project is a multi-agent system learners build themselves. The workshop’s files-and-references architecture provides a smaller starting point for the same practical discipline: preserving research so an agent can use it again when the work changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Companion plugin, installation instructions and three examples for building a persistent research wiki.
The workshop prompt for ingesting OpenCode, Pi and Hermes without running discovery rounds.
Open-source coding agent examined in the repository-ingestion demonstration.
Coding-agent harness with an agent runtime and shared model interface.
Nous Research's agent with persistent memory and reusable skills.
The creator's explanation of organizing information into Projects, Areas, Resources and Archives.
Course curriculum for building and integrating research and writing systems, with evaluation and deployment.
Updates since the talk
Current specification for querying, ingestion, discovery depth and optional wiki updates.
Read the complete timestamped transcript
- 0:00
I spent 18 months turning my second brain into my living research memory. Let me explain. So within my second brain, I currently have over five thousand notes in Obsidian, and another five thousand notes in Readwise, and some scattered in Notion and Google Drive.
- 0:15
And all of this is growing on average with two hundred and fifty files per month. And this is what I want. On the left, you can see my whole Obsidian vault, this huge mess, and whenever I start working on something, such as an article, a new project, a new code base, a new feature, or whatever, I want
- 0:31
to actually pull high-signal notes that are actually useful for my current work. And you would ask yourself, why not use directly Codex, Claude, or NotebookLM? And the thing is that I am, but you need a system that sits between those harnesses and your second brain.
- 0:49
Okay, so let's go back to the root of my problem, which is that I'm always losing my research. For example, my reading list is a graveyard. When I'm scrolling social media and I save that cool X post, a, a new article, a new YouTube video, a GitHub repository, it doesn't matter.
- 1:05
Whenever I actually want to start working on something, I never recall what I have in my second brain, or I have to spend a ton of time actually finding meaningful notes that I can use in my work, right?
- 1:18
And another problem that I have is that I want the system to actually be anchored into my personal notes, into my personal values, into my personal taste. I want the system to be personal, to reflect my own thoughts, right?
- 1:32
And that's why in today's video, Louis-François and I will teach you how to build your own AI Research OS. This also comes with code, so you can also try it out yourself.
- 1:43
And I'm Paul Iusztin. I'm the founder and CEO of, of Decoding AI, where I do a ton of content on courses on how to ship AI products, and I'm also the co-author of The LLM Engineer's Handbook bestseller.
- 1:55
And the system, the AI Research OS that I will teach you in this video is the system that I use in my daily work. And now I will pass the torch to Louis-François.
- 2:05
Thanks, Paul. So I'm Louis-François Bouchard. I'm the co-founder and CTO of Towards AI, where we build educational courses. And I'm also the creator of, uh, What's AI, a YouTube channel where I explain AI engineering techniques.
- 2:19
I used to explain AI research before, now focusing on AI engineering. I'm also the author of the book Building LLMs for Production. And, uh, before that, I was a PhD student.
- 2:31
So I honestly make research for a living. I used to do a, a PhD, as I said, in AI and doing tons of research and research work. Now, I build courses, I write videos, I research for videos, I build trainings for companies for a living.
- 2:46
And all of these things that I do start with a very good research, and also leveraging tons of knowledge and insights that we get, uh, at Towards AI from building for clients.
- 3:01
So I have tons of notes as well, just like Paul, and we try to leverage them the best possible. And as you'll see, we build some sort of tool to leverage our second brain, where, as you'll see, there will be some differences between how I use it and how Paul uses it.
- 3:17
And that's the core goal of the repository that we built and on, on this project, is that we want you to adapt it for your needs. The whole goal is how can we make research better, but more specifically, how can we better leverage what we have?
- 3:31
So let's dive into it. And first, we need to figure out which tool to use and when, because this whole research system that we build is not for every query.
- 3:43
If you just need a fast answer, like a few quick questions or just something where, uh, that, that you would just Google basically, well, obviously just Google it or ask ChatGPT, Claude, whichever system you want.
- 3:57
But the problem when doing that is that if you have a lot of follow-up question or it's a bigger project that you need to build on, uh, and have basically a very long context or tons of information to share, relying on ChatGPT isn't ideal.
- 4:12
And it also means that you f- are fully dependent on the architecture that OpenAI or ChatGPT's team built. So the next step here is to ask yourself for a more complex problem, do you need to act quickly, or do you want to build some next feature and, and do something very difficult?
- 4:32
If you just have a small repo for a quick change or write one article, just the one thing that you know won't be repeatable that much, definitely use Codex or Claude Code or some agent that you trust.
- 4:45
Sometimes you need to keep on digging to make it better, to improve efficiency, may optimize it more. And so typically, when you have to do that, you want your research sources, your research to stick and to be able to refer to them in the future.
- 5:00
So if you want a process like this, where the sources y- that you find, the notes that you take stick around in time and have an agent be able to leverage that efficiently, and being able to come back to this information, to ask follow-up questions, to digest content even more.
- 5:18
And right now, for instance, when I make a new video, I want also the agent and the system to understand the previous videos I made, to not duplicate content, to not repeat myself, and to refer to some other content.
- 5:30
In this case, there are some tools that are very interesting that you might have tried before, like NotebookLM, that is super powerful to do research, to digest content efficiently, and to come back to it.
- 5:42
But the problem with NotebookLM is that it's... Well, first, the main problem is that you don't own it. You cannot do anything you want with it. You cannot personalize as much as possible.
- 5:51
It's not agent native, and it's obviously weak for coding tasks, uh, since it's just browser-based. So it's far from ideal from Something that Paul and I needed, and that most AI engineers need in general.
- 6:07
So if you need your agents to be able to leverage all you do, uh, whether it is a big research, a new video, whatever you write, you do, you code, you typically want your other agents, your other projects to be able to leverage what you learned from what you just did.
- 6:22
And one thing that we advise, especially for production, obviously for a product, is to build some sort of retrieval RAG pipeline with vector databases. But this needs an infrastructure.
- 6:35
It's not really human friendly to be able to digest quickly, to check notes, to make edits. It's hard to inspect by hand. You need to build everything around it.
- 6:44
It's definitely far from ideal for just something I want to use on a daily basis. Obviously, it's super powerful at scale, very interesting, especially in a product. But as I said, this project is for us, and I don't want something live, super professional as a product.
- 7:01
I just want something I will use, and that my agents and different projects can leverage as best as possible. So the last question to ask ourselves here is that if you want everything there but more personalization, so a personalized research assistant that builds some sort of Wikipedia that compounds over time and it, and is easily inspectable
- 7:26
and usable, where you have tons of sources, documents, videos, comparisons, implementations, new research, new topics that you keep on adding and that you keep on wanting to leverage and review easily, this is where you may want to build something yourself.
- 7:44
And in our case, we built the personalized research OS that we will share in this talk with exactly what we built and how. But the downside is that it definitely needs a bit more setup than just opening Claude Code.
- 7:59
Right now, the main problem with using Claude Code and other agentic tool is that you give Codex links, PDFs, and different information. For example, my most recent loop engineering video.
- 8:12
And then the next session you use Codex, you have to paste it all again or ask it to use skills. And whatever structure that Codex or ChatGPT, whatever tool that you use, builds on the fly to leverage what you did, the scripts it ran, the scripts it had, you all lost it or kept it inside a skill
- 8:31
that you have to ask it to reuse. And that usually isn't ideal and just grows and grows over time. And the problem is that all this information that you give to the model is not the bottleneck.
- 8:45
The bottleneck is: how can you leverage it in the future? Meaning that with an agent, the context window becomes everything: the database, the file system, the memory, the reasoning space.
- 8:56
It has to do it all. And when you stop the conversation, it loses everything. And the thing is that we don't need necessarily to provide more and more and more context for a better research.
- 9:07
You need a proper memory and context management and ideally some personality with it, especially in my case when I do videos. So what we did is that we decided to build a system with plain files, mostly markdown files, that we can leverage easily and that agents can leverage easily.
- 9:26
I won't detail it very much here because Paul will talk about it in depth. And as I said, Paul has like five thousand or something notes. I have just a few hundred.
- 9:36
But that's just to say that we need to consider that we didn't start from nothing. We already both had some sort of large database. In my case, I made hundreds of videos, and I take many notes.
- 9:48
So I still need to leverage these years of content that I already made and tons of meetings that I have with, uh, my team, with clients when we build for them that I want to leverage as well, because we learn a lot by building for people.
- 10:04
We have highlights from interesting tweets that I see, interesting articles that I see, and I want all my projects to be able to leverage my agent skills. So I decided to pivot, and instead of having a folder for Claude Code skills and having all my meeting recaps in Granola, and having years of notes on Apple Notes, and
- 10:24
the tweets on a saved Chrome tab, instead, I moved everything automatically into Obsidian. It's just a note reader, obviously, so you don't have to use that. You can just save it locally.
- 10:36
But I used Codex to set up everything so that Granola is automatically saved there. My notes are now on Obsidian just because it's a nice UI, I like it, and I can use it from my phone, my computer, my Windows, Mac, everything.
- 10:49
So anyways, I moved everything to Obsidian, which means that it's saved locally in my file system, which means it's basically my companion for researching and building everything I build now.
- 11:01
And what we built, obviously, leverages that. We built a repo called AI Research OS for this workshop, where it's basically just skills for Claude Code and Codex with plugins to be able to do a very deep research about a topic or a simpler search or destination.
- 11:21
Different tools that you can use. The most useful and complete one will be the research tool that I use, for example, when I kick off a new video topic.
- 11:30
And the goal of this repo is to have you implement it, install the Claude plugins from it, and tune it to your needs. Right now it can connect to, as I said, Obsidian with my local notes.
- 11:44
It can use Readwise, NotebookLM, your GitHub repos, any links that you send for GitHub or YouTube videos, and, uh, web links obviously, and documents. But there are tons of things missing, as we will discuss in the end, that you can easily implement just asking Claude Code or Codex to do so.
- 12:02
Like for example, I implemented the YouTube video transcript in Honestly, a few seconds, just one prompt. It's super easy for Codex to implement it. So the thing is that this whole repository and this whole project is a very useful companion for my own work.
- 12:18
But as I said, it implements tons of features and state-of-the-art context management and memory management techniques that I believe AI engineers need to know. And now Paul will dive into all this three-layer system that we built with the raw content, the index that I mentioned, and the wiki-like, uh, synthesized version of all your notes, all your research,
- 12:41
all your work. So he'll cover everything we did, how it ended up, and show how to use it.
- 12:48
Okay, so now I want to go over the three versions of our system and how it progressed over time, and most importantly, why we added more complexity. So in the first version, we wanted to scope it just to create lessons for our agent engineering course.
- 13:05
So, uh, we wanted to keep it super simple, where we had as input a topic and a research MD as output. So within the input, we had the topic plus a set of golden links which were manually handpicked by us.
- 13:18
We applied this deep research algorithm, and we had as output a static research MD file. And if we go and zoom into the architecture, we first scraped the links of the golden links, right?
- 13:32
Because we already know them, and we use them as seed for context for the deep research algorithm, which was a really powerful technique because we had more context on how to frame our questions.
- 13:45
And during the query rounds, we basically used a very classic deep research algorithm where we had one main agent, the orchestrator, which created multiple questions based on the initial topic and the scraped context.
- 14:01
And each agent manage its own question and use Gemini, drawn it in Google to, to query basically Google and gather multiple resources. And each agen-agent gathers these resources, which return multiple links and create these, uh, executive summaries of each link.
- 14:20
And then it passed all this information back to the agent, to the main agent, where the main agent basically aggregated all this information into a, a summarized way, so it did not explode in the context.
- 14:34
And we did this for three rounds. So basically, after three rounds of generating six queries per round, we ended up with like forty, fifty links in total. So you can imagine that there's a lot of noise over there.
- 14:50
So that's why we also applied a ranking algorithm where we wanted to, like, find the, the information with the highest signal. And basically, we compared each source against the topic, the initial topic of the user.
- 15:05
And like that, we fully scraped only the top K elements based on the ranking score. And for the rest of the, of the links, we just kept the summaries.
- 15:16
And then we compiled everything into this research MD file as a single flat file, which we used for each lesson of our course in, in our particular use case.
- 15:27
But as you can imagine, it was pretty limited. For the course, it worked great, right? We, we generated thirty-five lessons really quick, but we wanted more. So we started to aim this deep research loop
- 15:42
to the second brain as well, right? Before, it was targeting only the public web, which made this, uh, pretty generic, and we had to manually find all those golden links.
- 15:55
So by aiming this deep research loop to the second brain, where we basically organically keep track of all the information that-- of all the research that we really want and is filtered by us, we can organically gather all those golden links into our deep research algorithm.
- 16:17
So let's look at how this new algorithm looks like. It's basically the same loop, right? But now we target our own sources instead of just the public web. Now, for input, we have only the topic because we don't need the golden links.
- 16:32
As I said, the golden links are actually a reflection of our second brain system. In theory, you can also add them if you really want to, but that's the beauty of this new strategy, because you can just put as input so-some topic, and it will find everything that it needs.
- 16:49
And then we use this topic only as seed for, for, for, uh, the context to generate the, the queries. And now we do the same deep research algorithm, right?
- 16:59
The same query rounds, but instead of targeting only the public web, now we plug in all our second brains, such as our Obsidian, our Readwise, our NotebookLM, our GitHub.
- 17:11
You can also use, for example, Gemini deep research for this, like similar to how we, we use NotebookLM. Or you can extend this with whatever you want. For example, YouTube, uh, Google Drive, Notion, or whatever makes sense on your infrastructure.
- 17:26
The idea is that now we, uh, target our queries from the deep research algorithm to our second brain, plus the public web. And after we apply the same algorithm such as ranking, fully scraping summaries, and compile everything into this research MD file.
- 17:45
But now we have another problem, right? This, this, this research MD file is static. It's a pile of static data, and usually research is not static, right? So after you end up with, with this file, you most often realize That you want to ask another question or some information is tell and you don't need it anymore, or,
- 18:06
or basically you want more out of this research MD file and whi-which means that you need to start all of this from scratch. And the operation that I showed you, uh, above is an expensive op-operation.
- 18:18
It consumes a lot of tokens, and it takes a lot of time, so you don't want to run it from scratch. And that's why you need to add a wiki layer on top of it.
- 18:28
And that's why V3 of this system is actually a deep research algorithm plus an LLM knowledge base on top of it, AKA, the wiki layer. So the new algorithm looks like this.
- 18:41
So we have sources in and a wiki out, and the sources, as I said before, can be like Obsidian, NotebookLM, Google Drive or YouTube, Notion, even custom URLs, right?
- 18:53
That, that's also powerful as well, where you use tools such as a Bright Data to, to parse basically any single page application, any type of site, any type of public information that's out there, we can put it in, and then you apply the same deep research algorithm.
- 19:09
You store everything into raw files, right? Instead of compiling everything into a research MD file, now we store each file individually,
- 19:19
and we create an index out of all these files, and ultimately, we generate a wiki on top of it, and which we can query. We can query basically the wiki plus the index.
- 19:31
Okay, so this is just the high level architecture of the new system. Let's zoom into it. So what I want to start with is that you should forget the infrastructure you think you need, such as vector databases, knowledge graphs, semantic search, text search.
- 19:47
All that is beautiful but adds a lot of complexity, especially for, like, these personal wikis, personal research operating systems that you want to use very lightly. So I want a system just based on files, right?
- 20:01
A simple mechanism that's, that's very rooted into how your computer works. And that's why we'll create all this system just based on files and just based on references. So no database, just a simple index based on references.
- 20:17
And how, how this works. We have an agent that reads an index.yaml file that's basically a catalog of all your data, plus the summaries of, of, of each source and some metadata around it.
- 20:30
For example, here on the right, you, you can see part of an index.yaml file that contains ten sources and 38 wiki pages as derivatives of the sources, where, uh, we can see there into the sources list of the YAML file, the first, uh, source, for example.
- 20:48
And as you can see, it has, like, the, the, the link to the original file plus some metadata, such the origin, the title, the authors, the, the, the publication date, the summary and, and things, things like this, which can be flexible, right?
- 21:01
And the next step is that based on this index.yaml file, we need to point to all the wiki pages, to all the wiki derivatives, to all the raw sources.
- 21:11
So basically, this index.yaml file is an entry point for our agent, right? It's what we will give to our agent to actually reason on how to find our data.
- 21:23
It's an index ultimately, right? So the next step is to understand how the wiki actually looks like. So on the left, you can see the high level structure of the wiki, where we have the raw folder, the wiki folder and the index.
- 21:38
In the raw folder, we actually just have the raw data, which is immutable. You don't want to touch that. The-- and the index points to everything that we need.
- 21:46
And in the wiki, we actually have derivatives created by the LLM, which contains things such as comparisons between multiple concepts, entities, or just simple notes as a reflection of our questions or repositories that we ingested, and we can create multiple notes based on, on, on a repository, right?
- 22:05
Or open questions that based on our questions that LLM couldn't answer yet, and everything that you can analyze on top of your raw data. And on the right, based on Obsidian, we can see, like, the subgraph reflected just based on in this index.
- 22:23
And this is just like the first iteration, but as the, the, the, the wiki grows, you can see connections made between entities and concepts. For example, concepts are things such as tool registry, context compaction, sandboxing, or entities are open code, Claude Code, MCP, right?
- 22:42
So as you can see, you can beautifully can start visually and practically create connections. Now, the next obvious question is how do we actually query this wiki? So as I said, the agent will have as input this index.yaml file, which contains summaries and metadata about each source.
- 23:04
But what happens next, right? The next step is actually to look into the source wiki page, where the source wiki page is like an executive summary of each page, which is basically not just a summary, but a more expanded summary of, of each source.
- 23:21
And sometimes the agent just looks into this, gets what it needs and goes back, which is very token efficient, right? And if it doesn't find within this, uh, source wiki page, we also need links into the wiki derivative, such as concepts, entities, notes, comparisons, and so on and so forth.
- 23:41
And only if it doesn't find the necessary information up to this point, it needs and it actually reads the whole raw source, right? Which basically contains the whole article, the whole paper, the, the, the whole video or whatever.
- 23:55
And this makes just through pure referencing and creating this simple hierarchy, this makes everything very token efficient. Now, the beautiful part is that this wiki is actually alive, right?
- 24:07
For example, every question leaves a trace into your wiki. So for every question, the LLM can create a new concept file, a new note file, a new comparison file, and every question is, is tracked into a log.
- 24:21
So basically, the, the, the wiki doesn't evolve only when you ingest new data or do a deep research round. It actually evolves as you start talking with it, right?
- 24:31
That's the beautiful part, actually. And like that, you can see a true reflection of, of yourself, of what you haven't understood, of all your questions from the past. And the beautiful part is that the, the wiki is never frozen, right?
- 24:46
Similar to the research MD files. At any point, you can ingest a new custom link that you think that you need into the wiki, or even run a new deep research round.
- 24:55
Or, as I said previously, the wiki keeps evolving just purely based on your questions. And another important thing to understand is that this wiki doesn't sit on top of, of your entire second brain, right?
- 25:09
For example, in my particular use case, I use the PARA method coined by Tiago Forte, where all my data is structured between project areas, resources, and archive, where all my notes, uh, resources that I save, sources that I save, PDFs, article or whatever, are just piped directly into the resource, a flat list.
- 25:30
And whenever I need something, I just reference them into projects and areas, right? And like this, Obsidian is just an immutable snapshot that the LLM never touches, right? So this is my data.
- 25:44
I don't really want the LLM to touch my personal notes that I manually write, right? So then how can we actually put this wiki to use, right? So as I said, we have the big Obsidian snapshot, which is our global second brain.
- 26:02
And then whenever we start to work on a new project, we reference this second brain through this deep research algorithm that I explained, and we scope it down to our own project, right?
- 26:15
So basically, whenever we want to start working on something, we run this deep research loop, or we start ingesting some particular repositories, articles, notes, and so on and so forth.
- 26:26
And we usually do that through a set of skills plugged into a harness. And a project can be basically anything, such as writing a new article, doing a new video, doing a set of slides.
- 26:38
I, I applied this technique doing this slide. Or y- you can even apply it for something more complex, such as writing a book, doing a course, or, or keeping track of a whole code base, right?
- 26:50
You can also u- use it for that. So basically, a project can be anything where you want, as I said initially, to transform research into work. The project is the work, and your second brain is the research.
- 27:04
So now I want to show you a few demos. So what you need to do is go to the AI Research OS workshop repository, and here you can find all the skills required to run what we presented into this presentation.
- 27:17
And everything is packed as a Claude Code plugin, but you can very easily tweak it and install it with any other harness. And also in the README, you can find details on how to install all, all the other dependencies, because the thing is that this system is dependent on tools such as Obsidian, Readwise, NotebookLM, and so on
- 27:34
and so forth. So you need to set up specific CLIs or authentification issues. But I don't really want to waste any of your time with setup issues, and I want to go straight directly into the examples.
- 27:45
So I prepared here three examples. The first one is a research on one of my previous articles on agentic engineering. And within these files, I have a brain dump of everything that I knew I wanted to talk on this subject.
- 28:00
And on top of that, I also added a few references that I knew one hundred percent that I want to add into this wiki. And what we need to do to actually trigger the, the algorithm on top of these files is to open up a Claude session and then just call the skill, the research skill, and pointed
- 28:19
it to this file, and that's it. Everything else is baked directly into the skill. It will understand my intent that I want to create a wiki on agentic harness engineering, just looking at these files and looking at the topic.
- 28:33
And it will know that before starting the deep research algorithm, it actually needs to scrape this information to, to use it as context when it frames the questions for the deep research algorithm.
- 28:45
Now, we need to wait a bit for, for the agent to reason on top of it, and I will actually just put it on auto mode to speed, speed this up, right?
- 28:54
I use this hundreds of times, so I know it won't delete anything [laughs] from my computer, or it won't do anything weird. Okay, so now this is the most important part, right?
- 29:04
So it asks me how deep I want the deep research algorithm to be. We have light, deep, fast. This mostly controls how many questions you want to run per one round and how many rounds you want to run.
- 29:20
And usually light or fast is more than enough because remember, this process consumes a lot of tokens, so you kind of need to do the deep one only when you, you really want to look over tons and tons o- of notes.
- 29:35
And for this use case, I will just pick the light one to, to, to speed this up. And in this use case, it just does one round of three queries, right?
- 29:44
And for the fast one, it does two rounds of three queries. So I would just keep it, keep it around that spectrum. And now the process will, will take around ten to twenty minutes to actually look around my Obsidian, to look around my Readwise, my NotebookLM, and run those queries on top of this.
- 30:06
And I actually run this, right? And now let's open this wiki that we created based on the prompt before in Obsidian, and let's look what's inside the, the wiki.
- 30:18
So we have three big objects, the raw files, which are basically a raw copy of what we found, the index, which contains all the references to- towards the wiki, right?
- 30:31
We can also in Obsidian have this beautiful, uh, subgraph where we can very quickly understand what is going on. This is created purely based on this, this file. And then the most interesting part is inside the wiki, where we have comparisons, concepts, entities, and sources.
- 30:52
The sources contain the executive summary of, of our raw sources, so the LLM doesn't really need to every time when it reads them, and it needs them to, to read the raw sources, but it needs just the executive summaries, which are computed just one time during the ingestion.
- 31:10
And for example, for the comparisons, it understood out of the box that it needs to do comparisons between like agentic RAG versus file systems or compaction versus recursive language models or, or, or anything of interest based on, on the sources.
- 31:25
And the most interesting part is actually the concepts. So it automatically extracted all the concepts that we need to understand from this pile of resources. For example, if we open the agent loop resources, we automatically
- 31:41
can look and get this beautiful summary containing like graphs, text, and explaining us everything that we need on this topic. And we can do the same on all the concepts from, from, from this wiki or on the entities from the wiki and so on and so forth.
- 31:58
Okay, so now let's go to the second example. It's, it's a simple example where I want to learn more on harness engineering and how harnesses work. So i- in this prompt over here, I just want to ingest the three open source repositories on OpenCode, PaI, and Hermes.
- 32:16
And I don't want to do deep research at all, right? I just want to ingest those repositories and explore topics such as the general architecture, agents architecture, sub-agents, memory system, the, and the agent permission flow.
- 32:31
And let's run the research on top of these prompts.
- 32:35
And now what the research will do will clone automatically all these repositories, and will explore all the repositories on the topics that I gave here. And it will create nodes at the individual level of each repository on how they work, on how, how the architecture works at the repository level.
- 32:55
And then we can create higher level nodes, right, within the wiki derivatives and compare other architectures or, or create a aggregate architectures on, on like the general trends o- of all those harnesses, and basically explore and learn everything that we want about harness engineering directly from the code.
- 33:15
And again, usually I just do auto mode and let it do its own gist, but I already run this, right? So here is the wiki for these GitHub repositories.
- 33:26
And again, we have the raw files, and we have the index and the wiki.
- 33:33
And here probably within the repos, we can see all the three repositories. And for example, inside the OpenCode repositories, we can see empty files explaining everything that we need on particular topics, such as the permission flow, the memory system, and so on and so forth.
- 33:50
And we can see this in all the other repositories. And the most i- interesting part here is actually that we have comparisons on all of these, right? So we can actually understand the differences within the architecture within these harnesses.
- 34:06
Or we also have all the concepts extracted from these repositories, and we can understand what are the key architectural decisions from these repositories. So we can go crazy with this, and this is super useful if you want to, for example, write your own harness.
- 34:25
And the third examples is the simplest one in, in reality, which is just based on ingesting some, some simple links, right? And again, I will just exit the, the previous run, and
- 34:38
I want to run this example from scratch. And here
- 34:43
I just want to show you that you can use this also with a very basic setup where I just want to ingest three custom random links. And as before, I just paste this prompt, and it will start the research process.
- 34:59
And I want to highlight that you can run example two and in, on the GitHub repositories and this example without any other setup, like without setting up Obsidian, Readwise, or anything else.
- 35:10
You can just install this plugin and run these examples because it here is not dependent on, on any other service than Git and using curl to get these, these URLs.
- 35:22
And again, here, if we go into Obsidian and explore the wiki created out of this example three, we, we can see the index, we can see the wiki itself with all the sources, right?
- 35:35
One executive summary for, for each source and all the concepts, extracted entities, and so on and so forth. And the idea is that as you start asking questions on top of this, everything starts to get more interesting.
- 35:48
So now let's assume that we want to ask, for example, a question on harness engineering based on the wiki created out of the GitHub repositories. So what we have to do is just again hit the research wheel, point it to the
- 36:04
wiki that we just created, and then just ask our question. And let's assume that I want to learn more on sandboxing
- 36:15
more exactly how remote sandboxing works and how it's plugged into the heart.
- 36:27
This can be basically any, any, any other question. And now what it will happen, it will basically query this wiki. It will give you an answer, and based on this, you can also start creating notes, comparisons, or maybe it will extract and find new entities that you care about.
- 36:46
Basically, it will start updating the wiki. All right, so now where is this project going? What is it? What should you do? First, it's still rough at some points, like it needs more connectors, as you saw.
- 36:59
We need to add Google Drive, Notion, Slack, and tons of other connectors that could be useful to you. But to be honest, it's not really useful to me and my current workflow or to Paul, so we didn't add them yet because the core of this project is to be useful for us and for you to take over
- 37:14
and add whatever you need. And the other main goal of this project is to teach memory and context management. So all these extra features aren't really useful towards that purpose.
- 37:25
There are other some weaknesses, like it's hard to know which sources are outdated or weak or strong compared to some other system that we built. So we know we can improve this, but again, it's not really the priority here.
- 37:39
And lastly, it's still obviously a builder workflow. You use it through Claude Code and Codex, and it's just, uh, to me in the terminal, and I just really like it.
- 37:48
So it's not a final polished product with a nice UI, nice UX, and, uh, honestly, that's by design. So we don't really care about this because our goal is to teach AI engineering.
- 37:58
It's not to build the next best product. Still, we have a few next improvements we want to do very shortly, from having a stronger linting to a better memory compaction, because that's a big issue, and it's just very complicated in general to manage memory correctly, and the state-of-the-art is always progressing there.
- 38:18
We, as I said, need better source provenance to trust the sources and be able to rank them properly and reuse them properly if needed, and be able to know as, as quickly as a user if this source is relevant or not.
- 38:33
And we have other next improvements to do. But those are mostly for optimization and for the future. And the thing is that we actually built all of that into another product that you can even build yourself, because we created a course called Agent Engineering, where we built a similar deep research system with a writing and research agent,
- 38:58
where we built a system with the same goal, to be able to learn best AI engineering practices. It's a very in-depth course where I assume it takes around sixty hours to complete, with a final project being the multi-agent system I just described, that you'll build for yourself.
- 39:14
So if this presentation and the demo repo that you saw was interesting, please consider checking out the Towards AI Academy with our courses on there, including the Agent Engineering course, to learn more on the best practices when building around and with agents.