Turning My Obsidian Vault Into a Local AI Engineer — Filip Makraduli, Superlinked

An Obsidian graph can organize Markdown files and show their connections. It does less to answer a practical question: what happens when a coding agent needs to work with sensitive documents? Filip Makraduli of Superlinked starts with that problem. Giving Claude Code or Cowork access to a vault can expose files, while processing scans and images through the main model can consume substantial context and tokens.

The proposed workflow preprocesses documents on infrastructure the operator controls, then returns a smaller artifact to the agent. A scan can become extracted text; sensitive entities can become masked labels. The main model receives material prepared for its next task instead of every page and image in its original form. Smaller artifacts also leave more room in context for the work the agent actually needs to do.

Three requirements shape the design:

• Lower token use. Move document processing, particularly OCR, away from the main agent model.
• Control over sensitive processing. Run preprocessing in a private cloud or self-hosted environment.
• Adaptable models. Test open models and task-specific LoRAs rather than committing every operation to one general model. LoRAs provide a way to adapt models to narrower domains.

Storage was the comparatively easy part. Obsidian already held the files and their relationships. Running several open models, comparing them and deploying inference at production scale created the harder problem. Superlinked’s open-source SIE project supplies both the inference system and cluster deployment. The workshop uses a Superlinked-managed cluster; a local Docker setup is another option, with slower processing expected in that setup. Here, “local” therefore describes a workflow that can remain under the operator’s control, rather than a requirement that every model run on the laptop.

The intended user experience stays inside Claude Code. When a task needs sensitive or token-heavy processing, the agent calls a tool and receives its result. Consider the talk’s example of a ten-page PDF containing images and other details: an open OCR model processes the document first, and the agent receives the extracted artifact. The work still happens, but a specialized model performs it before the main agent consumes the result.

MCP provides the interface between that familiar agent workflow and a separately operated cluster. Makraduli chose it to separate responsibilities and support additional tools over time. SIE controls the worker pools, GPU choices and model placement behind the interface. A skill can help the agent use the integration, while the server and workers perform the heavy processing.

Where does a tool request go, and where does inference happen? The three-tier layout separates the agent host, the SIE MCP server and the GPU cluster. Requests travel through the MCP server to the workers; results return through the same integration. In the self-hosted design, the MCP service and inference work run in the operator’s private cloud. That placement describes the processing path; the closing discussion will return to whether the agent can also reach raw files through another path.

The integration uses JSON-RPC requests, responses and notifications for initialization and communication. Tools become available through discovery. MCP can also support client capabilities such as sampling and elicitation, but this implementation deliberately avoids those additional exchanges: the SIE MCP service exposes tools and responds to their calls. That is a narrower interaction model chosen for the privacy-oriented setup.

For transport, the talk distinguishes local stdio use from the streaming HTTP approach chosen for the service. Makraduli describes the newer HTTP arrangement as consolidating communication that previously used separate endpoints for the handshake and streaming. The operational purpose is straightforward: expose remotely hosted processing through a tool interface the agent already understands.

The available tools cover several distinct transformations:

• Documents to Markdown. OCR extracts a text artifact from document inputs, including scans.
• Entity extraction. Named entity recognition models identify relevant entities in documents.
• Captioning and tagging. Models produce descriptions or labels for content.
• Structured and generative tasks. Other tools support structure generation, summarization and question answering.

Makraduli presents open OCR models as strong enough to make this offloading useful without an inherent quality sacrifice. That is his assessment of the available models, rather than a document-level accuracy measurement from this demo.

The tool layer is extensible: a contributor can add another tool while using the same inference infrastructure underneath it. Setup can use a locally launched cluster, a self-hosted cluster or a managed endpoint with an API key. The workshop proceeds with credentials and an endpoint already configured, so the live exercise focuses on the document transformations rather than provisioning hardware.

The demo vault contains an inbox of synthetic files standing in for emails and documents. Examples include invoice-related messages and an NDA containing sensitive fields such as a phone number and Social Security number. The request is ordinary language: “redact the data from the NDA document.” This gives the agent a task to delegate through the configured integration.

What does the agent receive after redaction? Makraduli first shows a previously produced example with masked entities and labels such as email and person, then describes the current run reporting a freshly redacted file, redacted spans and label counts. These are different parts of the result: the artifact carries the usable document, while spans and counts describe the redaction operation. The intended behavior is for the agent to receive this processed version rather than sensitive values from the original.

The next request adds an earlier processing stage: preprocess the scans and redact them. A scanned contract is an image, so its words first need to be extracted through OCR. The stated sequence is to convert the document to Markdown, then redact the extracted content. OCR changes the representation into text the next tool can process; redaction changes which values the resulting artifact reveals.

What survives that two-stage transformation? The before-and-after example moves from the scanned contract image to a Markdown file containing extracted material with sensitive content redacted. That is the useful result for the downstream agent: a text document it can work with, with masking applied before the intended handoff. The example demonstrates the workflow’s form, rather than establishing that every sensitive field across arbitrary scans will be detected.

A status line also attempts to count saved tokens. Makraduli says that counter still needs updating to work properly, so it does not establish a measured saving for this run. The supported cost mechanism is the preprocessing itself: the main agent can consume the resulting text artifact instead of processing the original image-heavy input.

Behind the simple tool call, SIE must serve different models without treating every task as the same GPU workload. A gateway connects work to different worker pools, and batching happens per worker rather than entirely upfront. Makraduli reports better performance from this arrangement. His explanation also refers to a shared queue workers pull from, though it does not fully specify the queue and batching topology; the clear design distinction is that batch formation happens at the workers.

Separate worker pools let hardware follow the workload. A heavier model may need more expensive hardware, while another can run well on an L4 spot instance. Sending both to the expensive option wastes that distinction. Autoscaling can reduce capacity when it is unused, adding another way to control the cost of keeping the service available.

Model hot swapping supports another kind of flexibility. Smaller models for named entity recognition, embeddings or generation can share GPU capacity through switching and swapping. The talk gives the example of five LoRAs fine-tuned for different subareas of work: being able to swap among them makes it practical to compare their behavior. The same infrastructure can support task bundles for document parsing, embeddings, code generation, captions or policy checks.

How close did the tested embedding models come to the comparison model, and at what price? Makraduli reports roughly 10× lower price and says the best embedding model per task reached 99.8% of the frontier comparison’s quality on MTEB tasks. These are reported embedding results, not measurements of the OCR, redaction or generation demo. The recording does not supply the full pricing conditions or benchmark protocol needed to reproduce the comparison.

The architectural lesson is about end-to-end performance. Queueing, batching and hardware placement affect how long a job takes and what serving it costs, even when the model’s individual inference operation has not changed. SIE works on both infrastructure and per-model optimization, but Makraduli emphasizes that the surrounding system alone can materially improve production behavior under suitable loads.

Task adaptation provides a separate benefit. Colleagues experimented with relatively inexpensive German-language LoRAs, with German legal language offered as a niche use case where performance improved. The operational advantage is the ability to fine-tune, deploy and compare several adaptations quickly. No numerical improvement is supplied for that example.

Deployment extends beyond publishing model-serving code. The project includes Helm charts, Kubernetes infrastructure and Terraform-based cluster deployment. The same MCP service can handle document questions, fact extraction and reading tasks, while the deployment materials make it possible to operate the processing infrastructure independently.

The demonstrated system calls tools backed by configured models. A proposed next step would also choose the model underneath each tool. An OCR request could express a preference such as cheap processing, allowing the system to select an appropriate model for that task. Makraduli explicitly places this routing capability on the roadmap rather than presenting it as built-in behavior.

Routing would have two kinds of inputs: the task’s requirements and the cluster’s current state. A model already warm on a GPU can respond sooner than one that must be loaded. A model with a higher nominal cost might therefore be cheaper for a particular request than a supposedly cheaper model that has been evicted and needs reloading. The proposed decision compares the cost of completing the request, including model availability, rather than comparing model prices alone.

Reasoning introduces another task-specific decision. While building a summarization tool, Makraduli encountered a model that spent its available generation on thinking tokens, leaving a very short summary or no summary at all. The failure was visible in the missing deliverable: computation happened, but useful output did not. Model selection and the thinking setting therefore need to fit the operation. Reasoning may help some tasks, while consuming resources without producing the required artifact in others.

Councils of models and model-based judges are also future possibilities, with corresponding evaluation problems. A judge can be biased; an optimization process can learn to exploit its scoring target; a reasoning model can occupy GPU time without finishing useful work. Testing models before routing work to them matters because more inference activity does not by itself imply a better result.

A shared interface does not make the underlying models interchangeable at runtime. Different model families can implement attention differently and require different execution paths for their forward passes. The catalog therefore distinguishes architectures and output types, while the inference system supports the runtimes those families need.

The catalog also distinguishes encoding from scoring. Encoding models support embedding-based retrieval; scoring models can rerank candidates. Exposing those capabilities through tools opens a path to custom retrieval workflows alongside generation. Makraduli points to examples, quick-start material, credential setup and migration guidance for people moving an existing API-based workflow onto SIE. These materials support trying the infrastructure with a real task rather than treating the document demo as its only application.

The final questions test the privacy premise at the moment before the tool call. An audience member asks whether Claude could read the raw document while deciding what to do. Returning only a redacted version answers what comes back from the processing service. It does not answer what the agent could read beforehand through its existing file access.

Makraduli says his tests appeared to work without leaking data, while acknowledging the need for further evaluation and stronger guardrails at that initial stage. The consequential limitation is that the demonstrated processing path does not establish an enforced restriction against raw-file reads. Hosting OCR and redaction privately controls where those tools run; it does not, by itself, remove another reader’s access to the source document.

Two possible changes emerge from the discussion:

• Control the harness. A custom harness using an open model could place more of the agent workflow under the operator’s control. The preprocessing benefits can still apply without Claude Code.
• Separate file access. Another audience member proposes advertising a file’s existence to the worker without connecting the agent to the same filesystem. SIE would retrieve the file for processing. This separates knowing which file needs work from being able to read its contents; it is a proposed arrangement in the discussion, rather than a demonstrated restriction in the live setup.

The remaining questions clarify ownership and ingestion. The demo uses hardware hosted by Superlinked, but users can configure and deploy their own cluster through the repository’s Terraform infrastructure. Asked whether the MCP integration has a file watcher, Makraduli answers that it sends the file to the main worker, where processing occurs; he does not establish a watcher mechanism. Managed compute is offered to make experimentation quicker, while self-hosting remains an option.

For sensitive use, the useful next test follows directly from the audience’s objection: can the agent reach the original document before preprocessing finishes? The scanned-contract workflow supplies a smaller, masked artifact for downstream work. Making that artifact the agent’s only available document requires the file-access design to support the same intention as the tools.