Self-Training Agents: Hermes Agent, HF Traces, Skills, MCP & Finetuning — Merve Noyan, Hugging Face

An assistant that can write code is useful. What would let it also choose a model, adapt it to your data and run the infrastructure? Merve Noyan, introducing herself as a member of Hugging Face’s open source team, begins with the foundation: access to the models and the systems that run them.

Openness comes in layers. Noyan distinguishes downloadable weights with noncommercial restrictions from commercially usable releases, citing DeepSeek and licenses such as MIT and Apache 2.0. Beyond the weights, a project can expose its code and an agent’s harness—the software that manages its execution. Commercial permission is one licensing dimension, rather than a complete definition of open-source AI.

Noyan invokes reports of declining Claude performance to explain why visibility matters: when you control the model and harness, you can inspect and manage changes. Weight access also lets you shrink, quantize or fine-tune a model. Deploying it on an edge device or in a browser can keep inference data local, provided the surrounding application and tools do not send that data elsewhere. Control over deployment makes privacy possible; the full deployment determines whether it holds.

Control would be less attractive if it required giving up too much capability. Noyan uses GLM 5.1 in her coding setup and points to an Artificial Analysis Intelligence Index chart with open models in green and closed models in black. Her reading of that displayed comparison is that open models have caught up; it is a talk-time comparison, not a claim about today’s changing leaderboard.

The Hugging Face Hub supplies the infrastructure for discovering and distributing those releases, alongside datasets and Spaces. Noyan estimates that the Hub hosts close to three million models at the time of the talk. Filtering for agentic models narrows that inventory. Within it, vision-language models add a useful capability: interpreting screenshots and locating where to click, which lets an agent operate a graphical interface.

Gemma 4 is her example of an omni model that is also agentic; Qwen 3.5 and Kimi K2.5 provide further examples of vision capabilities arriving with a model’s initial release. She expects that pattern to spread. Serving these models is a separate choice: vLLM, MLX and llama.cpp’s llama-server provide local runtime options.

For a compatible Hub repository, the shell command can be as small as this:

: "${HF_MODEL_ID:?Set HF_MODEL_ID to a llama.cpp-compatible Hub model ID}"
llama-server -hf "$HF_MODEL_ID"

The runtime still has to support the selected model and fit the available hardware. The useful change is that downloading and serving a supported model no longer requires building a serving stack yourself.

The Hub’s Benchmark Datasets filter offers a more focused starting point than browsing trending models. In the datasets sidebar, select the benchmark filter, then open a benchmark suited to the task. Noyan names SWE-bench Pro, Humanity’s Last Exam and AIME. For coding, she opens a SWE-bench comparison and reports GLM 5.1 at the top of the displayed open-model list.

After using a benchmark to make a shortlist, try the model on your own task. Hugging Face Inference Providers offers hosted access through providers including Groq, Cerebras and Novita. Its selection interface exposes cost and speed options, while a Tool Use column helps identify models suitable for an agent that must call tools.

These answer different questions: a strong benchmark result does not choose the serving provider or establish tool compatibility for you.

With a model selected, the next layer is the agent’s access to the ecosystem. The Hugging Face MCP server connects an assistant to Hub capabilities. Skills supply procedures for tasks such as training: instead of manually assembling every step, a user can ask an agent to train Qwen 3.5 on a dataset. Local coding agents provide another entry point, using models distributed through the Hub.

Noyan favors Pi for its simple setup. In the local arrangement she describes, llama.cpp serves the model and Pi consumes that endpoint. She also suggests Inference Providers as a possible remote route for Pi, but does not establish that integration in the demonstration. Another option is llama-agent, which she describes as a llama.cpp binary that starts an agent from a Hugging Face model ID; the local-agent documentation is the practical reference for that path.

Hermes Agent adds a persistent assistant around those model choices. Noyan prefers its memory management to OpenClaw’s. Here, learning through saved memory and reusable skills is distinct from fine-tuning: retaining knowledge or a procedure does not itself update model weights.

Hermes can use a local model or Hugging Face Inference Providers. Its setup wizard collects credentials and configures integrations such as Slack or WhatsApp. Noyan recommends GLM 5.1 based on her own experience: after she failed to get a Slack integration working, she asked Hermes running that model to repair it, and reports that it succeeded.

Her next experiments were still prospective: trying Gemma 4 and possibly a forthcoming MiniMax model she had seen rumored on Twitter. Those possibilities sit alongside the demonstrated setup experience, rather than serving as additional results.

An agent session also produces data. The Hub’s Agent Traces feature accepts sessions from Codex, Claude Code and Pi. Noyan introduces it as a traces repository type; current documentation describes traces stored in datasets or buckets with a dedicated viewer. In her walkthrough, clicking the traces column in Dataset Viewer opens a parsed session that can be explored rather than read as an undifferentiated log.

Those sessions can later become inputs to model training, but uploading them does not automatically train anything. The workflow starts with saved session files, followed by inspection and an optional training step. Review and redact secrets, private code and personal data before publishing traces. Hermes trace support is something Noyan expects to arrive, not something she demonstrates as available.

For local inference, start with the application you intend to use. On the Hub, Other → Apps filters models by support in tools such as LM Studio, Jan and llama.cpp. That reverses the usual search: instead of discovering a model and then asking whether it runs in your setup, first restrict the inventory to compatible models.

The model repository then exposes format and hardware information. GGUF, associated with llama.cpp, is supported by applications including Ollama and LM Studio. Noyan says a larger Gemma 4 model quantized to four bits fits an L4 GPU with 24 GB of VRAM. The example does not specify the variant, context length, batch size or runtime overhead, so it is not a complete deployment memory budget.

Hardware compatibility information also appears for MLX repositories. Once a suitable model is selected, Use This Model lists supported local applications and provides installation or serving commands. The sequence is straightforward: filter for your application, inspect hardware fit, then obtain the command for that model and runtime.

Hugging Face Skills give coding agents procedures for operating across the ecosystem. The CLI skill handles repositories, jobs and demos. Other skills specialize in the work performed on those resources:

Claude and Gemini appear as integration examples, not as an exhaustive list of supported agents.

Noyan’s concrete request to Claude Code is to train Qwen2-VL on LLaVA-InstructMix, a vision-language dataset. The assistant does not go straight from that sentence to an opaque training run. It turns the request into configuration decisions:

1. Ask which compute instance to use.
2. Estimate the VRAM needed for the model and proposed batch size.
3. Ask for the validation split and other training choices.
4. Launch the training job after resolving those choices.
5. Make the resulting model available on the Hub.

The engineering work has not disappeared: the agent performs the resource estimation and job setup, while the user supplies choices that affect the run.

The same approach extends beyond language models. Noyan describes skills for training object detectors and Segment Anything Model, including handling different bounding-box formats. That data-format work matters because a training request must ultimately become inputs the chosen model and trainer can consume.

MCP exposes another route into the same infrastructure: search for models, datasets and Spaces, including semantic search for Spaces. Noyan describes Spaces as an AI app store. Jobs add one-off compute: a job ends when it succeeds or fails, and billing follows its running time. An assistant can therefore discover a resource, invoke an application or launch work through tools connected to the Hub.

The demonstration makes this concrete with a request: “Generate image of a baklava made of yarn.” The assistant calls a remotely hosted Qwen-Image Space and returns the generated output. To broaden discovery beyond the initially configured Spaces, Noyan enables Dynamic Spaces, which she describes as experimental. Its scope is MCP-compatible Spaces, rather than every application hosted on the Hub.

The final workflow begins with a missing input. Hugging Face hosts paper pages that people should be able to ask questions about and share, but not every paper has Markdown suitable for indexing. Noyan introduces a project by her colleague Nils that fills that gap with OCR.

Noyan reports OCR processing of 30,000 papers using Codex, open OCR models and Hugging Face Jobs through prompting. The workflow combines the capabilities introduced earlier: choose an inexpensive, capable OCR model; ask the agent to write the processing code; launch it on Hugging Face infrastructure; and use a skill to arrange the model-serving instance.

For model selection, she returns to Benchmark Datasets and opens olmOCR-Bench, where Chandra OCR appears first in the displayed results. A leading score is only one input to the choice. She also describes a newly released skill that recommends OCR models for fine-tuning, including smaller alternatives when those better suit the task.

This project uses Chandra. The agent writes the processing script, estimates the required instance and calculates the expected cost of running the job. That connects model selection to an operational plan: the output is code and a way to run it on provisioned compute, not merely a recommendation in chat.

The processing jobs are intended to run again, which makes storage part of the design. Noyan closes with Hugging Face Buckets: mountable storage, compared with S3, that the jobs can use across repeated runs. She describes it as cheaper and faster without supplying comparison conditions or measurements. The substantive endpoint is a repeatable workflow—an OCR model, generated processing code, managed jobs and mounted storage—followed by links for getting started.