AI Engineer World's Fair 2025
MCP Agent Fine-Tuning Workshop - Ronan McGovern
Read the talk
Fine-tuning a browser agent from MCP traces
A browser agent’s useful training data includes its tools, reasoning, actions and observations. Ronan McGovern follows that record from collection through Qwen3 adapter training and back to inference.
From a talk by Ronan McGovern
Before you start: Familiarity with Python, chat-model APIs and basic model fine-tuning will help you follow the agent and notebook examples.
What should a browser agent learn from a successful run?
How do you turn a successful browser-agent run into training data for a smaller model? The final answer alone leaves out much of the work: choosing a tool, supplying its arguments, interpreting the result and deciding what to do next. Ronan McGovern’s workshop follows that complete sequence—collect good MCP agent traces, save their tool definitions and conversations, fine-tune Qwen3, then run the trained model to inspect its behavior.
The workshop materials live in the mcp-agent-fine-tune directory of Trelis Research’s AI Worlds Fair 2025 repository. The supplied task environment is browser navigation. MCP integrations for Stripe, GitHub or Gmail could replace it, but starting with the browser keeps the collection and training pipeline consistent while you learn it.
MCP connects tool discovery to tool execution. A server supplies descriptions and input schemas so the model can decide how to use its tools. When the model requests an action, the agent routes that request to the server. The server executes it and returns a result, sometimes with guidance for the next action. That observation enters the conversation, allowing the model to make another call or answer the user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translate between MCP, the API and model text
The model sits behind an OpenAI-compatible endpoint. This gives the agent a familiar request and response interface, but MCP schemas and generated model text still need translation at three boundaries.
| Boundary | Required translation |
|---|---|
| Tool discovery | MCP metadata → API JSON tool definitions |
| Model output | Generated tool-call text → structured call |
| Tool execution result | MCP response → conversation tool message |
The workshop’s Qwen model emits calls in Hermes format. A parser detects those calls in the generated tokens and exposes structured arguments through the API. The parser name describes the output convention; it does not require the model itself to be named Hermes.
The prompt makes this convention explicit. Its system portion introduces the available tools and instructs the model to put JSON function calls inside XML-style tool_call tags. The user then asks it to navigate to trelis.com. The assistant can produce reasoning followed by a browser call; after receiving the page, it can reason again and return ordinary answer text. A generated call is a request for execution, not an executed browser action. The surrounding agent must recognize it and carry it out.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Serve a teacher that exposes reasoning
After cloning the repository into Windsurf, McGovern opens the README’s data-collection section; fine-tuning comes later in a notebook. For collection he chooses Qwen3-30B-A3B, a mixture-of-experts teacher described in rounded terms as 30 billion total parameters with 3 billion active. The companion command selects its FP8 variant. He recommends keeping the teacher and student in the same family, although a different stronger model could supply demonstrations. The practical requirement here is access to the reasoning being trained on: OpenAI models do not expose their private thinking traces.
The demonstrated host is RunPod, using a linked affiliate one-click template and a vLLM Docker image. An H100 is one suggested GPU. Other hosting services can work, and local hosting depends on whether the serving application supports the required tool calls; McGovern leaves LM Studio support unchecked. The deployment settings each serve a distinct purpose:
- Reasoning extraction: enable reasoning and its parser so thinking tokens become a separate response field.
- Context budget: configure a maximum model length of 32,000.
- Tool selection: enable automatic tool choice so the model decides whether and which tool to call.
- Call parsing: select the Hermes parser to turn generated calls into structured API output.
The historical parser option is documented in vLLM’s versioned tool-calling guide. These are the workshop’s serving choices, rather than a version-independent deployment command.
Expose port 8000, apply the template overrides and start the server. McGovern estimates roughly three minutes for startup. The agent receives the model name and an OpenAI-compatible base URL containing the RunPod pod ID, exposed port and /v1 suffix. That connects the local agent workspace to the remotely served teacher.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Approve a navigation and read the page
Browser results can dominate the context. A navigation returns an accessibility tree: a structured text description of the page that the model can read. The agent’s --truncate option keeps only the beginning of that response. In the companion instructions, this limit is measured in characters, separately from the model’s token budget. It is a deliberately simple compromise: content farther down the page can disappear from the model’s view. A more complete agent needs a way to retrieve relevant content beyond that prefix.
Install uv if needed, then synchronize the project dependencies:
bash
pip install uv
uv sync
The agent starts the MCP servers listed in its configuration and loads their tools. The supplied Playwright MCP server provides browser navigation, history, tabs and link interactions. McGovern initially reports 25 tools. He estimates that open models may become confused beyond roughly 25–50 tools, while Claude might handle around 200; these are practical estimates, not measured capacity limits.
The first request is to navigate to trelis.com and read out the top two lines. Thinking tokens are hidden from the terminal to reduce clutter, but retained for logging. The model proposes navigation and waits for approval. McGovern keeps the browser visible so the audience can watch the action; --headless would run it in the background. After arranging the windows and submitting the request again, he approves the call. The browser opens, its accessibility structure returns to the model, and the assistant identifies both lines.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Save the whole run, then try a harder task
The agent logs two essential fields by default: messages and tools. The former preserves the conversation; the latter preserves the tool definitions available during that conversation. Here McGovern describes the saved inventory as 26 tools, including browser close and resize, rather than the 25 reported at startup. The companion configuration launches @playwright/mcp@latest, so it does not establish a fixed historical count.
The saved record contains the user request, assistant reasoning, navigation call, truncated page response, further reasoning and final answer. That is the behavior to preserve. McGovern keeps this successful trace for training a smaller model: the teacher already solved this example, whereas a less capable student may benefit from learning its sequence of decisions. The next collection step is to repeat the process and retain more good runs for upload to Hugging Face Hub.
The harder task is to find the third product on trelis.com, Training and Fine-Tuning, navigate to its page and report the feature section. McGovern first follows the route manually and notices that the product opens in a new tab. The agent then navigates, successfully opens the dropdown, but stops early with an answer and asks whether to click Learn More.
McGovern tells it to continue and explicitly reminds it to switch tabs. The agent clicks Learn More, opens the new tab and requests tab selection. He is unsure whether selecting that tab automatically obtains a fresh snapshot, but the subsequent answer retrieves the correct features: GitHub repository access and the ability to post issues. This is a successful guided run, with the corrective user turn forming part of the recorded conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn guided runs into curated demonstrations
One way to collect cleaner behavior is to supply a task-specific system prompt. It can explain which tools to use, how to reach the product and when to switch tabs. McGovern’s proposal is to use that guidance during collection, then omit the task-specific prompt from the training record while retaining the resulting trace. The goal is for the student to learn the demonstrated sequence without requiring the same detailed instructions each time.
He then opens the more complicated trace, leaving its tool definitions unchanged, and considers two editing approaches:
- Combine the interrupted sequence: remove the corrective user turn and combine the assistant sections, manually or with ChatGPT. Remove the intervening assistant answer content while retaining the reasoning needed for the continued action.
- Reduce the correction: retain a short instruction to proceed, but move the possible need to switch tabs into the assistant’s reasoning. This is the edit he demonstrates.
The resulting examples are curated demonstrations, including retrospective edits, rather than a collection consisting only of untouched autonomous successes. That distinction matters when interpreting what the student is being taught.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Unroll conversations into training prefixes
Uploading a trace means preserving both its conversation and its tool list. Unrolling also saves shorter conversation prefixes so intermediate responses can become training endpoints. McGovern illustrates a conversation with three back-and-forths:
| Training row | Conversation retained | Endpoint |
|---|---|---|
| Short prefix | First exchange | Earlier assistant response |
| Longer prefix | First two exchanges | Later assistant response |
| Full record | All three exchanges | Final assistant response |
His motivation includes reasoning retention: he describes the workshop template as including reasoning only from the most recent turn. Making an earlier response the endpoint lets that reasoning appear in its own training row.
This behavior depends on the uploader and template revisions. The current companion uploader adds prefixes at assistant messages immediately following user messages, plus the full conversation—not at every assistant tool step. The current Qwen3-4B chat template retains reasoning after the last real user query, which can span several assistant/tool steps. For a reproduction, inspect those boundaries together rather than assuming every tool call produces a row or that only one assistant message can retain reasoning.
The upload helper takes a repository ID and an optional unroll argument. Hugging Face authentication needs a token with write permissions. In the demonstrated upload, four collected examples become nine dataset rows: one contributes three rows and the other three contribute two each. The displayed dataset includes an ID, timestamp, model, messages, tools and a flag described as truncated or unrolled. Here truncation means shortening the conversation to a prefix; it is separate from shortening a browser’s accessibility response.
Inspecting the rows makes the difference concrete. A full record reaches the answer about the product page. A shorter record ends with reasoning and a tool call to select the tab. The reasoning remains in a separate field because the endpoint’s reasoning parser extracted it during collection. Training will later combine each row’s messages and tools through the student’s chat template.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the agent’s translation responsibilities explicit
The implementation walkthrough maps the visible browser behavior to a small set of runtime responsibilities. The agent configures its OpenAI-compatible client, boots the MCP services, discovers their tools and translates their schemas. An MCP tool’s description and input schema become an API function definition; the Python mapping has this shape:
python
def to_openai_tool(tool):
return {
"type": "function",
"function": {
"name": tool.name,
"description": tool.description or "",
"parameters": tool.inputSchema,
},
}
The reverse path takes a structured model call back to the appropriate MCP tool. Once execution finishes, the agent wraps the result as a conversation message the API can accept. These conversions are the code-level counterparts of the earlier architecture diagram.
The chat loop ties discovery, model requests, tool execution and subsequent model responses together. It keeps parsed reasoning separate from assistant answer content, while a trace-logging helper saves the details needed for training. Server configuration and command-line arguments control the environment without changing those core responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Template the records and test the untuned student
Fine-tuning moves to a modified Unsloth notebook. After the Colab-specific installation, McGovern loads Qwen3-4B in 16-bit precision, without 4-bit weight quantization. Larger models may exceed the free GPU’s capacity; his alternative is a RunPod fine-tuning template with Jupyter, where the notebook can be uploaded. He postpones attaching LoRA adapters so he can first inspect the untouched student’s behavior.
The notebook loads the dataset and inspects messages, tools and the tokenizer’s chat template. That template is the serialization contract: it turns structured records into the single sequence the model consumes. It is also available in the model’s tokenizer configuration on Hugging Face.
After the runtime restart and weight download, the records are formatted into a text field. The central operation is:
python
def format_record(row, tokenizer):
return {
"text": tokenizer.apply_chat_template(
row["messages"],
tools=row["tools"],
tokenize=False,
add_generation_prompt=False,
)
}
The output starts with system-level tool instructions and definitions, then includes the user messages, assistant messages, calls and tool responses. Those tool definitions and page observations make the sequence long. Before training, the notebook checks the longest formatted row against its configured maximum length of 32,000.
The baseline probe uses the first dataset request: navigate to trelis.com. An output already visible in the notebook came from an earlier fine-tuned run, so McGovern reruns the missing dataset cells before starting the fresh baseline. He reports nine rows and a longest formatted-row length of approximately 18,000. The raw 4B student receives the navigation request with 26 tools available. Its output is streamed to inspect whether it chooses a tool; a 2,000-token generation limit could be consumed by thinking before a call appears.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit adapters within a small memory and data budget
With baseline generation underway, McGovern applies rank-32 LoRA adapters to attention modules and MLP layers. The base matrices stay frozen while the smaller adapter matrices learn the update. He also enables rank-stabilized LoRA. Although he describes it in terms of adapting the learning rate to adapter size, the precise change is adapter scaling: PEFT’s LoRA configuration uses α/√r for rank-stabilized LoRA instead of ordinary LoRA’s α/r. This does not automatically change the optimizer’s learning-rate setting.
The trainer consumes the previously prepared text field. Its intentionally small configuration is:
| Setting | Workshop choice |
|---|---|
| Device batch size | 1 |
| Gradient accumulation | 1 |
| Epochs | 1 |
| Warm-up | Remove for this short run |
| Learning-rate schedule | Constant |
| Optimizer | AdamW 8-bit |
With nine rows, one epoch, device batch size one and gradient accumulation one, the run has nine update steps. McGovern describes the learning rate as relatively high for the small model, without giving its numeric value aloud. The 8-bit optimizer saves memory; a larger GPU such as an A40 or H100 would allow a larger batch. With this tiny batch, the training loss will be noisy.
Before training, McGovern reports approximately 15 GB of device memory, with 10 GB already reserved. Returning to the baseline output, he finds that the untuned student has more or less chosen the correct action. Reading both requested lines is his prediction, rather than a completed browser rollout in this notebook. He expects harder multi-step tasks to remain difficult, pointing back to the assistance needed by the larger teacher.
The training display reports that 1.62% of the model’s parameters are trainable; the main weights remain frozen. The experiment is therefore updating a small adapter set, not retraining the entire student.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Export the student and design a meaningful evaluation
The notebook provides a route back to the agent environment:
- Save the trained model and tokenizer.
- Authenticate to Hugging Face.
- Merge the adapters into 16-bit weights for the recommended export, then upload the merged model and tokenizer to Hub.
- Replace the model repository name in the existing RunPod serving configuration with that uploaded repository.
This would expose the student through the same style of inference endpoint used during collection. McGovern describes these export and deployment options while training proceeds.
A falling training loss is only a first diagnostic. McGovern recommends collecting a few hundred examples, reserving an evaluation split and logging with TensorBoard. The displayed loss is bumpy, consistent with the batch size of one. He suggests a batch around 32 for a larger experiment, with gradient accumulation as another way to increase the effective batch. But using the entire tiny dataset in one update would leave almost no sequence of learning steps; more data is part of the solution, not just a larger batch setting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give reward training a useful starting policy
Could trace generation be automated, or replaced with reward-based training such as GRPO? McGovern recommends beginning with manually curated, high-quality domain traces and supervised fine-tuning. His rationale is about exploration: if the initial model rarely completes the task, it generates many unsuccessful trajectories and receives few positive rewards. Teaching useful behavior first can make subsequent reward training more productive.
For browser tasks, he proposes a simple source of verifiable rewards: choose obscure websites, record the correct text from their top lines, and ask the model to navigate to those pages and retrieve it. The dataset then contains a ground-truth answer against which the result can be checked. The essential requirement is a systematic way to produce correct targets and reward successful retrieval; the workshop itself remains a supervised fine-tuning exercise.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A correct tool call closes the loop, not the evaluation
Training finishes with a generally lower loss. McGovern reports peak memory use of 75% and doubts that a device batch size of two would have fit. The notebook starts post-training generation with about 2,000 tokens available. He explicitly allows for the possibility that this small, noisy training run could disturb an answer the baseline already got right.
The fine-tuned model emits the expected tool call. That is a useful integration check, but the untuned student also selected the basic navigation action, and the workshop presents no held-out score or measured improvement in complete agent success. The next evaluation is to serve the student and run actual browser rollouts in the earlier workspace. The displayed inference configuration uses temperature 0.6, consistent with Qwen’s thinking-mode guidance rather than a universal setting for reasoning models.
The practical extension is to add the MCP servers needed for a real task and curate more examples of its recurring action sequences. McGovern suggests that roughly 50–100 manually generated examples could improve common, narrow task paths even without reinforcement learning; that is his expectation, not a measured result from this run. The workshop repository provides the starting materials, and its two companion MCP videos cover server setup and building a custom server. The useful specialization target is a repeatable task whose successful decisions, tool calls and observations you can capture well enough to teach.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Agent scripts, trace collection instructions and links to the fine-tuning notebook, slides and introductory MCP videos.
Tokenizer configuration showing how tool definitions, reasoning, calls and responses become model input.
Browser automation server exposing navigation and page inspection through MCP and accessibility snapshots.
Further reading
Versioned documentation for tool-call parsers, including the Hermes parser option used in the workshop.
Original explanation of square-root rank scaling for LoRA adapters.
Read the complete timestamped transcript
- 0:00
Okay. Welcome to this workshop on MCP Agent Fine-tuning. I'm Ronan from the Trelis Research YouTube channel. And today I'll be showing you how to run an agent that has access to tools via MCP servers, and then take traces or logs from the high-quality runs and use them to fine-tune and improve the performance of a model.
- 0:22
By the end of the workshop, you should be able to generate high-quality MCP agent reasoning traces, save the tools and the multi-turn traces from each of those runs, fine-tune, uh, a model.
- 0:36
I'm going to show you how to fine-tune Qwen3, but you'll be able to fine-tune other models too, and then run the fine-tuned model and see if the performance, uh, is any better.
- 0:45
Now, all of the materials are available online in this repo here, Trelis Research AI Worlds Fair 2025, and they're in the mcp-agent-fine-tune folder.
- 0:56
I'll be cloning this and working through it very shortly, but first I do want to give a brief introduction to what MCP is, that's Model Context Protocol, how it allows a language model access to tools, and I'll emphasize what we need to understand about those interactions when we're building both the fine-tuning scripts and also the agent scripts
- 1:15
and saving of those logs. So what is MCP, Model Context Protocol? It is a protocol for providing services to LLMs, namely access to tools. The tool we're going to focus on is browser use, so using an LLM to navigate through websites.
- 1:30
But there are MCPs for things like Stripe, GitHub, Gmail, and many more you can find. You can sub those into these examples today, but I recommend running through first using the MCP tool that I've selected until you're comfortable with how everything works.
- 1:45
MCP does a few things. It's first of all a store of information on tools. That information helps the LLM to know how it can make calls to the tools or make use of them.
- 1:56
But the MCP tool service also runs the tools. So when the LLM decides to make a call, MCP, the tool service, will take the action, whether that's adding numbers or navigating to a page, and it will then return a response which contains details of the result or maybe some extra helping guidance for the LLM, so the LLM
- 2:16
can then loop back and make another tool call or provide some kind of text-based response.
- 2:22
Now, to bring it to one more level of detail, we're going to wrap the language model here, or we're going to expose it in the form of an API that we can hit, and we're going to expose it as an OpenAI endpoint, which is, um, pretty common.
- 2:37
Of course, it's the one OpenAI itself uses, but using that library, there are many models and many libraries that will make use of that style of API endpoint if you want to set up, uh, an endpoint for your language model.
- 2:48
So this is what we're going to use. And in integrating this API endpoint, there are a few points of integration or translation we need to take care of. Don't worry if you don't understand this first time around because it'll become apparent in the code.
- 3:01
The first one is we will get tool information from the MCP services, and we need to convert that into lists of JSON tools because that's what OpenAI endpoints expect.
- 3:11
Second of all, we will need to convert the tool response into a format that the language model expects. And lastly, when we have, uh, the language model calling a tool, and it will call it just by emitting tokens or text, we need to extract, detect and extract whether it wants to make a tool call or not.
- 3:31
And the form that the text will take when it wants to make a cool- tool call is going to be, uh, in Hermes format actually, even though it's a Qwen model.
- 3:40
And I'll talk about that, uh, in just a little bit. To give you a better idea of how all of this fits together, it's helpful to look at a prompt.
- 3:47
We can take a look at, um, this little chat I've prepared. It just shows a little pseudo prompt. It's the string that is being sent into the LLM. It starts off with a system message just after a system start tag, and it starts by describing to the LLM how to make tune-- tool calls, namely by passing them
- 4:08
as JSONs within these tool XML tags here. And the LLM is being told, "Okay, you have access to deal to these tools." There'll only be one in our case, which is the browser.
- 4:19
"Please return each function call as JSON inside tool call tags if you wish to call a function." So this... We're telling the LLM, "Here's how to call a, a tool by giving us these XML tags and then the JSON."
- 4:31
After the system message, we have the first user message, which would be the first message like navigate to trellis.com, and then the assistant will respond, and it can respond in a few ways.
- 4:41
Typically, it might do some thinking here because it's a thinking model, and then decide to call, uh, some tool like navigating with the browser to a certain URL. Or if it's finished navigating, it might decide to think and then just provide an assistant response as text.
- 4:57
So this is, uh, a little pseudo prompt here, and we'll see this come into play in the code when we want to set up and inject these different parts.
- 5:05
Okay. So that brings me to the end of my very quick overview of MCP and the prompt structure. If you're not following fully, that's okay because it'll become a lot more clear when I run the examples, which I'll do now by cloning this repo, ai-worlds-fair, into Windsurf.
- 5:22
If you clone it, you want to open up then the mcp-agent-fine-tune folder and go to the README. Now, in this README, there are two sections. There's the data collection section where we run the agent to generate these sample runs, and then there's the fine-tuning, which will be done in a notebook.
- 5:38
So starting off with the data collection, we need to make sure we have an endpoint. We need to make sure it's an OpenAI style endpoint. And if we're not using OpenAI itself, which we won't be because I'm going to generate the traces using a Qwen-type agent because I'm going to fine-tune a Qwen model later, I do recommend
- 5:55
keeping some consistency between the model to generate data and the model you're going to fine-tune. In principle, you could just use a stronger model, any stronger model Um, but we want the stronger model to share its reasoning traces so we can train on those, and OpenAI models don't share their thinking traces.
- 6:10
So we're going to use a Qwen model. Specifically, we're going to use the thirty billion parameter one with, uh, three billion activated, uh, parameters. It's a mixture of experts model.
- 6:21
And I'm going to run it on RunPod. I'm going to use this one-click affiliate template here. You can run it, though, on any service. In principle, I think you could run it locally as well, provided local, uh, your local
- 6:34
app supports tool calls. I'd need to look into whether LM Studio does, uh, support that. But if you click that one-click link, you'll pick a GPU, maybe a H100, and you can inspect then the template.
- 6:46
So a few things here. We're running a Docker image for VLLM. We're running this Qwen model. We're going to enable reasoning and a reasoning parser. What this means is that when we see a response that includes think tokens, we're going to det-detect those and extract it as reasoning into a JSON, uh, type style in the response.
- 7:08
Next, we're going to set the max model length to thirty-two thousand. We'll set the port. We're going to host it on port eight thousand, and we're going to enable automatic tool choice.
- 7:17
This means the LLM will decide whether or not to call a tool and which tool to call. Here we have the tool parser that we're choosing. You'll remember in my presentation, I said we need to specify how to extract the tool into JSON format.
- 7:31
What this means is when the LLM makes a tool call like this, we're saying, "Okay, the tool call is in this format. Hermes, extract it into this JSON here and provide it to the MCP in a format it's expecting."
- 7:45
So this is specifically this point here where it converts from the language model string into a JSON format that OpenAI, uh, the API expects. Okay, so these are all of the arguments we'll use, and then we click here to start the server.
- 7:59
We do need, need to expose port eight thousand because that's where we're serving it. We're going to hit that port and server from outside, and the way we're going to hit it...
- 8:07
By the way, I would click Set Overrides and start the server. I actually have a server that's running already here. It will take about three minutes to start up.
- 8:15
And to hit the server, we need to specify the RunPod pod ID and also the port number, which is eight thousand. So you'll see here when we run this agent, and I'm going to start just by running it without explaining it because it'll be more intuitive when you see an example.
- 8:27
I'm going to run the agent with this model here. I'm going to run it with this base URL, which has got the pod ID and the port number, and it's got the V1 that is required for the OpenAI style endpoint.
- 8:42
And there's one more argument here I'm using, which is truncate. This is truncating the length of the tool response. So when you have a tool response, for example, here there's a tool response coming back.
- 8:53
If we call a browser navigate tool to navigate to trials.com, it's going to return an accessibility tree, which is a text description of that page, and that will be provided back in the tool response so the language model can read the page.
- 9:07
But this could be really long, and so I'm just naively truncating it, so we just take the top part. Now bear in mind that means that the LLM won't probably be able to see the full length of the page, and you would need to do something more complex, uh, in a deeper implementation of what I'm showing.
- 9:23
So we're going to run the agent here and first run UV sync just to make sure you've synced the requirements. If you don't have UV, do pip install UV, and then you can run the agent here.
- 9:35
So you'll see the agent has started up. We've started the MCP server. This is the only server that's configured, and it's configured in the config file here. You can add in other servers here, and it will load up those tools.
- 9:48
Playwright offers twenty-five tools, so there's quite a few, like I think navigate forwards, backwards, switch tab, navigate to link, different things like this. We'll see more of them later.
- 9:58
And those twenty-five tools are all available to the LLM. For open source models, probably you don't want more than maybe twenty-five to fifty tools. I think for Claude you can probably handle two hundred.
- 10:07
More than that, the LLM will get confused by a lot of, uh, context.
- 10:12
So now it's asking me to put in an input, and I'm going to say navigate to trellis.com and read out the top two lines.
- 10:23
Let's see what it does with that. It should start to think now. That user message has gone to the LLM. It should be thinking now, generating thinking tokens. They won't be shown here just for brevity.
- 10:33
I'll save them later on. And then, yeah, it's asking for a tool call to move and navigate to trellis.com. Okay, and actually before I say yes to that tool, which, uh, requires my approval, I'm just going to put this screen on the side.
- 10:47
So I leave space here because the browser's going to pop up. And the reason the browser will pop up is because I'm running this browser. Uh, I'm not running it in headless mode.
- 10:55
If you want the browser to operate in the background, you would just put in, uh, dash dash headless. But I actually want to do it because I want to see the browser and what it's doing.
- 11:03
So I'm going to, um, I'm going to ask this, uh, question here, which is navigate
- 11:12
to trellis.com and read out the top two lines. Okay.
- 11:22
So I'm expecting the model to start thinking here. The tokens won't be printed because I'm saving them just to avoid clutter on the screen. And then, yeah, it's decided to call a tool, which is trellis.com.
- 11:32
Navigate there. I've built in approval to the agent, so I'm going to say yes. And now it has opened up that browser, which is good. And the structure, the accessibility structure should be sent now to the LLM, and it is picking out, uh, the top line here.
- 11:47
And it's also picking out the second line here. So this looks really good. I'm just going to exit. And that was our first, uh, quick demo.
- 11:57
So I've got a few little demos here. I'm just going to clean up two demos, uh, from before where I didn't have my screen set up And you can see the logs here.
- 12:07
So I've got a feature where by default when you run the agent, things are going to be logged, uh, like this with two parts, with the messages part and the tools part.
- 12:16
And this is exactly what we need for fine-tuning. We need to know a list of the tools, and we need the full conversation history. So you'll see in the tools a list of the 26, browser close, browser resize, et cetera.
- 12:28
And under messages, you can see my user request,
- 12:32
then the assistant starts thinking and, yeah, it was quite a bit of thinking. User wants me to navigate to trellis.com, et cetera, et cetera. So it goes through all this logic and then it decides to make a tool call here.
- 12:44
The tool call is sent to the MCP, and then Playwright basically opens that up, as you saw, and sends back the accessibility trace, which includes details on the page but truncated.
- 12:56
And then the assistant takes that and says, "Oh, I have the information I need. Here's some reasoning, and here's my final answer." So this is a very nice trace, and we're going to keep it.
- 13:07
We're going to keep it, and we're going to use this for fine-tuning. Now, it wouldn't make sense to fine-tune using this on the same model because the model can already do this, but it can make sense to fine-tune a smaller model that doesn't have, uh, this kind of capability.
- 13:23
So your goal now is you should set this up and you should run this, and I want you to run it multiple times and try to get some good traces, and then we'll save those traces and we're going to upload them to Hugging, Hugging Face Hub.
- 13:35
Okay, so we'll run one more example here. I'm going to make it a bit more complicated this time, and I'm going to say navigate to trellis.com and then find the third product, navigate to that page and tell me what is on...
- 13:51
Um, you know what? Let's try and ask a specific question. So let's go to trellis.com and manually do the navigation first. Oops. Oh, I can't type. Here we go.
- 14:04
Trellis.com. The third model is training and fine-tuning. We'll navigate there. It opens in a new tab, which by the way is tricky for navigation. And let's ask it, uh, what the first line is.
- 14:15
Yeah, let's ask what is in the feature section. Okay, we'll try that. So the model should be thinking. We should see the browser pop up
- 14:26
in a second once we approve that navigation. It should get a snapshot,
- 14:32
and we'll see what it decides to do next. It should try to do this drop-down menu here, which is a bit tricky. It is clicking on something, so that's good.
- 14:40
It did get the drop-down menu. That's also good. And okay, it's actually just given an answer, which is not exactly what we want.
- 14:50
Yeah. Well, the answer's correct. It says, "Do you want to click the Learn More section?" Yes, click that and then navigate.
- 14:57
Um, and then make sure to switch tabs because it will open in a new tab. So yeah, I mean this is kind of working out well. There's always going to be noise, so some of the traces won't be good, and I'll show you some ways to deal with that.
- 15:13
One is to just manually adjust the trace. All right, it's going to click on Learn More, and that has opened a new tab, which is good.
- 15:21
And now it's going to think. It's thinking in the background. We'll save the thinking logs. And it's asking to select that, uh, tab too, which is this tab.
- 15:31
And I think at this point it will get a browser snapshot, but I'm not entirely sure of that. And let's see. So
- 15:39
it, um, the core features are GitHub repo access and ability to post issues. So yeah, this is correct. Um, it's getting the correct content, so it's actually fulfilled this task very well, and I'm going to exit.
- 15:54
Now, before I show you the trace, there's one more trick. If it's not following what you want it to do, you can kind of guide it like I did in the user prompt and adjust retrospectively, but you can also pass in a system prompt.
- 16:05
So you could pass in a system prompt like this that literally explains very directly how it should do this task and what tools to call. And this can also help performance, and the idea is you get it to create a nice tidy trace, and then you don't have to include the system prompt in your training data.
- 16:21
You just train on that nice trace because our goal is to get nice traces for, for training data. So I'll just show you that, and then you should go yourself and try to make some of these different traces.
- 16:31
Let's find that more complex one, which I just made. Let's see if I want to make some adjustments here. So we have the tools that are saved, and I'm, I'm not going to change those.
- 16:43
But right here when I say, uh, make sure to switch tabs, um, let's just expand that thinking. I need to find... I'm trying to find the section where I said, yeah, click that and yeah, okay.
- 17:01
So one thing you could do is literally delete that user turn and combine [laughs], and combine kind of these two sections here. So you could kind of do that manually or just literally copy paste it into ChatGPT and ask ChatGPT to generate, uh, just a single version.
- 17:16
You would need to delete the assistant content, just provide reasoning. Um, but another approach here is just to say, uh, yes, click that and proceed. Um,
- 17:29
and i- instead of saying and switch to a new tab,
- 17:33
I may then need to switch tabs if it opens in a new tab [laughs]. So you can see I'm cleaning up the trace here,
- 17:43
and yeah, this is some nice data. So we have a few data points that I've saved, and what we want to do now is push this up, uh, to, to Hub.
- 17:54
So we want to create a data set that has the tools and that has the conversations that we'll use for fine-tuning. So for that, I'm just going to run push to Hub, and quite simply, it's going to grab the tools and it's going to grab the, the messages and push them up to a data set Now, there's
- 18:10
one subtle point here that I'll show you, which is called, uh, unrolling the data because where we want to train on multiple turns, not just the first response, but the second and the third.
- 18:20
So actually, if we have, say, three back-and-forths, we want to unroll that into three rolls. One roll with all of them, one roll with just two, and one roll with one.
- 18:28
So actually we get kind of three for the price of one in terms of, of training data here. And there is a subtle point that the template of Qwen will always include the reasoning from the most recent turn.
- 18:40
So even if we include reasoning and save it for all of the turns, it will only inject the most recent reasoning, and that's another reason why doing this unrolling is important.
- 18:49
Uh, ask me a question in the comments, uh, if you want more clarification on that. It's kind of a subtle point. If you want to push to Hub, you can just look up, uh, in the read me.
- 18:59
There's a function here, push to Hub. You need to specify the repo ID and then pass in unroll if you want to do that unrolling.
- 19:08
And this data set, uh, should get pushed up. Now, I do need to, uh, log into Hugging Face,
- 19:15
so I'm just gonna do that. So I've put in my, uh, my token. It needs to have write permissions, and then I can run push to Hub again. And it's a pretty small data set.
- 19:26
There's nine examples, so it's unrolling these four into nine. One of them has three, the others have two when unrolled. And then I can check out my data set, which has got, um, ID timestamp, the model, the messages, and then the tools.
- 19:40
And you can see, uh, I think there's also a flag, whether it's been truncated, which means unrolled. So say this last example,
- 19:49
uh, well, it's not the last example, but there's an example here where we have unrolled it twice. So the first example here, second example, third example has got two-- it's got three turns, so there are two unrolls.
- 20:03
So you'll see if you look at the messages here, this will give the full three turns. So it'll be fairly long. It will have all of the messages. And notice how the reasoning content is extracted.
- 20:14
That's because we've turned on the reasoning parser. Uh, but if we scroll right down to the bottom of this, and I know it's kind of long, that's because the tool responses are long.
- 20:22
You can see the last response is the answer, or at least it should be. Yeah. It's saying what's on that page. Whereas if you go up to the one above that where it's been truncated, um, it should just be a tool call, I think.
- 20:36
Or it may actually be that assistant response we got.
- 20:40
Uh, yeah, it's, uh, got some reasoning content, and then it's making a tool call down here where it's deciding to select the tab. So you can see we've got nine rows of data here, and we will grab these messages, and we will template them along with the tools in our fine-tuning template, uh, when we want to fine-tune
- 20:58
the model, which we're going to do very shortly now. Okay, so at this point, you should be able to go ahead and run this a few times, try to modify some of the traces, try to push some of that data up to Hub.
- 21:11
I will maybe very briefly show you the agent. Um,
- 21:17
let me see how I can do this quickly.
- 21:20
I'm basically setting up an OpenAI endpoint. Um,
- 21:25
and what can I say that won't be too long? I need to set up the server. So I have server management here. Any MCP service that I want to run, I need to boot it up when I start the agent.
- 21:36
So that's one of the first things. I need to do schema conversion then. So I talked about how you have MCP that has a specific syntax, but OpenAI expects, the endpoint expects a different syntax, so there's conversions that need to happen.
- 21:50
There are some kind of helper functions, but then here's how we convert the turl, turn, the tools from MCP to the OpenAI format, and then we need to go OpenAI to MCP.
- 21:58
So we need to go both directions. Here's where we execute the tools, and here's how we wrap the result in a format that's going to be suitable for OpenAI.
- 22:08
So that's all those connections I showed you in the PowerPoint.
- 22:12
This here is discovering all the tools, then booting up the servers, and this is the chat loop, which you saw operating below. There's not a whole lot that's, uh,
- 22:21
to explain beyond what you saw me doing. You can see reasoning has been parsed separately to the assistant content. Um, and
- 22:31
we have a helper here for trace logging just to save the details. And yeah, we've some logic here for, for doing the logging and then some different, uh, some different arguments that we can pass.
- 22:43
So I'll let you dig on that in your own time. In the meantime, let's move to the fine-tuning steps. So we've got these traces, and we want to use them to improve the performance of the LLM.
- 22:52
We'll go to the fine-tuning section and open up this notebook. I have it open here.
- 22:59
Uh, my runtime is disconnected. I'll just restart. This is based on an Unsloth notebook for fine-tuning Qwen, but it's got some things that I've changed.
- 23:08
First thing we'll do is install Unsloth, which has got a bit more of a detailed, detailed install if you're on Colab, and then we load a model. Now I'm going to train the four billion parameter model.
- 23:18
You can train a larger model if you want better performance. The max sequence length needs to be fairly large. If you want to train on the free GPU, you probably can't cha-train a larger model.
- 23:29
Um, you'll need to run maybe on RunPod by starting up. You can find a fine-tuning te- one-click template actually in the one-click LLMs repo. You can start that up and then open Jupyter and then open the n- download the notebook here and upload it.
- 23:42
We're going to run it in full precision in 16 bits.
- 23:46
And once we've loaded the model, we'll want to set it up for training. We're actually not going to train all the parameters. We're going to only train adapters that we connect to certain parts, certain matrices.
- 23:57
They're called, uh, low rank adapters or LoRA. And I'll talk about applying those adapters in a moment. But I want to actually get a benchmark on performance without fine-tuning, so I'm not going to apply the LoRA adapters just yet.
- 24:08
I'm going to prepare the data, and then I'm going to run inference without having done fine-tuning. And the data we're going to load is that data set that we've been looking at right here So we load it up and we just print it out to see how many rows.
- 24:21
It should be nine rows now. If you want, you can take a look at, uh, the format of the messages or the tools just by running these to inspect the zeroth row.
- 24:31
You can also take a look at the chat template. Uh, the template is what takes in the tools and the messages and puts it into a single long string.
- 24:39
In fact, it's probably worth looking at, so we just run that. You can also find it if you go to the model. You can go to the tokenizer config on the model on Hugging Face, and you can see a copy of that there as well.
- 24:49
By the way, um, my runtime restarted, so I need to re-download the model weights. They're just downloading here. It'll take a few moments. And yeah, as I said, we take a look here at the data set.
- 25:04
Sorry, I've got a really long data set row saved.
- 25:08
Once we have the data set downloaded, which means we're going to have access to the messages and the tools, we basically pass them into the chat template, which will make a single long string, and that single long string is basically this.
- 25:20
It's the formatted data set as a single long string, and it is very long because it contains all of those, uh, tool responses. But you can see it starts off with a system message.
- 25:29
It starts off with those tools. It then gives a list of the tools available, which is very long because there's 26 of them. Then it'll give the user message, the assistant message, the tool calls, et cetera.
- 25:39
So that's, uh, what it's going to look like. When we rerun this, we will have nine rows, and then we'll just print out to check what the length of the longest row is, because we don't want it to be longer than 32,000, which is our max length.
- 25:50
Um, so we'll do a check there. Now we will train the model, but I said first we'll just do a quick run to see how performance is without training.
- 25:59
And for that, we'll just run on the first row of the data set, and we will see if the model decides to call a tool correctly when I ask it to navigate to, um, Trelis.com.
- 26:14
So yeah, if we print out, um... Yeah, if we print this out,
- 26:22
we should start to get it to, uh, stream a response. Here's one I streamed earlier with a fine-tuned model and it correctly called a tool. But if I, if I call this again now, let's just quickly check the length of the data set.
- 26:33
Oh, the data set is not defined. Why is that? I don't think I ran all of my data set rows.
- 26:40
So let's just check. Downloaded the data set.
- 26:45
All these rows will run quickly. It's quite a small data set. Yeah, nine rows. The longest is 18,000. That would be fine.
- 26:55
And yeah, you can see if I print out the zeroth message, I'm asking it to navigate to Trelis.com. And if I run inference on that, it should just start to stream its output string, and we'll see whether it decides to call a tool or not.
- 27:11
So we're just testing here the raw 4 billion parameter model, which should be capable of tool calling. This is a lot of tools for a 4 billion parameter model, though, 26 tools.
- 27:21
So, you know, it's going to blab on here with a lot of thinking, and then we'll check whether it gives a tool call. It may or may not. It may decide to blab on for a full 2,000 tokens and then just stop when it hits the limit.
- 27:34
So we'll come back to this. But what you're going to do now that this model is loaded is you're going to prepare it for fine-tuning by applying LoRA adapters.
- 27:42
And we're going to apply LoRA adapters to the attention modules and also to the MLP layers. We will use a rank of 32, which means the adapter matrices will, uh, have effectively a, a width or height of 32.
- 27:54
We'll use, uh, rescaled LoRA, which will adapt the learning rate of the LoRA adapters according to their size. Larger LoRA adapters means loring-- lower learning rate. Uh, so I'll go ahead and apply that.
- 28:06
The data should all be prepped, so we can go right to training the model here. We're going to pass in the data set, and it's going to train based on the text field because we've prepared that text field.
- 28:17
Uh, I went through this quickly, but if you recall, when we formatted the messages and the tools, we templated them into one string of text. That's the text key there.
- 28:28
And when we did that, we essentially prepared it so we could pass in just rows of text to our trainer. Batch size of one. This is not ideal because you don't get much, uh, smoothing.
- 28:38
It's going to make it very jumpy in training loss, but we don't have enough VRAM here. If you run on an A40 or on a H100, you'll have more VRAM, so you can use a larger batch size.
- 28:48
We just trained for one epoch. Um, we actually shouldn't have warm-up steps because we don't have enough steps here at all. If I have nine rows, I'm only going to have nine steps with gradient accumulation one and device batch size of one.
- 29:00
Learning rate is fairly high because it's a small model. Using the AdamW 8-bit optimizer to save a bit of VRAM, and we'll use a constant learning rate. Everything here is quite simple.
- 29:10
Um, I'll probably do a, a follow-up video with some more advanced techniques on the YouTube channel, on the Trelis Research channel afterwards, uh, so keep an eye out for that.
- 29:18
But I wanted to keep this fairly simple so we can go through it, uh, quickly. So yeah, we'll run the training. That's actually just setting up the training. We'll check the device memory.
- 29:27
It's got, uh, 15 gigabytes, 10 of it's reserved already, and we can start to train. After training, we'll show the final code, and then we'll run inference again. Now, let's just take a look at what it did without training, and actually pretty good.
- 29:42
So on this basic task here, um, it more or less got the problem correct. Would it be able to read out the top two lines? Probably it would, even with the 4B model.
- 29:52
But I can tell you that if you do a more complicated task like multi-step, um, it will struggle. It already struggled when we did the 30B model, as I showed you.
- 30:00
It was kind of lucky I got it to work when I did. Uh, but this 4B model will definitely struggle on multi-step.
- 30:06
So after the training, I'll rerun that and we can see, uh, what it does. Hopefully, it'll also get it right.
- 30:12
You can see the training is underway here. We've got nine steps. We're only training 1.62% of the parameters, so those adapters are tiny. Everything else is frozen. All the main weights are frozen.
- 30:24
And it'll take a little bit of time for this to complete and then inference to run. And when that's done, we've the option to save the model and the tokenizer.
- 30:34
We also have the option then to push the model up to Hub. If you want to do that, you can just log into Hugging Face here, and then I recommend merging it to 16 bits.
- 30:43
If you merge it with this name here, for example, you can actually go back into your pod, you could edit the pod and just swap, drop in here the name of your fine-tuned model, and now you've got an inference endpoint, uh, ready to go.
- 30:57
So, let's see how our training is doing. About two minutes left and our training loss is falling. You know, in a more advanced implementation, you would want to have an eval set.
- 31:07
So you want to have much more data, not just nine traces. You want to have probably a few hundred. You want to split some of those off into an eval set.
- 31:15
You also want to do logging with TensorBoard. I go through that in most of my videos, but, uh, I wanted to keep this fairly simple. Okay, so we have the training completed, and generally the loss is kind of falling.
- 31:29
Like, it's very bumpy, and that's-- it's not quite completed by the way, it's just finishing up. But it's not helped by the fact that we only have batch size of one.
- 31:37
Ideally, we'd have batch size of like 32 maybe, which would require quite a bit more data. Um, you could use gradient accumulation, but if you have a very small amount of data, you actually just have to set the batch size smaller.
- 31:48
Um, you don't want to do just one update step with all of the data. So we're kind of doing the best we can here given, uh, the limited amount of data, and this is something you can definitely add a lot more traces to, hopefully.
- 31:59
Now, just one other comment because I know one question will be around reinforcement learning. Can we automate this so that we generate traces automatically or use some rewa-reward-based system?
- 32:09
And my recommendation there is start off with manual traces and curate them like this because even if you want to do rewards afterwards, like GRPO, and you can check some of the other videos on that, it's going to be beneficial to SFT supervise fine-tune like I am now on some traces that are high quality.
- 32:28
If you just start with a model that you want to do reinforcement learning on, but you've not done any supervised fine-tuning for that domain, it's going to struggle. It's going to generate lots of traces and very rarely get to the correct answer where it sees a positive reward.
- 32:41
So it will speed up your training a lot, even if you do reinforcement learning afterwards, um, if you first generate some high-quality traces and then do the fine-tuning. Now, if you want to do GRPO, you need to define rewards.
- 32:54
That means you need to have a data set where you have answers that could involve picking some random niche websites and picking, finding maybe the text on the top, and then asking the language model to navigate to those kind of obscure pages, and then you have a ground truth of what those lines of text are.
- 33:10
That's just a really simple example, but you need to come up with a way to systematically generate data that's verifiably correct and then reward the model when it gets that answer correct.
- 33:19
Okay, that was a bit of a sidebar on reinforcement learning, which is not the focus here, but it's very much related. We have finished the training. The loss, I'd say, generally has, has gone down.
- 33:31
Uh, we used peak memory 75%, so we didn't have space for probably doing a batch size of two,
- 33:37
and now we're generating response. So the model is thinking,
- 33:41
and it's got about 2,000 tokens to think. Let's see if it finishes up, uh, with recommending a tool call. It may or may not. It's possible that with this noisy of a training, we bumped it off what was a good answer, at least the first time around.
- 33:54
By the way, I'm using default parameters for running inference. You can set them manually here. There's recommended parameters for Qwen for the thinking model. And yeah, it looks good.
- 34:04
It's calling the tool as expected. So you'd want to do a much more elaborate setup on the evaluation. You probably want to run your model on the endpoint and actually try to run some of the rollouts over in, um, the workspace on Windsurf that I showed you earlier.
- 34:18
Just in case you want to see, uh, what the default configuration parameters are, you can see the temperature's .6, and I think these are the recommended parameters if you're using, uh, a reasoning type model.
- 34:29
Okay, with that, it brings us full circle. Hopefully, you've had an opportunity to create some good traces and understand now how to do fine-tuning, at least in principle, even if we didn't use all that much data.
- 34:41
This is something you can build on by adding in different MCP servers, curating many traces. And I think it's, it's probably quite powerful, even without moving to RL. If you've got a small number of examples, um, and I mean maybe 100 or 50, something that you can realistically generate yourself, I think you can get, uh, significant improvements
- 34:58
in performance, particularly if there are some kind of narrow trees that are common, uh, but important for your use case. As I said at the start, all of the materials are in Trelis Research AI Worlds Fair 2025.
- 35:11
You might also want to check out the two MCP videos that are on Trelis Research on YouTube if you want to understand in a bit more detail how those servers are not just set up, but how you would make a custom one by yourself.
- 35:22
In the meantime, any questions, uh, let us know by posting down below in the comments. Cheers folks.